Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(aio): move datasets into offline evaluations (#121572)

Radu Raicea35cfd4e69ae8
diff --git a/docs/internal/ai-offline-evaluation-reporting.md b/docs/internal/ai-offline-evaluation-reporting.md
index 068a31b1c7fa..c6165771a7bb 100644
--- a/docs/internal/ai-offline-evaluation-reporting.md
+++ b/docs/internal/ai-offline-evaluation-reporting.md
@@ -553,6 +553,23 @@ Opening those resources requires their own permissions.
## Inspecting offline results
Open **Evaluations → Offline evals** to see the newest experiments and chosen score trends.
+When offline evaluations are enabled, **Experiments** is selected by default. **Datasets** opens the list on the same page
+with `tab=datasets` in the query string. MCP-generated dataset-list links retain this tab selector.
+Existing `/ai-evals/datasets` and `/llm-analytics/datasets`
+links redirect there, preserving filters and pagination. Dataset creation and detail pages live at
+`/ai-evals/evaluations/offline/datasets/:id`, within the Offline evals navigation. Both legacy dataset
+detail routes redirect there, preserving item deep links, revisions, tabs, filters, and hashes.
+The Evaluations heading and navigation stay in place on dataset creation and detail pages. Dataset items appear
+directly in the detail view; **Details** opens the dataset metadata without replacing the items table.
+The back arrow beside the dataset name returns to the list and preserves its filters, including after creating a dataset.
+Creation and editing actions appear below the dataset form. Switching from Experiments to Datasets clears
+experiment search and filters and restores the saved dataset tag selection.
+Projects with Datasets enabled but without offline evaluations open **Datasets** automatically,
+including from experiment-list links. **Experiments** stays visible with a disabled **Coming soon** label.
+When Datasets is disabled, its tab stays disabled and direct list links show an availability message.
+The chevron beside **Score trends** collapses or expands the charts. It starts expanded and remembers
+your choice in local storage. Charts load on the first expansion if the page opens collapsed, and
+keep their data when collapsed and expanded again. Refresh or a filter change updates loaded charts.
A shared time range, run source, and upload state filter applies to both the score charts and the experiment list.
The overview starts with the last 30 days and all upload states. Uploading and failed runs can show partial score summaries.
Projects without experiments show setup steps; a filter with no matching experiments keeps the overview available.
diff --git a/frontend/bin/build-app-url-manifest.mjs b/frontend/bin/build-app-url-manifest.mjs
index 265997dabd2a..b99e94a8eb0f 100644
--- a/frontend/bin/build-app-url-manifest.mjs
+++ b/frontend/bin/build-app-url-manifest.mjs
@@ -63,6 +63,8 @@ const EXCLUDED_HELPERS = new Set(['absolute', 'default', 'project', 'currentProj
// `tracingOperation` renders its missing-parameter error, and `dashboardTemplates` opens the
// dashboard list without its templates modal.
const EXCLUDED_BUILDERS = new Set(['tracingOperation', 'dashboardTemplates'])
+// Dropping these builders' fixed query strings opens a different destination.
+const BUILDERS_WITH_REQUIRED_QUERY = new Set(['aiObservabilityDatasets'])
// Flag-gated surfaces for PostHog staff only: a link an agent hands a customer would land on a scene that refuses them.
const INTERNAL_BUILDERS = new Set(['decisionPlayground', 'library', 'todayReport'])
@@ -245,7 +247,7 @@ function readSignature(fn) {
// Invoke with the fewest sentinel args that yield a clean path — drops optional trailing params and
// hits guarded default branches (`replay()` -> `/replay/home`) instead of erroring on a sentinel.
-function probeBuilder(fn, signature) {
+function probeBuilder(fn, signature, preserveQuery = false) {
const maxArgs = signature.kind === 'object' ? 0 : signature.names.length
for (let count = 0; count <= maxArgs; count++) {
const args = signature.kind === 'object' ? [{}] : signature.names.slice(0, count).map(sentinel)
@@ -258,7 +260,8 @@ function probeBuilder(fn, signature) {
if (typeof raw !== 'string') {
continue
}
- const pathOnly = raw.split('?')[0].split('#')[0]
+ const withoutHash = raw.split('#')[0]
+ const pathOnly = withoutHash.split('?')[0]
if (
!pathOnly.startsWith('/') ||
pathOnly === '/' ||
@@ -267,7 +270,7 @@ function probeBuilder(fn, signature) {
) {
continue
}
- return pathOnly
+ return preserveQuery ? withoutHash : pathOnly
}
return null
}
@@ -320,7 +323,7 @@ function buildManifest(urls) {
excluded.push({ name, reason: 'could not parse signature' })
continue
}
- const probePath = probeBuilder(fn, signature)
+ const probePath = probeBuilder(fn, signature, BUILDERS_WITH_REQUIRED_QUERY.has(name))
if (!probePath) {
excluded.push({ name, reason: 'no clean path' })
continue
diff --git a/frontend/snapshots.yml b/frontend/snapshots.yml
index 5abc309df5d4..a5b59aa6120c 100644
--- a/frontend/snapshots.yml
+++ b/frontend/snapshots.yml
@@ -4509,9 +4509,9 @@ snapshots:
layout-products-and-files--products-starred-collapsed--light:
hash: v1.k794b7964.44533a12c58c83a253be3424bd24a6eae261feec47d3411bd896ac21cf584525.8lDvM03JlIuE1ki0q3wpcBen63AqJnQiiGqmFu87yk4
layout-products-and-files--search--dark:
- hash: v1.k794b7964.6695fe610e3f2c6defc39393d92d7075ee02c494f49c840a6aacb90f521bfe1f.6pIULWHpgkRwZOKuJOF4mNBxJlVemTXRwX0f-wbvnXI
+ hash: v1.k794b7964.e1fe572d5b4b897dcf4e0ad0b1ffdec60b9bf74f5061b4f49bc6e89b10d0baa4.sfgp48RdHVp8bSgi-ZqJIBOQHghQCA3koUdq5CxwTUg
layout-products-and-files--search--light:
- hash: v1.k794b7964.2f1fc9d1cd3b0c4141c564ba244b0cf17599ff92a95fbea4937f5a7c05fd5040.hwD6HSMsaa7NWobGJRS5hjX89po3O5KU10hOKbE-Zbs
+ hash: v1.k794b7964.1f6e05d594715431db27c7c8c5c55d60bf0aae1ce6ec8b88b79183749d865870.Rg_tcvdTUI6VRn6u3XBDxkYmNSVAXw0C-iyN8XtV6wE
layout-products-product-tooltip--dark--dark:
hash: v1.k794b7964.52ba01b050e579d7c600df08497a2dfab2dba9cf670553e032afce3819d6d00d.PNNtTamfHv5qRTsFnRwxhlFEaIteMOgYQgAab2QRRj0
layout-products-product-tooltip--dark--light:
@@ -8822,6 +8822,26 @@ snapshots:
hash: v1.k794b7964.6a8f9cfbd5283164176a115eb6aa1ac80fe67c9f7a5fc44ba3c08ef04ff575be._962ei0cP3XkOKCVtwajtH83Pjlx2Vdaaj9bUI3qsP4
scenes-app-ai-observability-conversation-display--tools--light:
hash: v1.k794b7964.cc6f7770155f4b720ff0597c3dd5deff699a267cfbdd8fe76a131cb51740215d.lKCAOsQBSoBkK46EHSYdGmWxAi4CQ4wUrx2J1iRF_f0
+ scenes-app-ai-observability-dataset--datasets-only-narrow--dark:
+ hash: v1.k794b7964.9bea36774533c6f125e958f3469a59b8080d36b9ff5546bb92770ee5789db6ce.NLbIs2qFi3yuLycJ1To_LN5AdGNrWH25ayWiPQMsUmc
+ scenes-app-ai-observability-dataset--datasets-only-narrow--light:
+ hash: v1.k794b7964.c3caaf1aac1bdfe99eca9ebe496276c2f6cf43ce915731c33be5a31be4559a12.aXsTGkDbE3QxmaPfNA5i0Ii5mhLkyjvZoDQF3pHXA7g
+ scenes-app-ai-observability-dataset--details--dark:
+ hash: v1.k794b7964.1122d600365d38303883c8e6d986964fe7ccbd096922d18024722364a618779e.NSoJGML-0Zw9WarRka5NuBy1dItmCNgsiRRZCA5uKYA
+ scenes-app-ai-observability-dataset--details--light:
+ hash: v1.k794b7964.1fbe29482ffd475101b36f6ac411b80a7bfbf25c6486b970463727aa976f08e7.ARBMFFBiA_F7UvW7eyBy5IbGasKLGw5yBAFTTk_CQ1o
+ scenes-app-ai-observability-dataset--items--dark:
+ hash: v1.k794b7964.e1c2f9a6a56d543640d473a518dd09e37f61160d9086c174b412eb5bd293ae42.29BCUSzH5UfdEWOViSV7jxXsfuocRv0Rd7bA3O-A4Q0
+ scenes-app-ai-observability-dataset--items--light:
+ hash: v1.k794b7964.1c015595c8578810412d647be5d2024cc10f89550afcad9b5718eb03034d1694.KdlzqK5EDz_RBcSlVhL_3dh0Qw6Q41cWdOqnAeG9U28
+ scenes-app-ai-observability-dataset--new-dataset--dark:
+ hash: v1.k794b7964.7e1118fe1a3df1cbcce546cd7abe512a0263df5554566ed4828d68a5a7a8b474.wXttiynU_OFB8vlU5bXmyrHZIelPEw07Q0Fp61UCa0U
+ scenes-app-ai-observability-dataset--new-dataset--light:
+ hash: v1.k794b7964.224de55acdf166af1f369d618e308efe3a3aa46ae965737dc05a6204b2e5b375.7U2SsbfISxaMVddaKRLuHiWNt9Qj-6eLotsB6JKtaAM
+ scenes-app-ai-observability-dataset--read-only--dark:
+ hash: v1.k794b7964.682c3453692dd77c305690f91a7a77ddbe2de1b323aadea84174cdee39b97239.7UWvSw1xUNRBYr2VWzbKN22wfuhShLBrsxvpywsjbSs
+ scenes-app-ai-observability-dataset--read-only--light:
+ hash: v1.k794b7964.8365628e228326828fe8768e604dde02ea03e30af59047759367ec263cc4d21d.7A820WLz__G-1QESZjLMCWMb6ZJ1e06o20bttjsjnMo
scenes-app-ai-observability-evaluation-explanation--reasoning--dark:
hash: v1.k794b7964.df59798a73898fed2019aaa67ba440611c743d7bab0ff9c282db9eaeec4602dd.RDTleoCWmufVyJ6bte4BtB45JV51kbNvUbRZAa8lvUo
scenes-app-ai-observability-evaluation-explanation--reasoning--light:
@@ -8886,14 +8906,26 @@ snapshots:
hash: v1.k794b7964.501bc38da197ae45e045d236336316a18cb78d533ee3008c678c25afe6f3624a.It6rPCGENocYlEoHpN4OPbTNCP4EOKtWe3DmtpiP8CM
scenes-app-ai-observability-offline-experiments-experiment--request-failure--light:
hash: v1.k794b7964.a31d57c72b94117a24a7a8948a68c6959b4fd00cde6d24aa3cdd206e1ca12eb7.392D8AhKK3JzKggndN4ojaEVt_vsHWraj2vVAWxz5NI
+ scenes-app-ai-observability-offline-experiments-navigation--dataset-detail--dark:
+ hash: v1.k794b7964.b52ff6e7af233221985da0bd9f25d650d605a85f4689e9bdd254c2ddfb625a7b.ar4RTmAEuV5gpBCtbtPCBwaP_kF2-6X2TFbRzU05Vo0
+ scenes-app-ai-observability-offline-experiments-navigation--dataset-detail--light:
+ hash: v1.k794b7964.7b1a2bbdc0ca9f1644bf91d1b6c8bbb6757abc39b8664602d91d2ddf8d92ba71.XlVuI4J64mWHM7DonkiOKD8bxPhcUIyxDQ62Z3NnVcU
+ scenes-app-ai-observability-offline-experiments-navigation--datasets-only--dark:
+ hash: v1.k794b7964.cfe60d1bb96149e5c07f38be4ff3c70dbb41b7e70c230a9ef127b121618326af.f7b7gNJVtuTFiIiGUMpXsbhB9uCrtd_OpC_--TfrTrs
+ scenes-app-ai-observability-offline-experiments-navigation--datasets-only--light:
+ hash: v1.k794b7964.35b96b7629c48614494653876b9e6f09df4d8a94a6857e7bb1542af21c73fd01.DmMuGfWJT-L59Z5Zy_Xl-GFe7Yn0C2w7H2biCvSBmp0
+ scenes-app-ai-observability-offline-experiments-navigation--datasets-unavailable--dark:
+ hash: v1.k794b7964.2a9db18dd5a878696f7b1af18c6a3608da26907d0f5eb65f278b9860d0a29a56.LqQr1nb1bEcpIIvExJLIuIgSGs4Aua4QFigCFIlT-ig
+ scenes-app-ai-observability-offline-experiments-navigation--datasets-unavailable--light:
+ hash: v1.k794b7964.dcfb6dc4d1b0290142c394df9165c1d1bac1972fa1179ebf1d4e1221f5f59354.biHIGb-srQvpY-WtpRgH3alC-AFxpd-J3cJOcgegL5o
scenes-app-ai-observability-offline-experiments-overview--all-time-synchronized-hover--dark:
- hash: v1.k794b7964.fb3ce13091c4c7f3c908e142c2b4e79064192344b38097db465eda6c4cf79a3b.0PHKo_RVtCnJr9EefOBcHXNht8MHl_FoBKHcqd9ol-Y
+ hash: v1.k794b7964.d592fa62f27081449d44b3c03494292be8a043e255da1d70a02b9581221d41a6.hVExLS77L3Jez2RZ2180Yes9ksKzGjOGfoMXsP2mO1w
scenes-app-ai-observability-offline-experiments-overview--all-time-synchronized-hover--light:
- hash: v1.k794b7964.3df907417177586aa0f059463468cbab2be1d8c569fbfc349b0e00cddd7dea63.B9QMiTZkEOtrF1hQIUwQc25TjgyNMLgK3uRlboFL2UI
+ hash: v1.k794b7964.5b8b807f13c80ffd34c7c4ec6e86529e8c3b2079a0aae53f07085c53a47da970.95QtsJLf8OrKvsEnaISXHWtG_It-Q38o1-5fuhvFxRM
scenes-app-ai-observability-offline-experiments-overview--default-scorers--dark:
- hash: v1.k794b7964.fa4c20dac4b64c4e62be12fb9f263082fe0ece58b4c5f3392c6fb19c5e403a59.yw0Qflw3sth1GPGHq6TOq4_fdSvLLHsN30lffaOu4uw
+ hash: v1.k794b7964.73de5e6dff8607fbad95d027b0b2856095e68f28922faa4f06315f6338c8012f.iH_6hZJBzDtbyRJu-cLJduev5kRJYJf7MPCFhd_q4oo
scenes-app-ai-observability-offline-experiments-overview--default-scorers--light:
- hash: v1.k794b7964.fc5e13d18d4567e67d59f1e4e2d330dcecddb178178f77555998dee7bdaac38f.qodNHeE8aFki2ZeCyyGaY_J-Q9P1fo3j2k2n80SoaVk
+ hash: v1.k794b7964.726dc1aa28e3c1b1075ee590b0f24bd8acd6f62efc5bbfad63afbba5d0ba6e0a.u16euuIPiAp98GLWkoqQz4E2i4kGWjiuRZEgpugNcC0
scenes-app-ai-observability-offline-experiments-overview--empty--dark:
hash: v1.k794b7964.977ade824abb5791ed84c5b6d125bc9589d0787b69daea3efeb6d0fe91f45f70.czJW_b0HTFrXWZGbmMZfAs1UD-NkCfVNLy36XZiTH7Y
scenes-app-ai-observability-offline-experiments-overview--empty--light:
@@ -8903,53 +8935,53 @@ snapshots:
scenes-app-ai-observability-offline-experiments-overview--empty-narrow--light:
hash: v1.k794b7964.754079aa9ab7ee416590c9bc8f94fa93f59b7769e98d4e63400883bf4d23ffae.hLRMtJxEnlm36ceer7qs_5s0s0euWyZB0BAqWPZz8c0
scenes-app-ai-observability-offline-experiments-overview--filtered-trends--dark:
- hash: v1.k794b7964.b364f69b2d316e73b525d40ae1d3e7fbae7ee1f9013e869125d8346005736145.ytOrnuZ9UQOyKRmTwQhFrIOFDlDDJn9oUzzsAGKA5Po
+ hash: v1.k794b7964.3c78d5fe30a6ac258d072c3c285ee5f89bf1216f371d8d567207278d6976cd75.Lj5t_bkyrialU7kZJ3OCIREFPSJHuQ7KjxXmoBS5WEI
scenes-app-ai-observability-offline-experiments-overview--filtered-trends--light:
- hash: v1.k794b7964.377ad2181d398e9a57a1d7fac8fc7fb2237fc680c3c7c5563c5973c2d39f34a5.6hIWHqTofDepWwl5hoetAWimrE1eEatrRlEJwxUG_oE
+ hash: v1.k794b7964.812fede378b45377df67ce7a81b6a4f8ebda638209ff66b2f340b4275e3e745e.I6zJjCdwYoP-wj4Ix6ysSL3AXq7Pflfb-9q8KEgi90g
scenes-app-ai-observability-offline-experiments-overview--list-error--dark:
- hash: v1.k794b7964.8b831fef9bfedf72e76e149939a00bf4a76900156748d3bafb93f1d3200f61d6.CUFTGjUs_D9pgPOcndn6bS7R25bmn-9mLEb9b4PDi4o
+ hash: v1.k794b7964.8b08646679e7e9bc99f28b7964580747c49d3499add523991ecd0c37999a5730.DRysc1W_RehJmLU8rb-CBI6FdtvC6LZ4WxiHULuUsbM
scenes-app-ai-observability-offline-experiments-overview--list-error--light:
- hash: v1.k794b7964.fa8e8ecc24f4173a2267281305834cffe9f7eb099785fb88d0830aa165544ed2.j-uksYVVEc6BUX0THSvf1i60NT5xraYlyAP2m8byBgQ
+ hash: v1.k794b7964.f334897ee8fefcc511aa2b2c9110ab86087d6b8635e654e004f87ba36182b3f2.4ARq9cFYhHpXg4hwzj5Qo5HzyM_8Vm37M3RX2wUzxuA
scenes-app-ai-observability-offline-experiments-overview--multiple-versions--dark:
- hash: v1.k794b7964.84a57ee92f9203aa1869bee3548823fb04a7288edec4eaffb8cc294206bb6290.rG4uZItUyrl_vHXhYQY-Ctz_8_jlXYrmALwq89ln5dE
+ hash: v1.k794b7964.e19b98e1146e62df304fb74a78609f238061bf04b0eb5a0ecbdd60c3e08706db.scNCgHa-qolVB6l0ISpAbWQ2OysYbr-aagU7JP4bjhY
scenes-app-ai-observability-offline-experiments-overview--multiple-versions--light:
- hash: v1.k794b7964.ebc934d093a8e63a084fe2d3e5cb9ad3871728bb40672f6cf61539975338ac16.dZzCqNHkhveuOCds4g7xVYzXzoGFMWJ7Dx46IMrxaDY
+ hash: v1.k794b7964.3aa4c8b9363dc26ddc19a2fe796fda8a6bf0f7333e2ff27095de47f0e7504160.jwtwbP8ON-5hP8Bl9TEryP5AuF0GXOL5PL1pSdybbNY
scenes-app-ai-observability-offline-experiments-overview--multiple-versions-narrow--dark:
- hash: v1.k794b7964.030b960d331a7b7bea03eb21353c286d47a6268a333f940f8433ad624942dd28.WNcOt3ZcTgD9RG5goeQrngfLjg4IPg_xrlGQqtbA9P8
+ hash: v1.k794b7964.f8ef3b6bb3ad6d5cc1607c2464b0c24f6092dfdd1a36e5371841f96d0862ffde.phw7DAjJ0YPTz-TlOwBt_tWGhoxQ2Xayx7tpP46HEts
scenes-app-ai-observability-offline-experiments-overview--multiple-versions-narrow--light:
- hash: v1.k794b7964.423be802a7cf08f619cbcb4cb44a8a340f376f8ed37517a6a21b9c51f5f806ee.tvZnyRoE2qTCtCf1uFDX9gX8UYXqIQMGTlySrXoXrv4
+ hash: v1.k794b7964.c95966719f13fabaa65824dff4241e4ff45e3f672cdda558f8856074ee393d30.yLUEYWkQdofVB_u3RiCPns1GWYX0pHmNQ5FV_2S_SiQ
scenes-app-ai-observability-offline-experiments-overview--narrow--dark:
- hash: v1.k794b7964.18ed7d7ec785488c7ab89a9e96c1ccd5b275354f306ecb34ae2fe5e432052176.xM7r2_DGEEvvWion1un3ajeDOXQjNS9SOo3ftnVC4mE
+ hash: v1.k794b7964.076b09003e2f95dc77677a233d6aed6b3cac091eb9dc1082e5c66c3dab103daa.8GpuwmX5FXnNibzd6F9j1JHkzQY18Euc0dBy21UZWIA
scenes-app-ai-observability-offline-experiments-overview--narrow--light:
- hash: v1.k794b7964.1179772dcf2581f883409bea23a188fe79829a2886dd436a602f166534331a1f.2JhsYQ8qLMUOeSuRKMqsSsf2tXBMu77U0qoOHwP7g70
+ hash: v1.k794b7964.cf41bb77c212ae4f3902b07cdd1b427e28dfc35c7ef7d97a13c74284c250f671.poKtRWEEhip6WUeBDRiZOHhk454BV5PqeKR_xKg31yo
scenes-app-ai-observability-offline-experiments-overview--no-matching-experiments--dark:
- hash: v1.k794b7964.1fcbfbaeaadd5301cc28702cbf6c93dc47f2fe56318e9c087303b0af08455def.9P2-61KkzzeyH14B56FzMWousQ3HXkJazoiEBMr6b3A
+ hash: v1.k794b7964.b9a6e968b876bc9cb4c7419a7207a855c02bf59260bce65f3e1d6a4cf43f8162.D-a81APnf9lM8eqv7yefQZe4LFolm_3V2ESmXW4b8js
scenes-app-ai-observability-offline-experiments-overview--no-matching-experiments--light:
- hash: v1.k794b7964.ef2304ac785a197aa675c9e826062246bb19cb244a12047629d26dcbd375a3cc.P9LAmLcHzhobEftRQ30_-UJJGpnLGA9-Z_th_nwd73Q
+ hash: v1.k794b7964.a46726aef52244de1229d9c340e86b1b2dec844f804f0259a3a2355a9d0132b6.rrzNUBXeCapNsY6HZ0_Dty-kR8DEMF3ICGgugfM1cOs
scenes-app-ai-observability-offline-experiments-overview--partial-history--dark:
- hash: v1.k794b7964.dfc8dfdcbc8b02b1593e54be9125973c174ac67cb4c5fde46f119fcbdce70d93.gWtzSWjSRhkQeb5LVEF_qLMpAQGHk4qOXPo6YvN7cNM
+ hash: v1.k794b7964.ed2754c3530dbc22dd79a579206750a9a9ca25e7e02671af70ecfda2d52621cc.q2C2PvHpo2u4er1wViaYpchyoRMXUhuUpftRlIyp9tI
scenes-app-ai-observability-offline-experiments-overview--partial-history--light:
- hash: v1.k794b7964.7d61c4c030e2ac777b8eac0c612e36d23f9cb701bb09764986957c834e8160d3.uOegrRMSXAU7ry1PHSKRdlWrrskTbFcXEnGkaJcgULM
+ hash: v1.k794b7964.302ca88f07d59cdabb3dd8b286b13ff39c03b318a35bca99405683c8e3789d64.-Jd-kXQ46NBmqRsF90-2Y23EI8pzdxvbU9FADQWauYQ
scenes-app-ai-observability-offline-experiments-overview--partial-history-narrow--dark:
- hash: v1.k794b7964.263decd33e83edddaaa509224c0ab69be4cab9d0f1f0bb2aa6e42ec89b685417.sACn0pHOVuNcLWjOXOSdVZXvVOprawYthGBMuGinn3c
+ hash: v1.k794b7964.4c450840d8d01ddb2eb4817e1fea7f7fdbd288e93ea3c375555b4ceb1ae08311.jwIcdD5ZnyQniw64ArTCvFuX43hlRtSDbHAIU6qowFU
scenes-app-ai-observability-offline-experiments-overview--partial-history-narrow--light:
- hash: v1.k794b7964.a02e0465d32c0184cca031c780901fd142ce28b32d792065811caf8eed6c8f21.RU-d8SUARzAYGDeVLBN8Hn1XeabYC0gaLz2BfNN3odU
+ hash: v1.k794b7964.5b69ff2c107ab0450df0d7495d19da2d19293d8fe3b453224845365f8b402f3f.UkyFocgNR0N1T0QWFTOa6HcQYW4BdRMv5_qSzgjjFmI
scenes-app-ai-observability-offline-experiments-overview--recent-experiments--dark:
- hash: v1.k794b7964.fa4c20dac4b64c4e62be12fb9f263082fe0ece58b4c5f3392c6fb19c5e403a59.qSvAMWBW7apGQFrYPcJoNARPkAgjlSg94f8WuUuWek8
+ hash: v1.k794b7964.73de5e6dff8607fbad95d027b0b2856095e68f28922faa4f06315f6338c8012f.YbQpwN5pANDtCZDWgkZ8z9-XfMnI8dB5_HsfibRZ1z0
scenes-app-ai-observability-offline-experiments-overview--recent-experiments--light:
- hash: v1.k794b7964.fc5e13d18d4567e67d59f1e4e2d330dcecddb178178f77555998dee7bdaac38f.ium6vEX51-vJpa_q1nCmYTwF67NJ68RsV-mLLX4AVkE
+ hash: v1.k794b7964.726dc1aa28e3c1b1075ee590b0f24bd8acd6f62efc5bbfad63afbba5d0ba6e0a.wGg0Or9ZpTIpeXaLA386097O4aeCtkFCyB8O4kj-Y6g
scenes-app-ai-observability-offline-experiments-overview--score-error--dark:
- hash: v1.k794b7964.bc28fdc2405851f5b61cfedfa355a27d090fb84a4e496d9f2fc549eaa173435f.S1IV4V9DdF-gbm2AKqw4-4bc9gz3-eVSXk2sMG8Zs9Y
+ hash: v1.k794b7964.689beee8e1c93df52ab1400230008ed48f2828cf75d9992961c81c99816b7fd1._LEb7qSea9grjGiQ8KguiqaT6g270Z8Il2C2ABMqHaI
scenes-app-ai-observability-offline-experiments-overview--score-error--light:
- hash: v1.k794b7964.dece407dd02844758e8950c45573df48c523954726b16521214138d4a10f38d7.LaZL9pX6seylgIWgDtukSLSffO92cydY_RRpFKgEJHw
+ hash: v1.k794b7964.3e158e8aaa911450d7e23670499ae4475a9dc63631c33163a0126867e493983f.nudFomjOot039FZfsD0h2aexX1D7lizpFjF0-mzhF98
scenes-app-ai-observability-offline-experiments-overview--synchronized-hover--dark:
- hash: v1.k794b7964.f82a77c70a10f644911e40efbbd9e5049ab6ddaa708ecdc1c0463b7b554ffb5b.MvIo2T4NRSyRK8PhTkhObMSioTbx_3kXKE_wOCYEk9U
+ hash: v1.k794b7964.668130859c652243deae419e7ed1859d817725f5309c93f05b870cf901aea6a9.SI6bFs6z8YKTICZfTS7AWmzizhfGYyb6TlxfRpzpcfY
scenes-app-ai-observability-offline-experiments-overview--synchronized-hover--light:
- hash: v1.k794b7964.76fa7ae08158f6597e0763661932c7f9ae895dc25805bc304c56e8e45a64f15f.QkSBo1cy4H4qsYFYadwaRIuZtl1k7KgiTku4bgcGtgE
+ hash: v1.k794b7964.6f807533006b80fd5f76474da92184ef23a3732b79550d63253a56bd1bd87d90.lS1pIeWc5KqSo7MjkKRrU4LvjA4xbQDn4Szolkmi1m8
scenes-app-ai-observability-offline-experiments-overview--synchronized-hover-narrow--dark:
- hash: v1.k794b7964.56d79a197d4687fca8bcd4dcda8346f5e0914feb1db5466ec6e302599315c94d.HEivagJC0ow1_ZIhKVgJFkEo8CIv4ssSW4DEHXTqe8M
+ hash: v1.k794b7964.d24b4ebb39da187de88b75cb4fc95646dd62e623baad293435d54393d5f6f500.uGmSH6SyeGMPuhz5VtsXsT7TcZ7e-LYsl_WAaMmgLQs
scenes-app-ai-observability-offline-experiments-overview--synchronized-hover-narrow--light:
- hash: v1.k794b7964.c34bcc78bfa8f503b5cabac0aee2b534ac814a7387219c007ddd24d636b87df3.b-l0k8WJngZByacU9IzDwQDBGU6gXMfFeOLtK9jMwM0
+ hash: v1.k794b7964.deb49ddb9dd172730240051497fe2de1326e28b43dac6280075eb58d8d448dc1.X50r68eZ8nAkCo6w7QqWQs1OINlCknBraztf1U6tuho
scenes-app-ai-observability-promptreferencetags--label-and-version-references--dark:
hash: v1.k794b7964.4cc699d7489063a64cf4ec1f930d11fc062eb0d9c9e46150f939503945bb8841.jKmH2ZmjZex98fImM00HE_xBBayalWKHtqXbVlhE_VM
scenes-app-ai-observability-promptreferencetags--label-and-version-references--light:
diff --git a/frontend/src/lib/components/ObjectTags/ObjectTags.tsx b/frontend/src/lib/components/ObjectTags/ObjectTags.tsx
index 4eb9f86cd809..1a8ab9ea95d0 100644
--- a/frontend/src/lib/components/ObjectTags/ObjectTags.tsx
+++ b/frontend/src/lib/components/ObjectTags/ObjectTags.tsx
@@ -36,6 +36,7 @@ interface ObjectTagsPropsBase {
* room surrounding table columns get.
*/
wrap?: boolean
+ compact?: boolean
editorFullWidth?: boolean
}
@@ -82,6 +83,7 @@ export function ObjectTags({
maxVisibleTags,
showOverflowLabel = false,
wrap = false,
+ compact = false,
editorFullWidth = false,
}: ObjectTagsProps): JSX.Element {
const objectTagId = useId()
@@ -107,7 +109,8 @@ export function ObjectTags({
style={style}
className={clsx(
className,
- 'inline-flex flex-wrap gap-0.5 items-center',
+ 'inline-flex gap-0.5 items-center',
+ compact ? 'flex-nowrap min-w-0' : 'flex-wrap',
wrap && 'min-w-0 max-w-full',
editingTags && editorFullWidth && 'w-full min-w-0'
)}
@@ -141,10 +144,17 @@ export function ObjectTags({
key={index}
type={COLOR_OVERRIDES[tag] || colorForString(tag)}
onClick={onTagClick ? () => onTagClick(tag) : undefined}
- className={wrap ? 'max-w-full' : undefined}
+ size={compact ? 'small' : 'medium'}
+ className={compact ? 'min-w-0' : wrap ? 'max-w-full' : undefined}
wrap={wrap}
>
- {tag}
+ {compact ? (
+ <span className="truncate" title={tag}>
+ {tag}
+ </span>
+ ) : (
+ tag
+ )}
</LemonTag>
)
})}
diff --git a/frontend/src/products.json b/frontend/src/products.json
index 4a059abf6973..a1fb76cb42a8 100644
--- a/frontend/src/products.json
+++ b/frontend/src/products.json
@@ -21,19 +21,12 @@
"type": "llm_clusters",
"intents": ["llm_clusters"]
},
- {
- "path": "Datasets",
- "category": "AI engineering",
- "iconType": null,
- "type": "llm_datasets",
- "intents": ["llm_datasets"]
- },
{
"path": "Evaluations",
"category": "AI engineering",
"iconType": null,
"type": "llm_evaluations",
- "intents": ["llm_evaluations"]
+ "intents": ["llm_evaluations", "llm_datasets"]
},
{
"path": "Taggers",
diff --git a/frontend/src/products.tsx b/frontend/src/products.tsx
index 014fb2544c4c..598e8f784de6 100644
--- a/frontend/src/products.tsx
+++ b/frontend/src/products.tsx
@@ -88,8 +88,7 @@ export const productRoutes: Record<string, [string, string]> = {
'/ai-observability/clusters': ['AIObservabilityClusters', 'aiObservabilityClusters'],
'/ai-observability/clusters/:runId': ['AIObservabilityClusters', 'aiObservabilityClusters'],
[AI_OBSERVABILITY_CLUSTER_URL_PATTERN]: ['AIObservabilityCluster', 'aiObservabilityCluster'],
- '/ai-evals/datasets': ['AIObservabilityDatasets', 'aiObservabilityDatasets'],
- '/ai-evals/datasets/:id': ['AIObservabilityDataset', 'aiObservabilityDataset'],
+ '/ai-evals/evaluations/offline/datasets/:id': ['AIObservabilityDataset', 'aiObservabilityDataset'],
'/ai-evals/taggers': ['AIObservabilityTags', 'aiObservabilityTags'],
'/ai-evals/taggers/:id': ['AIObservabilityTag', 'aiObservabilityTag'],
'/ai-evals/evaluations': ['AIObservabilityEvaluations', 'aiObservabilityEvaluations'],
@@ -411,6 +410,10 @@ export const productRedirects: Record<
},
'/ai-evals': (_params, searchParams, hashParams) =>
combineUrl(urls.aiObservabilityEvaluations(), searchParams, hashParams).url,
+ '/ai-evals/datasets': (_params, searchParams, hashParams) =>
+ combineUrl(urls.aiObservabilityDatasets(), { ...searchParams, tab: 'datasets' }, hashParams).url,
+ '/ai-evals/datasets/:id': (params, searchParams, hashParams) =>
+ combineUrl(urls.aiObservabilityDataset(params.id), searchParams, hashParams).url,
'/ai-evals/evaluations/offline': (_params, searchParams, hashParams) =>
combineUrl(urls.aiObservabilityOfflineEvaluations(), searchParams, hashParams).url,
'/ai-evals/tags': (_params, searchParams, hashParams) =>
@@ -468,7 +471,7 @@ export const productRedirects: Record<
'/llm-analytics/clusters/:runId/:clusterId': (params, searchParams, hashParams) =>
combineUrl(urls.aiObservabilityCluster(params.runId, params.clusterId), searchParams, hashParams).url,
'/llm-analytics/datasets': (_params, searchParams, hashParams) =>
- combineUrl(urls.aiObservabilityDatasets(), searchParams, hashParams).url,
+ combineUrl(urls.aiObservabilityDatasets(), { ...searchParams, tab: 'datasets' }, hashParams).url,
'/llm-analytics/datasets/:id': (params, searchParams, hashParams) =>
combineUrl(urls.aiObservabilityDataset(params.id), searchParams, hashParams).url,
'/llm-analytics/tags': (_params, searchParams, hashParams) =>
@@ -1379,13 +1382,13 @@ export const productUrls = {
return `/ai-observability/sessions/${id}${stringifiedParams ? `?${stringifiedParams}` : ''}`
},
aiObservabilityPlayground: (): string => '/ai-observability/playground',
- aiObservabilityDatasets: (): string => '/ai-evals/datasets',
+ aiObservabilityDatasets: (): string => '/ai-evals/evaluations/offline/experiments?tab=datasets',
aiObservabilityDataset: (
id: string,
params?: {
item?: string
}
- ): string => combineUrl(`/ai-evals/datasets/${id}`, params).url,
+ ): string => combineUrl(`/ai-evals/evaluations/offline/datasets/${id}`, params).url,
aiObservabilityTags: (): string => '/ai-evals/taggers',
aiObservabilityTag: (id: string): string => `/ai-evals/taggers/${id}`,
aiObservabilityEvaluations: (): string => '/ai-evals/evaluations',
@@ -2338,7 +2341,6 @@ export type ProductTreePath =
| 'Dashboards'
| 'Data catalog'
| 'Data warehouse'
- | 'Datasets'
| 'Early access features'
| 'Endpoints'
| 'Engineering analytics'
@@ -2604,42 +2606,6 @@ export const getTreeItemsProducts = (): FileSystemImport[] => [
'WarehouseDestinations',
],
},
- {
- path: 'Datasets',
- intents: [ProductKey.LLM_DATASETS],
- category: ProductItemCategory.AI_ENGINEERING,
- type: 'llm_datasets',
- iconType: 'llm_datasets' as FileSystemIconType,
- iconColor: ['var(--color-product-llm-datasets-light)', 'var(--color-product-llm-datasets-dark)'],
- href: urls.aiObservabilityDatasets(),
- searchKeywords: ['test sets', 'golden set'],
- flag: FEATURE_FLAGS.LLM_ANALYTICS_DATASETS,
- tags: ['beta'],
- sceneKey: 'AIObservabilityDatasets',
- sceneKeys: [
- 'AIObservability',
- 'AIObservabilityTrace',
- 'AIObservabilitySession',
- 'AIObservabilityUsers',
- 'AIObservabilityPlayground',
- 'AIObservabilityDatasets',
- 'AIObservabilityDataset',
- 'AIObservabilityEvaluations',
- 'AIObservabilityScorers',
- 'AIObservabilityScorer',
- 'AIObservabilityOfflineExperiments',
- 'AIObservabilityOfflineExperiment',
- 'AIObservabilityOfflineScorerHistory',
- 'AIObservabilityEvaluation',
- 'AIObservabilityEvaluationTemplates',
- 'AIObservabilityTags',
- 'AIObservabilityTag',
- 'AIObservabilityPrompts',
- 'AIObservabilityPrompt',
- 'AIObservabilityClusters',
- 'AIObservabilityCluster',
- ],
- },
{
path: 'ETL',
displayLabel: 'ELT',
@@ -2735,19 +2701,20 @@ export const getTreeItemsProducts = (): FileSystemImport[] => [
},
{
path: 'Evaluations',
- intents: [ProductKey.LLM_EVALUATIONS],
+ intents: [ProductKey.LLM_EVALUATIONS, ProductKey.LLM_DATASETS],
category: ProductItemCategory.AI_ENGINEERING,
type: 'llm_evaluations',
iconType: 'llm_evaluations' as FileSystemIconType,
iconColor: ['var(--color-product-llm-evaluations-light)', 'var(--color-product-llm-evaluations-dark)'],
href: urls.aiObservabilityEvaluations(),
- searchKeywords: ['llm judge', 'llm as a judge', 'grading'],
+ searchKeywords: ['llm judge', 'llm as a judge', 'grading', 'datasets', 'test sets', 'golden set'],
searchTabs: [
{
name: 'Offline evals',
href: urls.aiObservabilityOfflineEvaluations(),
flag: FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS,
},
+ { name: 'Datasets', href: urls.aiObservabilityDatasets(), flag: FEATURE_FLAGS.LLM_ANALYTICS_DATASETS },
{ name: 'Scorers', href: urls.aiObservabilityScorers() },
],
sceneKey: 'AIObservabilityEvaluations',
diff --git a/frontend/src/scenes/sceneLogic.test.tsx b/frontend/src/scenes/sceneLogic.test.tsx
index 55553757981a..55d22e6623e7 100644
--- a/frontend/src/scenes/sceneLogic.test.tsx
+++ b/frontend/src/scenes/sceneLogic.test.tsx
@@ -67,6 +67,7 @@ const testScenes: Record<string, () => any> = {
[Scene.AgenticAuthorize]: sceneImport,
[Scene.Alerts]: sceneImport,
[Scene.AIObservabilityEvaluations]: sceneImport,
+ [Scene.AIObservabilityOfflineExperiments]: sceneImport,
[Scene.Billing]: sceneImport,
[Scene.DataManagement]: sceneImport,
[Scene.OrganizationCreateFirst]: sceneImport,
@@ -464,9 +465,11 @@ describe('sceneLogic', () => {
})
it.each([
- [AccessControlLevel.Viewer, Scene.AIObservabilityEvaluations],
- [AccessControlLevel.None, Scene.ErrorAccessDenied],
- ])('gates the combined evaluations entry with scorer access %s', async (scorerAccess, expectedScene) => {
+ [Scene.AIObservabilityEvaluations, AccessControlLevel.Viewer, Scene.AIObservabilityEvaluations],
+ [Scene.AIObservabilityEvaluations, AccessControlLevel.None, Scene.ErrorAccessDenied],
+ [Scene.AIObservabilityOfflineExperiments, AccessControlLevel.Viewer, Scene.AIObservabilityOfflineExperiments],
+ [Scene.AIObservabilityOfflineExperiments, AccessControlLevel.None, Scene.ErrorAccessDenied],
+ ])('gates %s with scorer and dataset access %s', async (targetScene, scorerAccess, expectedScene) => {
const priorAppContext = window.POSTHOG_APP_CONTEXT
try {
window.POSTHOG_APP_CONTEXT = {
@@ -477,7 +480,7 @@ describe('sceneLogic', () => {
[AccessControlResourceType.LlmAnalytics]: scorerAccess,
},
} as AppContext
- logic.actions.setScene(Scene.AIObservabilityEvaluations, 'aiObservabilityEvaluations', {
+ logic.actions.setScene(targetScene as Scene, targetScene, {
params: {},
searchParams: {},
hashParams: {},
diff --git a/frontend/src/scenes/sceneTypes.ts b/frontend/src/scenes/sceneTypes.ts
index 15e333fb1897..e211d49a48f8 100644
--- a/frontend/src/scenes/sceneTypes.ts
+++ b/frontend/src/scenes/sceneTypes.ts
@@ -487,7 +487,10 @@ export const sceneToAccessControlResourceType: Partial<
[Scene.AIObservabilityEvaluations]: [AccessControlResourceType.Evaluation, AccessControlResourceType.LlmAnalytics],
[Scene.AIObservabilityScorers]: AccessControlResourceType.LlmAnalytics,
[Scene.AIObservabilityScorer]: AccessControlResourceType.LlmAnalytics,
- [Scene.AIObservabilityOfflineExperiments]: AccessControlResourceType.Evaluation,
+ [Scene.AIObservabilityOfflineExperiments]: [
+ AccessControlResourceType.Evaluation,
+ AccessControlResourceType.LlmAnalytics,
+ ],
[Scene.AIObservabilityOfflineExperiment]: AccessControlResourceType.Evaluation,
[Scene.AIObservabilityOfflineScorerHistory]: AccessControlResourceType.Evaluation,
[Scene.AIObservabilityEvaluationTemplates]: AccessControlResourceType.Evaluation,
diff --git a/frontend/src/styles/base.scss b/frontend/src/styles/base.scss
index f9f0d449a9c3..c1024f82aeac 100644
--- a/frontend/src/styles/base.scss
+++ b/frontend/src/styles/base.scss
@@ -348,13 +348,13 @@
--color-product-data-warehouse-dark: rgb(160 135 255);
--color-product-data-catalog-light: rgb(101 78 196);
--color-product-data-catalog-dark: rgb(139 118 232);
- --color-product-llm-analytics-light: rgb(182 42 217);
- --color-product-llm-analytics-dark: rgb(212 106 240);
- --color-product-llm-clusters-light: rgb(108 93 238);
+ --color-product-llm-analytics-light: rgb(99 102 241);
+ --color-product-llm-analytics-dark: rgb(129 140 248);
+ --color-product-llm-clusters-light: rgb(110 92 238);
--color-product-llm-datasets-light: rgb(121 87 235);
- --color-product-llm-evaluations-light: rgb(135 81 232);
- --color-product-mcp-analytics-light: rgb(150 76 229);
- --color-product-mcp-analytics-dark: rgb(181 124 249);
+ --color-product-llm-evaluations-light: rgb(126 85 234);
+ --color-product-mcp-analytics-light: rgb(142 78 231);
+ --color-product-mcp-analytics-dark: rgb(173 125 248);
--color-product-notebooks-light: rgb(239 68 68);
--color-product-notebooks-dark: rgb(252 165 165);
--color-product-tracing-light: rgb(6 182 212);
@@ -383,13 +383,13 @@
--color-product-annotations-dark: rgb(94 235 238);
--color-product-warehouse-destinations-dark: rgb(249 115 22);
--color-product-warehouse-destinations-light: rgb(194 65 12);
- --color-product-llm-clusters-dark: rgb(133 128 248);
+ --color-product-llm-clusters-dark: rgb(136 128 248);
--color-product-llm-datasets-dark: rgb(149 127 248);
- --color-product-llm-evaluations-dark: rgb(165 126 248);
- --color-product-llm-playground-light: rgb(165 70 226);
- --color-product-llm-playground-dark: rgb(198 123 249);
- --color-product-llm-prompts-light: rgb(180 65 222);
- --color-product-llm-prompts-dark: rgb(215 122 249);
+ --color-product-llm-evaluations-dark: rgb(154 126 248);
+ --color-product-llm-playground-light: rgb(160 72 227);
+ --color-product-llm-playground-dark: rgb(192 124 249);
+ --color-product-llm-prompts-light: rgb(178 66 222);
+ --color-product-llm-prompts-dark: rgb(212 122 249);
--color-product-llm-tags-light: rgb(196 60 218);
--color-product-llm-tags-dark: rgb(232 121 249);
--color-product-event-definitions-light: rgb(24 145 200);
diff --git a/products/ai_observability/frontend/aiObservabilityLogic.test.ts b/products/ai_observability/frontend/aiObservabilityLogic.test.ts
index e2749eba070e..9251e2b558b1 100644
--- a/products/ai_observability/frontend/aiObservabilityLogic.test.ts
+++ b/products/ai_observability/frontend/aiObservabilityLogic.test.ts
@@ -11,7 +11,7 @@ import { emptySceneParams } from 'scenes/scenes'
import { Scene } from 'scenes/sceneTypes'
import { urls } from 'scenes/urls'
-import { productRedirects } from '~/products'
+import { productRedirects, productRoutes } from '~/products'
import { isEventsQuery, isTracesQuery } from '~/queries/utils'
import { initKeaTests } from '~/test/init'
import { PropertyFilterType, PropertyOperator } from '~/types'
@@ -43,7 +43,7 @@ describe('LLM analytics URL split', () => {
expect(urls.aiObservabilitySelfDriving()).toBe('/ai-observability/self-driving')
expect(urls.aiObservabilityReviews()).toBe('/ai-observability/reviews')
expect(urls.aiObservabilityTrace('trace-1')).toBe('/ai-observability/traces/trace-1')
- expect(urls.aiObservabilityDatasets()).toBe('/ai-evals/datasets')
+ expect(urls.aiObservabilityDatasets()).toBe('/ai-evals/evaluations/offline/experiments?tab=datasets')
expect(urls.aiObservabilityTags()).toBe('/ai-evals/taggers')
expect(urls.aiObservabilityEvaluations()).toBe('/ai-evals/evaluations')
expect(urls.aiObservabilityScorers()).toBe('/ai-evals/evaluations/scorers')
@@ -63,7 +63,7 @@ describe('LLM analytics URL split', () => {
'/ai-observability/traces/trace-1?event=event-1'
)
expect(redirectUrl('/llm-analytics/datasets/:id', { id: 'dataset-1' }, { item: 'item-1' })).toBe(
- '/ai-evals/datasets/dataset-1?item=item-1'
+ '/ai-evals/evaluations/offline/datasets/dataset-1?item=item-1'
)
expect(redirectUrl('/llm-analytics/tags/:id', { id: 'tagger-1' })).toBe('/ai-evals/taggers/tagger-1')
expect(redirectUrl('/llm-analytics/evaluations/:id', { id: 'evaluation-1' })).toBe(
@@ -95,6 +95,60 @@ describe('LLM analytics URL split', () => {
}
)
+ it.each([
+ ['/ai-evals/datasets/:id', 'new'],
+ ['/ai-evals/datasets/:id', 'dataset-1'],
+ ['/llm-analytics/datasets/:id', 'new'],
+ ['/llm-analytics/datasets/:id', 'dataset-1'],
+ ])('redirects dataset creation and detail bookmarks from %s (%s)', (path, id) => {
+ const searchParams = {
+ tab: 'metadata',
+ item: 'item-1',
+ revision: '3',
+ item_status: 'archived',
+ datasets_page: '2',
+ search: 'quality',
+ }
+ const redirected = new URL(redirectUrl(path, { id }, searchParams, { section: 'items' }), 'https://example.com')
+ expect(redirected.pathname).toBe(`/ai-evals/evaluations/offline/datasets/${id}`)
+ expect(Object.fromEntries(redirected.searchParams)).toEqual(searchParams)
+ expect(redirected.hash).toBe('#section=items')
+ expect(productRoutes['/ai-evals/evaluations/offline/datasets/:id']).toEqual([
+ 'AIObservabilityDataset',
+ 'aiObservabilityDataset',
+ ])
+ })
+
+ it.each(['/ai-evals/datasets', '/llm-analytics/datasets'])(
+ 'redirects dataset bookmarks with their filters and hash: %s',
+ (path) => {
+ const redirected = new URL(
+ redirectUrl(
+ path,
+ {},
+ {
+ tab: 'metadata',
+ search: 'quality',
+ dataset_status: 'archived',
+ datasets_page: '2',
+ tags: 'regression',
+ },
+ { section: 'list' }
+ ),
+ 'https://example.com'
+ )
+ expect(redirected.pathname).toBe(urls.aiObservabilityOfflineEvaluations())
+ expect(Object.fromEntries(redirected.searchParams)).toEqual({
+ tab: 'datasets',
+ search: 'quality',
+ dataset_status: 'archived',
+ datasets_page: '2',
+ tags: 'regression',
+ })
+ expect(redirected.hash).toBe('#section=list')
+ }
+ )
+
it('redirects AI observability settings to the project-level BYOK setting', () => {
expect(redirectUrl('/ai-observability/settings')).toBe(
'/settings/project-ai-observability#ai-observability-byok'
diff --git a/products/ai_observability/frontend/components/ResourceTagFilter.tsx b/products/ai_observability/frontend/components/ResourceTagFilter.tsx
index 835f25c954ca..1c8f50ca0588 100644
--- a/products/ai_observability/frontend/components/ResourceTagFilter.tsx
+++ b/products/ai_observability/frontend/components/ResourceTagFilter.tsx
@@ -8,15 +8,17 @@ import type { ResourceTagFilters } from './resourceTagFilters'
export function ResourceTagFilter({
filters,
onChange,
+ size = 'medium',
}: {
filters: ResourceTagFilters
onChange: (filters: ResourceTagFilters) => void
+ size?: 'small' | 'medium'
}): JSX.Element {
return (
<div className="flex flex-wrap items-center gap-2">
<TagSelect
defaultLabel="Tags"
- size="medium"
+ size={size}
value={filters.tags}
onChange={(tags) => {
if (tags.length > 20) {
@@ -29,6 +31,7 @@ export function ResourceTagFilter({
/>
{filters.tags.length > 1 && (
<LemonSelect
+ size={size}
value={filters.tag_match}
onChange={(tag_match) => onChange({ tags: filters.tags, tag_match })}
options={[
diff --git a/products/ai_observability/frontend/components/ResourceTags.tsx b/products/ai_observability/frontend/components/ResourceTags.tsx
index f204e5187f7f..74a1578bc7d1 100644
--- a/products/ai_observability/frontend/components/ResourceTags.tsx
+++ b/products/ai_observability/frontend/components/ResourceTags.tsx
@@ -21,6 +21,7 @@ export function ResourceTags({
onTagClick,
userAccessLevel,
disabledReason,
+ compact = false,
}: {
entity: TaggedResource
resourceId: string
@@ -29,18 +30,20 @@ export function ResourceTags({
onTagClick: (tag: string) => void
userAccessLevel?: AccessControlLevel
disabledReason?: string
+ compact?: boolean
}): JSX.Element {
const { currentTeamId } = useValues(teamLogic)
const logic = resourceTagsLogic({ entity, resourceId, tags, onSaved, teamId: currentTeamId ?? 0 })
const { isOpen } = useValues(logic)
const { open } = useActions(logic)
return (
- <div className="flex items-start gap-1 min-w-0 max-w-64">
+ <div className="flex items-center gap-1 min-w-0 max-w-64">
<ObjectTags
tags={[...tags]}
staticOnly
- maxVisibleTags={2}
- wrap
+ maxVisibleTags={compact ? 1 : 2}
+ compact={compact}
+ wrap={!compact}
onTagClick={onTagClick}
data-attr="aio-resource-tags"
/>
diff --git a/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.stories.tsx b/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.stories.tsx
new file mode 100644
index 000000000000..709afd51fa93
--- /dev/null
+++ b/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.stories.tsx
@@ -0,0 +1,119 @@
+import type { Meta, StoryObj } from '@storybook/react'
+import { waitFor, within } from '@testing-library/dom'
+import { BindLogic } from 'kea'
+import { combineUrl } from 'kea-router'
+
+import { FEATURE_FLAGS } from 'lib/constants'
+import { urls } from 'scenes/urls'
+
+import { mswDecorator } from '~/mocks/browser'
+
+import type { DatasetItemReadApi, DatasetReadApi } from '../generated/api.schemas'
+import { aiObservabilityDatasetLogic } from './aiObservabilityDatasetLogic'
+import { AIObservabilityDatasetScene } from './AIObservabilityDatasetScene'
+
+const dataset: DatasetReadApi = {
+ id: 'example-dataset',
+ name: 'Answer quality',
+ description: 'Questions and expected answers for evaluating the support assistant.',
+ metadata: { owner: 'support', language: 'en' },
+ tags: [],
+ archived: false,
+ current_revision: 1,
+ current_revision_id: 'example-revision',
+ created_at: '2024-01-01T00:00:00Z',
+ updated_at: '2024-01-01T00:00:00Z',
+ created_by: null,
+ team_id: 997,
+ user_access_level: 'editor',
+}
+
+const item: DatasetItemReadApi = {
+ id: 'example-item',
+ dataset: dataset.id,
+ client_item_id: null,
+ version: 1,
+ version_id: 'example-item-version',
+ dataset_revision: 1,
+ dataset_revision_id: 'example-revision',
+ archived: false,
+ input: { question: 'How do I reset my password?' },
+ expected_output: { answer: 'Use the password reset link on the sign-in page.' },
+ source_output: null,
+ metadata: { category: 'account' },
+ source_trace_id: null,
+ source_timestamp: null,
+ source_event_id: null,
+ created_by: null,
+ created_at: dataset.created_at,
+ updated_at: dataset.created_at,
+ version_created_at: dataset.created_at,
+ version_created_by: null,
+ team_id: 997,
+}
+
+const meta: Meta<typeof AIObservabilityDatasetScene> = {
+ title: 'Scenes-App/AI observability/Dataset',
+ component: AIObservabilityDatasetScene,
+ parameters: {
+ mockDate: '2024-01-02T12:00:00Z',
+ pageUrl: urls.aiObservabilityDataset(dataset.id),
+ featureFlags: [FEATURE_FLAGS.LLM_ANALYTICS_DATASETS, FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS],
+ },
+ decorators: [
+ mswDecorator({
+ get: {
+ '/api/projects/:team/datasets/:id/': dataset,
+ '/api/projects/:team/datasets/:id/revisions/': { count: 0, results: [], next: null, previous: null },
+ '/api/projects/:team/dataset_items/': { count: 1, results: [item], next: null, previous: null },
+ },
+ }),
+ (Story, { parameters }) => (
+ <BindLogic logic={aiObservabilityDatasetLogic} props={{ datasetId: parameters.datasetId || dataset.id }}>
+ <Story />
+ </BindLogic>
+ ),
+ ],
+ play: async ({ canvasElement }) => {
+ await within(canvasElement).findByText(item.id)
+ },
+}
+export default meta
+type Story = StoryObj<typeof AIObservabilityDatasetScene>
+
+export const Items: Story = {}
+export const Details: Story = {
+ parameters: { pageUrl: combineUrl(urls.aiObservabilityDataset(dataset.id), { tab: 'metadata' }).url },
+}
+export const NewDataset: Story = {
+ parameters: { datasetId: 'new', pageUrl: urls.aiObservabilityDataset('new') },
+ play: async ({ canvasElement }) => {
+ await waitFor(() => {
+ if (!canvasElement.querySelector('.monaco-editor')) {
+ throw new Error('Waiting for the dataset metadata editor')
+ }
+ })
+ },
+}
+export const DatasetsOnlyNarrow: Story = {
+ parameters: {
+ featureFlags: {
+ [FEATURE_FLAGS.LLM_ANALYTICS_DATASETS]: true,
+ [FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS]: false,
+ },
+ },
+ decorators: [
+ (Story) => (
+ <div className="w-[520px]">
+ <Story />
+ </div>
+ ),
+ ],
+}
+export const ReadOnly: Story = {
+ decorators: [
+ mswDecorator({
+ get: { '/api/projects/:team/datasets/:id/': { ...dataset, user_access_level: 'viewer' } },
+ }),
+ ],
+}
diff --git a/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.tsx b/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.tsx
index be08ebc1aecd..45b787a53cea 100644
--- a/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.tsx
+++ b/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.tsx
@@ -1,49 +1,31 @@
import { useActions, useValues } from 'kea'
import { Form } from 'kea-forms'
-import { combineUrl, router } from 'kea-router'
+import { router } from 'kea-router'
import { useEffect } from 'react'
-import { IconArchive, IconDownload, IconPlusSmall } from '@posthog/icons'
-import {
- LemonBanner,
- LemonButton,
- LemonDivider,
- LemonSegmentedButton,
- LemonSelect,
- LemonTab,
- LemonTabs,
- LemonTag,
- Link,
- ProfilePicture,
-} from '@posthog/lemon-ui'
+import { IconArchive, IconArrowLeft, IconDownload, IconInfo } from '@posthog/icons'
+import { LemonBanner, LemonButton, LemonTag } from '@posthog/lemon-ui'
import { Button } from '@posthog/quill'
import { AccessControlAction } from 'lib/components/AccessControlAction'
import { AccessDenied } from 'lib/components/AccessDenied'
-import { HighlightedJSONViewer } from 'lib/components/HighlightedJSONViewer'
import { NotFound } from 'lib/components/NotFound'
import { SceneMenuBarFileItems } from 'lib/components/Scenes/SceneMenuBarFileItems'
import { FEATURE_FLAGS } from 'lib/constants'
import { useFeatureFlag } from 'lib/hooks/useFeatureFlag'
-import { More } from 'lib/lemon-ui/LemonButton/More'
import { LemonDialog } from 'lib/lemon-ui/LemonDialog'
import { LemonSkeleton } from 'lib/lemon-ui/LemonSkeleton'
-import { createdAtColumn, updatedAtColumn } from 'lib/lemon-ui/LemonTable/columnUtils'
import { featureFlagLogic } from 'lib/logic/featureFlagLogic'
import { ButtonPrimitive } from 'lib/ui/Button/ButtonPrimitives'
import { toAccessControlLevel } from 'lib/utils/accessControlUtils'
-import { isObject } from 'lib/utils/guards'
import { SceneExport } from 'scenes/sceneTypes'
-import { urls } from 'scenes/urls'
-import { SceneContent } from '~/layout/scenes/components/SceneContent'
import {
SceneMenuBar,
SceneMenuBarItem,
SceneMenuBarMenu,
SceneMenuBarSeparator,
} from '~/layout/scenes/components/SceneMenuBar'
-import { SceneTitleSection } from '~/layout/scenes/components/SceneTitleSection'
import {
ScenePanel,
ScenePanelActionsSection,
@@ -52,24 +34,15 @@ import {
} from '~/layout/scenes/SceneLayout'
import { SceneTextarea } from '~/lib/components/Scenes/SceneTextarea'
import { SceneTextInput } from '~/lib/components/Scenes/SceneTextInput'
-import { LemonTable, LemonTableColumn, LemonTableColumns } from '~/lib/lemon-ui/LemonTable'
import { ProductKey } from '~/queries/schema/schema-general'
-import { AccessControlLevel, AccessControlResourceType, type UserBasicType } from '~/types'
+import { AccessControlLevel, AccessControlResourceType } from '~/types'
-import type { DatasetItemReadApi as DatasetItem, DatasetReadApi as Dataset } from '../generated/api.schemas'
-import { truncateValue } from '../utils'
-import {
- DATASET_ITEM_VERSIONS_PER_PAGE,
- DatasetLogicProps,
- DatasetTab,
- isDataset,
- aiObservabilityDatasetLogic,
-} from './aiObservabilityDatasetLogic'
+import { OfflineEvaluationsScene } from '../offline-evaluations/OfflineEvaluationsScene'
+import { OfflineEvaluationsTabs } from '../offline-evaluations/OfflineEvaluationsTabs'
+import { DatasetLogicProps, DatasetTab, isDataset, aiObservabilityDatasetLogic } from './aiObservabilityDatasetLogic'
import { getDatasetListUrl } from './aiObservabilityDatasetsLogic'
-import { DatasetItemModal } from './DatasetItemModal'
+import { DatasetContents } from './DatasetContents'
import { EditDatasetForm } from './EditDatasetForm'
-import { JSONColumn } from './JSONColumn'
-import { RefreshButton } from './RefreshButton'
const RESOURCE_TYPE = 'dataset'
@@ -110,6 +83,7 @@ export function AIObservabilityDatasetScene(): JSX.Element {
exportDataset,
setDatasetFormValue,
onUnmount,
+ setActiveTab,
} = useActions(aiObservabilityDatasetLogic)
const { searchParams } = useValues(router)
const { featureFlags } = useValues(featureFlagLogic)
@@ -156,113 +130,52 @@ export function AIObservabilityDatasetScene(): JSX.Element {
useEffect(() => () => onUnmount(), [onUnmount])
if (isDatasetMissing) {
- return <NotFound object="dataset" />
+ return (
+ <OfflineEvaluationsScene>
+ <NotFound object="dataset" />
+ </OfflineEvaluationsScene>
+ )
}
if (datasetLoadError?.status === 403) {
- return <AccessDenied object="dataset" />
+ return (
+ <OfflineEvaluationsScene>
+ <AccessDenied object="dataset" />
+ </OfflineEvaluationsScene>
+ )
}
if (datasetLoadError && !isDataset(dataset)) {
return (
- <SceneContent>
+ <OfflineEvaluationsScene>
<LemonBanner type="error" action={{ children: 'Try again', onClick: loadDataset }}>
{datasetLoadError.detail || "Couldn't load this dataset. Try again."}
</LemonBanner>
- </SceneContent>
+ </OfflineEvaluationsScene>
)
}
if (shouldDisplaySkeleton) {
return (
- <div className="flex flex-col gap-2">
- <LemonSkeleton active className="h-4 w-2/5" />
- <LemonSkeleton active className="h-4 w-full" />
- <LemonSkeleton active className="h-4 w-full" />
- <LemonSkeleton active className="h-4 w-3/5" />
- </div>
+ <OfflineEvaluationsScene>
+ <div className="flex flex-col gap-2">
+ <LemonSkeleton active className="h-4 w-2/5" />
+ <LemonSkeleton active className="h-4 w-full" />
+ <LemonSkeleton active className="h-4 w-full" />
+ <LemonSkeleton active className="h-4 w-3/5" />
+ </div>
+ </OfflineEvaluationsScene>
)
}
return (
<Form id="dataset-form" formKey="datasetForm" logic={aiObservabilityDatasetLogic}>
- <SceneContent>
+ <OfflineEvaluationsScene>
{datasetLoadError && (
<LemonBanner type="error" action={{ children: 'Try again', onClick: loadDataset }}>
{datasetLoadError.detail || "Couldn't refresh this dataset. Try again."}
</LemonBanner>
)}
- <SceneTitleSection
- name={datasetForm.name}
- resourceType={{ type: 'llm_analytics' }}
- isLoading={datasetLoading}
- actions={
- !shouldDisplaySkeleton && displayEditForm ? (
- todayRailNav ? (
- <>
- <Button
- variant="outline"
- data-attr="cancel-dataset"
- onClick={cancelEditing}
- disabledReason={isDatasetFormSubmitting ? 'Saving…' : undefined}
- >
- Cancel
- </Button>
- <AccessControlAction
- resourceType={AccessControlResourceType.LlmAnalytics}
- minAccessLevel={AccessControlLevel.Editor}
- userAccessLevel={
- isDataset(dataset)
- ? toAccessControlLevel(dataset.user_access_level)
- : undefined
- }
- >
- <Button
- variant="primary"
- data-attr="save-dataset"
- onClick={submitDatasetForm}
- loading={isDatasetFormSubmitting}
- >
- {isNewDataset ? 'Create dataset' : 'Save'}
- </Button>
- </AccessControlAction>
- </>
- ) : (
- <>
- <LemonButton
- type="secondary"
- data-attr="cancel-dataset"
- onClick={cancelEditing}
- disabledReason={isDatasetFormSubmitting ? 'Saving…' : undefined}
- size="small"
- >
- Cancel
- </LemonButton>
- <AccessControlAction
- resourceType={AccessControlResourceType.LlmAnalytics}
- minAccessLevel={AccessControlLevel.Editor}
- userAccessLevel={
- isDataset(dataset)
- ? toAccessControlLevel(dataset.user_access_level)
- : undefined
- }
- >
- <LemonButton
- type="primary"
- data-attr="save-dataset"
- onClick={submitDatasetForm}
- loading={isDatasetFormSubmitting}
- size="small"
- >
- {isNewDataset ? 'Create dataset' : 'Save'}
- </LemonButton>
- </AccessControlAction>
- </>
- )
- ) : undefined
- }
- />
-
{isDataset(dataset) && sceneMenuBarEnabled && (
<SceneMenuBar>
<SceneMenuBarMenu label="File" dataAttr={`${RESOURCE_TYPE}-menubar-file`}>
@@ -345,455 +258,127 @@ export function AIObservabilityDatasetScene(): JSX.Element {
</ScenePanel>
)}
- {displayEditForm ? <EditDatasetForm /> : isDataset(dataset) ? <DatasetTabs dataset={dataset} /> : null}
- </SceneContent>
- </Form>
- )
-}
-
-function DatasetTabs({ dataset }: { dataset: Dataset }): JSX.Element {
- const {
- activeTab,
- isDatasetItemModalOpen,
- selectedDatasetItem,
- selectedDatasetItemLoadError,
- selectedDatasetItemDetailsLoading,
- selectedDatasetItemVersions,
- datasetItemVersionsLoading,
- datasetItemVersionsLoadError,
- datasetItemVersionsPage,
- restoringDatasetItemVersion,
- archivingDatasetItemId,
- canEditDataset,
- canManageDataset,
- isHistoricalRevision,
- filters,
- datasetExportLoading,
- datasetExportLoadError,
- datasetExportDisabledReason,
- } = useValues(aiObservabilityDatasetLogic)
- const {
- closeModalAndRefetchDatasetItems,
- loadDatasetItemDetails,
- restoreDatasetItem,
- restoreDatasetItemVersion,
- loadDatasetItemVersions,
- setFilters,
- exportDataset,
- editDataset,
- } = useActions(aiObservabilityDatasetLogic)
- const { searchParams } = useValues(router)
- const selectedItemReadOnly = !canEditDataset || !!selectedDatasetItem?.archived
- const selectedItemReadOnlyReason = isHistoricalRevision
- ? `You're viewing revision ${filters.revision}. Select Latest to edit items.`
- : dataset.archived
- ? 'This dataset is archived. Unarchive it to edit items.'
- : selectedDatasetItem?.archived
- ? 'This item is archived. Unarchive it to edit.'
- : !canManageDataset
- ? 'You have view-only access to this dataset.'
- : undefined
-
- const tabs: LemonTab<DatasetTab>[] = [
- {
- key: DatasetTab.Items,
- label: 'Items',
- content: <DatasetItems dataset={dataset} />,
- link: combineUrl(urls.aiObservabilityDataset(dataset.id), { ...searchParams, tab: DatasetTab.Items }).url,
- },
- {
- key: DatasetTab.Metadata,
- label: 'Metadata',
- content: <DatasetMetadata dataset={dataset} />,
- link: combineUrl(urls.aiObservabilityDataset(dataset.id), { ...searchParams, tab: DatasetTab.Metadata })
- .url,
- },
- ]
-
- return (
- <>
- <div className="flex flex-col gap-2">
- <div className="flex items-center justify-between gap-2">
- <div className="flex items-center gap-2">
- <LemonTag type={dataset.archived ? 'muted' : 'success'}>
- {dataset.archived ? 'Archived' : 'Active'}
- </LemonTag>
- {isHistoricalRevision && <LemonTag type="default">Revision {filters.revision}</LemonTag>}
+ <div className="flex flex-wrap items-center gap-3">
+ <OfflineEvaluationsTabs activeTab="datasets" />
+ <div className="flex flex-1 basis-80 flex-wrap items-center gap-2">
+ <nav aria-label="Dataset" className="flex min-w-0 flex-1 basis-40 items-center gap-2">
+ <LemonButton
+ size="small"
+ icon={<IconArrowLeft />}
+ aria-label="Back to datasets"
+ tooltip="Back to datasets"
+ data-attr="back-to-datasets"
+ to={getDatasetListUrl(searchParams)}
+ />
+ <h2 className="text-base font-semibold m-0 truncate" title={datasetForm.name}>
+ {isNewDataset ? 'New dataset' : (isDataset(dataset) && dataset.name) || 'Dataset'}
+ </h2>
+ {isDataset(dataset) && dataset.archived && (
+ <LemonTag type="muted" size="small">
+ Archived
+ </LemonTag>
+ )}
+ </nav>
+ <div className="flex flex-wrap items-center gap-2">
+ {!displayEditForm && isDataset(dataset) ? (
+ <>
+ <LemonButton
+ size="small"
+ onClick={() => setActiveTab(DatasetTab.Metadata)}
+ data-attr="dataset-details"
+ icon={<IconInfo />}
+ >
+ Details
+ </LemonButton>
+ {canEditDataset && (
+ <LemonButton
+ type="secondary"
+ size="small"
+ onClick={() => editDataset(true)}
+ data-attr="edit-dataset"
+ >
+ Edit
+ </LemonButton>
+ )}
+ </>
+ ) : null}
+ </div>
</div>
- {canEditDataset && (
- <LemonButton
- type="secondary"
- onClick={() => editDataset(true)}
- data-attr="edit-dataset"
- size="small"
- >
- Edit
- </LemonButton>
- )}
</div>
- <p className="m-0">{dataset.description || <span className="italic">Description (optional)</span>}</p>
- {isHistoricalRevision && (
- <LemonBanner
- type="info"
- action={{
- children: 'View latest',
- onClick: () => setFilters({ revision: null, page: 1 }, false),
- }}
- >
- This is a read-only snapshot of the dataset.
- </LemonBanner>
+ {!displayEditForm && isDataset(dataset) && dataset.description && (
+ <p className="text-muted m-0">{dataset.description}</p>
)}
- {datasetExportLoadError && (
- <LemonBanner
- type="error"
- action={{
- children: 'Try again',
- onClick: () => exportDataset(filters.revision ?? undefined),
- loading: datasetExportLoading,
- disabledReason: datasetExportDisabledReason,
- }}
- >
- {datasetExportLoadError.detail || "Couldn't add the dataset to exports. Try again."}
- </LemonBanner>
- )}
- </div>
-
- <LemonTabs activeKey={activeTab} data-attr="dataset-tabs" tabs={tabs} />
-
- <DatasetItemModal
- isOpen={isDatasetItemModalOpen}
- onClose={closeModalAndRefetchDatasetItems}
- partialDatasetItem={selectedDatasetItem}
- datasetId={dataset.id}
- displayBulkCreationButton
- readOnly={selectedItemReadOnly}
- readOnlyReason={selectedItemReadOnlyReason}
- loading={selectedDatasetItemDetailsLoading}
- loadError={selectedDatasetItemLoadError}
- versions={selectedDatasetItemVersions.results}
- versionsLoading={datasetItemVersionsLoading}
- versionsLoadError={datasetItemVersionsLoadError}
- versionsCount={selectedDatasetItemVersions.count}
- versionsPage={datasetItemVersionsPage}
- versionsPageSize={DATASET_ITEM_VERSIONS_PER_PAGE}
- canRestoreVersions={canEditDataset && !selectedDatasetItem?.archived}
- restoringVersion={restoringDatasetItemVersion}
- onRestoreVersion={restoreDatasetItemVersion}
- onVersionsPageChange={
- selectedDatasetItem
- ? (page) => loadDatasetItemVersions({ itemId: selectedDatasetItem.id, page })
- : undefined
- }
- onRetryVersions={
- selectedDatasetItem
- ? () =>
- loadDatasetItemVersions({
- itemId: selectedDatasetItem.id,
- page: datasetItemVersionsPage,
- })
- : undefined
- }
- onRetry={
- searchParams.item
- ? () =>
- loadDatasetItemDetails({
- itemId: String(searchParams.item),
- revision: filters.revision ?? undefined,
- })
- : undefined
- }
- onUnarchive={
- selectedDatasetItem?.archived && canManageDataset && !dataset.archived && !isHistoricalRevision
- ? () => restoreDatasetItem(selectedDatasetItem.id, selectedDatasetItem.version)
- : undefined
- }
- unarchiving={archivingDatasetItemId === selectedDatasetItem?.id}
- />
- </>
- )
-}
-
-function DatasetItems({ dataset }: { dataset: Dataset }): JSX.Element {
- const {
- archivingDatasetItemId,
- datasetItems,
- datasetLoading,
- datasetItemsLoading,
- datasetItemsLoadError,
- datasetRevisionsLoading,
- datasetRevisionsLoadError,
- pagination,
- filters,
- revisionOptions,
- canEditDataset,
- canManageDataset,
- isHistoricalRevision,
- isArchivingDataset,
- datasetExportLoading,
- datasetExportDisabledReason,
- } = useValues(aiObservabilityDatasetLogic)
- const {
- archiveDatasetItem,
- restoreDatasetItem,
- loadDatasetItems,
- loadDatasetRevisions,
- refreshDataset,
- setFilters,
- restoreDataset,
- exportDataset,
- triggerDatasetItemModal,
- } = useActions(aiObservabilityDatasetLogic)
- const { searchParams } = useValues(router)
- const canChangeDatasetStatus = canManageDataset && !isHistoricalRevision
-
- const columns: LemonTableColumns<DatasetItem> = [
- {
- title: 'ID',
- dataIndex: 'id',
- key: 'id',
- width: '10%',
- render: function renderID(_, item) {
- return (
- <Link to={urls.aiObservabilityDataset(dataset.id, { ...searchParams, item: item.id })}>
- {truncateValue(item.id)}
- </Link>
- )
- },
- },
- {
- title: 'Trace',
- dataIndex: 'source_trace_id',
- key: 'source_trace_id',
- width: '10%',
- render: function renderSourceTraceId(_, item) {
- if (!item.source_trace_id || !item.source_timestamp) {
- return <span>-</span>
- }
-
- return (
- <Link
- to={urls.aiObservabilityTrace(item.source_trace_id, {
- event: item.source_event_id ?? undefined,
- timestamp: item.source_timestamp,
- })}
- >
- {truncateValue(item.source_trace_id)}
- </Link>
- )
- },
- },
- {
- title: 'Input',
- dataIndex: 'input',
- key: 'input',
- width: '20%',
- render: function renderInput(_, item) {
- return <JSONColumn>{item.input}</JSONColumn>
- },
- },
- {
- title: 'Expected output',
- dataIndex: 'expected_output',
- key: 'expected_output',
- width: '20%',
- render: function renderExpectedOutput(_, item) {
- return <JSONColumn>{item.expected_output}</JSONColumn>
- },
- },
- {
- title: 'Source output',
- dataIndex: 'source_output',
- key: 'source_output',
- width: '20%',
- render: function renderSourceOutput(_, item) {
- return <JSONColumn>{item.source_output}</JSONColumn>
- },
- },
- {
- title: 'Metadata',
- dataIndex: 'metadata',
- key: 'metadata',
- width: '25%',
- render: function renderMetadata(_, item) {
- return <JSONColumn>{item.metadata}</JSONColumn>
- },
- },
- {
- title: 'Created by',
- dataIndex: 'created_by',
- render: function renderCreatedBy(_, item) {
- const { created_by } = item
- return (
- <div className="flex flex-row items-center flex-nowrap">
- {created_by && <ProfilePicture user={created_by as UserBasicType} size="md" showName />}
- </div>
- )
- },
- },
- {
- title: 'Status',
- key: 'status',
- render: function renderStatus(_, item) {
- return (
- <LemonTag type={item.archived ? 'muted' : 'success'}>
- {item.archived ? 'Archived' : 'Active'}
- </LemonTag>
- )
- },
- },
- createdAtColumn<DatasetItem>() as LemonTableColumn<DatasetItem, keyof DatasetItem | undefined>,
- updatedAtColumn<DatasetItem>() as LemonTableColumn<DatasetItem, keyof DatasetItem | undefined>,
- {
- width: 0,
- render: function renderMore(_, item) {
- return (
- <More
- overlay={
- <>
- <LemonButton
- to={urls.aiObservabilityDataset(dataset.id, { ...searchParams, item: item.id })}
- data-attr={`dataset-item-${item.id}-dropdown-edit`}
- fullWidth
- >
- {canEditDataset && !item.archived ? 'Edit' : 'View'}
- </LemonButton>
-
- {canManageDataset && !dataset.archived && !isHistoricalRevision && (
- <LemonButton
- status={item.archived ? undefined : 'danger'}
- onClick={() =>
- item.archived
- ? restoreDatasetItem(item.id, item.version)
- : archiveDatasetItem(item.id, item.version)
- }
- loading={archivingDatasetItemId === item.id}
- disabledReason={
- archivingDatasetItemId && archivingDatasetItemId !== item.id
- ? 'Another dataset item is being archived'
+ {displayEditForm ? (
+ <div className="flex max-w-160 flex-col gap-4">
+ <EditDatasetForm />
+ <div className="flex flex-wrap items-center gap-2">
+ {todayRailNav ? (
+ <>
+ <Button
+ variant="outline"
+ data-attr="cancel-dataset"
+ onClick={cancelEditing}
+ disabledReason={isDatasetFormSubmitting ? 'Saving…' : undefined}
+ >
+ Cancel
+ </Button>
+ <AccessControlAction
+ resourceType={AccessControlResourceType.LlmAnalytics}
+ minAccessLevel={AccessControlLevel.Editor}
+ userAccessLevel={
+ isDataset(dataset)
+ ? toAccessControlLevel(dataset.user_access_level)
: undefined
}
- data-attr={`dataset-item-${item.id}-dropdown-${
- item.archived ? 'unarchive' : 'archive'
- }`}
- fullWidth
>
- {item.archived ? 'Unarchive' : 'Archive'}
+ <Button
+ variant="primary"
+ data-attr="save-dataset"
+ onClick={submitDatasetForm}
+ loading={isDatasetFormSubmitting}
+ >
+ {isNewDataset ? 'Create dataset' : 'Save'}
+ </Button>
+ </AccessControlAction>
+ </>
+ ) : (
+ <>
+ <LemonButton
+ type="secondary"
+ data-attr="cancel-dataset"
+ onClick={cancelEditing}
+ disabledReason={isDatasetFormSubmitting ? 'Saving…' : undefined}
+ size="small"
+ >
+ Cancel
</LemonButton>
- )}
- </>
- }
- />
- )
- },
- },
- ]
-
- return (
- <>
- <div className="flex justify-between items-center gap-2 flex-wrap">
- <div className="flex items-center gap-2 flex-wrap">
- <LemonSegmentedButton
- value={filters.archived ? 'archived' : 'active'}
- onChange={(value) => setFilters({ archived: value === 'archived', page: 1 }, false)}
- options={[
- { value: 'active', label: 'Active' },
- { value: 'archived', label: 'Archived' },
- ]}
- size="small"
- data-attr="dataset-items-status-filter"
- />
- <LemonSelect
- value={filters.revision === null ? 'latest' : String(filters.revision)}
- onChange={(value) =>
- setFilters({ revision: value === 'latest' ? null : Number(value), page: 1 }, false)
- }
- options={revisionOptions}
- loading={datasetRevisionsLoading}
- size="small"
- data-attr="dataset-revision-select"
- />
- </div>
- <div className="flex items-center gap-2 flex-wrap">
- <RefreshButton
- onClick={refreshDataset}
- isRefreshing={datasetItemsLoading || datasetLoading || datasetRevisionsLoading}
- />
- <LemonButton
- type="secondary"
- onClick={() => exportDataset(filters.revision ?? undefined)}
- loading={datasetExportLoading}
- disabledReason={datasetExportDisabledReason}
- data-attr="export-dataset"
- icon={<IconDownload />}
- size="small"
- >
- Export
- </LemonButton>
- {canEditDataset && (
- <LemonButton
- type="primary"
- onClick={() => triggerDatasetItemModal(true)}
- data-attr="add-dataset-item"
- icon={<IconPlusSmall />}
- size="small"
- >
- Add item
- </LemonButton>
- )}
- {dataset.archived && canChangeDatasetStatus && (
- <LemonButton
- type="primary"
- onClick={restoreDataset}
- loading={isArchivingDataset}
- data-attr="unarchive-dataset"
- size="small"
- >
- Unarchive
- </LemonButton>
- )}
- </div>
- </div>
-
- {datasetItemsLoadError && (
- <LemonBanner type="error" action={{ children: 'Try again', onClick: () => loadDatasetItems(false) }}>
- {datasetItemsLoadError.detail || "Couldn't load dataset items. Try again."}
- </LemonBanner>
- )}
-
- {datasetRevisionsLoadError && (
- <LemonBanner type="error" action={{ children: 'Try again', onClick: loadDatasetRevisions }}>
- {datasetRevisionsLoadError.detail || "Couldn't load dataset revisions. Try again."}
- </LemonBanner>
- )}
-
- <LemonDivider className="my-4" />
-
- <LemonTable
- loading={datasetItemsLoading}
- columns={columns}
- dataSource={datasetItems?.results || []}
- rowKey="id"
- pagination={pagination}
- nouns={['dataset item', 'dataset items']}
- emptyState="No items in this dataset yet."
- />
- </>
- )
-}
-
-function DatasetMetadata({ dataset }: { dataset: Dataset }): JSX.Element {
- return (
- <div className="flex flex-col gap-4 max-w-160">
- <div className="flex flex-col gap-2">
- <h3 className="text-sm font-semibold m-0">Metadata</h3>
- {isObject(dataset.metadata) ? (
- <div className="bg-bg-light p-4 rounded border overflow-x-auto">
- <HighlightedJSONViewer src={dataset.metadata} />
+ <AccessControlAction
+ resourceType={AccessControlResourceType.LlmAnalytics}
+ minAccessLevel={AccessControlLevel.Editor}
+ userAccessLevel={
+ isDataset(dataset)
+ ? toAccessControlLevel(dataset.user_access_level)
+ : undefined
+ }
+ >
+ <LemonButton
+ type="primary"
+ data-attr="save-dataset"
+ onClick={submitDatasetForm}
+ loading={isDatasetFormSubmitting}
+ size="small"
+ >
+ {isNewDataset ? 'Create dataset' : 'Save'}
+ </LemonButton>
+ </AccessControlAction>
+ </>
+ )}
+ </div>
</div>
- ) : (
- <p className="text-muted m-0">No metadata available.</p>
- )}
- </div>
- </div>
+ ) : isDataset(dataset) ? (
+ <DatasetContents dataset={dataset} />
+ ) : null}
+ </OfflineEvaluationsScene>
+ </Form>
)
}
diff --git a/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx b/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx
index 4252dc54025e..03815c7aea88 100644
--- a/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx
+++ b/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx
@@ -1,5 +1,6 @@
import { useActions, useValues } from 'kea'
import { router } from 'kea-router'
+import type { ReactNode } from 'react'
import { AccessControlAction } from 'lib/components/AccessControlAction'
import { LemonButton } from 'lib/lemon-ui/LemonButton'
@@ -8,8 +9,6 @@ import { Link } from 'lib/lemon-ui/Link'
import { ProfilePicture } from 'lib/lemon-ui/ProfilePicture'
import { SceneExport } from 'scenes/sceneTypes'
-import { SceneContent } from '~/layout/scenes/components/SceneContent'
-import { SceneTitleSection } from '~/layout/scenes/components/SceneTitleSection'
import { LemonInput } from '~/lib/lemon-ui/LemonInput'
import { LemonSegmentedButton } from '~/lib/lemon-ui/LemonSegmentedButton'
import { LemonTable, LemonTableColumn, LemonTableColumns } from '~/lib/lemon-ui/LemonTable'
@@ -32,7 +31,7 @@ export const scene: SceneExport = {
emptyState: datasetsEmptyState,
}
-export function AIObservabilityDatasetsScene(): JSX.Element {
+export function AIObservabilityDatasetsScene({ navigation }: { navigation?: ReactNode }): JSX.Element {
const { loadDatasets, setFilters, archiveDataset, restoreDataset } = useActions(aiObservabilityDatasetsLogic)
const { archivingDatasetId, datasets, datasetsLoading, sorting, pagination, filters, datasetCountLabel } =
useValues(aiObservabilityDatasetsLogic)
@@ -44,13 +43,13 @@ export function AIObservabilityDatasetsScene(): JSX.Element {
title: 'Name',
dataIndex: 'name',
key: 'name',
- width: '25%',
+ width: '21%',
render: function renderName(_, dataset) {
return (
<Link
to={datasetUrl(dataset.id)}
data-testid="dataset-link"
- className="block max-w-64 @max-[50rem]/datasets:max-w-32 truncate font-semibold"
+ className="block max-w-full @max-[50rem]/datasets:max-w-48 truncate font-semibold"
>
{dataset.name}
</Link>
@@ -61,15 +60,20 @@ export function AIObservabilityDatasetsScene(): JSX.Element {
title: 'Description',
dataIndex: 'description',
key: 'description',
- width: '20%',
+ width: '16%',
className: '@max-[50rem]/datasets:hidden',
render: function renderDescription(description) {
- return <span className="text-muted">{String(description) || <i>–</i>}</span>
+ return (
+ <span className="block max-w-full truncate text-muted" title={String(description) || undefined}>
+ {String(description) || <i>–</i>}
+ </span>
+ )
},
},
{
title: 'Created by',
- className: '@max-[50rem]/datasets:hidden',
+ width: '14%',
+ className: '@max-[50rem]/datasets:hidden [&_.profile-package]:max-w-full [&_.profile-name]:truncate',
dataIndex: 'created_by',
render: function renderCreatedBy(_, item) {
const { created_by } = item
@@ -94,9 +98,10 @@ export function AIObservabilityDatasetsScene(): JSX.Element {
{
title: 'Tags',
key: 'tags',
- width: '20%',
+ width: '16%',
render: (_, dataset) => (
<ResourceTags
+ compact
entity="datasets"
resourceId={dataset.id}
tags={dataset.tags}
@@ -109,14 +114,16 @@ export function AIObservabilityDatasetsScene(): JSX.Element {
},
{
...(createdAtColumn<Dataset>() as LemonTableColumn<Dataset, keyof Dataset | undefined>),
- className: '@max-[60rem]/datasets:hidden',
+ width: '11%',
+ className: '@max-[50rem]/datasets:hidden',
},
{
...(updatedAtColumn<Dataset>() as LemonTableColumn<Dataset, keyof Dataset | undefined>),
+ width: '11%',
className: '@max-[50rem]/datasets:hidden',
},
{
- width: 0,
+ width: 40,
render: function renderMore(_, dataset) {
return (
<More
@@ -163,38 +170,32 @@ export function AIObservabilityDatasetsScene(): JSX.Element {
]
return (
- <SceneContent className="@container/datasets">
- <SceneTitleSection
- name="Datasets"
- description="Create and manage datasets to test changes and evaluate your AI outputs. [Learn more](https://posthog.com/docs/ai-evals/datasets)"
- markdown
- resourceType={{ type: 'llm_datasets' }}
- />
- <div className="flex gap-x-4 gap-y-2 items-center flex-wrap py-4 -mt-4 mb-4 border-b justify-between">
- <div className="flex gap-2 items-center flex-wrap">
- <LemonSegmentedButton
- value={filters.archived ? 'archived' : 'active'}
- onChange={(value) => setFilters({ archived: value === 'archived' }, true, false)}
- options={[
- { value: 'active', label: 'Active' },
- { value: 'archived', label: 'Archived' },
- ]}
- size="small"
- data-attr="datasets-status-filter"
- />
- <ResourceTagFilter filters={filters} onChange={(tags) => setFilters(tags, true, false)} />
- <LemonInput
- type="search"
- placeholder="Search datasets..."
- value={filters.search}
- data-attr="datasets-search-input"
- onChange={(value) => setFilters({ search: value })}
- className="max-w-md"
- data-testid="search-datasets-input"
- />
- </div>
- <div className="flex gap-2 items-center flex-wrap">
- <div className="text-muted-alt">{datasetCountLabel}</div>
+ <div className="@container/datasets space-y-4">
+ <div className="flex gap-2 items-center flex-wrap" aria-label="Dataset filters">
+ {navigation}
+ <LemonSegmentedButton
+ value={filters.archived ? 'archived' : 'active'}
+ onChange={(value) => setFilters({ archived: value === 'archived' }, true, false)}
+ options={[
+ { value: 'active', label: 'Active' },
+ { value: 'archived', label: 'Archived' },
+ ]}
+ size="small"
+ data-attr="datasets-status-filter"
+ />
+ <ResourceTagFilter size="small" filters={filters} onChange={(tags) => setFilters(tags, true, false)} />
+ <LemonInput
+ size="small"
+ type="search"
+ placeholder="Search datasets..."
+ value={filters.search}
+ data-attr="datasets-search-input"
+ onChange={(value) => setFilters({ search: value })}
+ className="max-w-md"
+ data-testid="search-datasets-input"
+ />
+ <div className="flex gap-2 items-center flex-wrap ml-auto">
+ <div className="text-muted-alt text-xs">{datasetCountLabel}</div>
<AccessControlAction
resourceType={AccessControlResourceType.LlmAnalytics}
minAccessLevel={AccessControlLevel.Editor}
@@ -214,7 +215,9 @@ export function AIObservabilityDatasetsScene(): JSX.Element {
<LemonTable
id="datasets"
- className="@max-[50rem]/datasets:[&_colgroup]:hidden"
+ className="@max-[50rem]/datasets:[&_colgroup]:hidden @max-[50rem]/datasets:[&_table]:table-auto"
+ size="small"
+ tableLayout="fixed"
loading={datasetsLoading}
columns={columns}
dataSource={datasets.results}
@@ -232,6 +235,6 @@ export function AIObservabilityDatasetsScene(): JSX.Element {
loadingSkeletonRows={DATASETS_PER_PAGE}
nouns={['dataset', 'datasets']}
/>
- </SceneContent>
+ </div>
)
}
diff --git a/products/ai_observability/frontend/datasets/DatasetContents.tsx b/products/ai_observability/frontend/datasets/DatasetContents.tsx
new file mode 100644
index 000000000000..8c3ac36510f3
--- /dev/null
+++ b/products/ai_observability/frontend/datasets/DatasetContents.tsx
@@ -0,0 +1,139 @@
+import { useActions, useValues } from 'kea'
+import { router } from 'kea-router'
+
+import { LemonBanner } from '@posthog/lemon-ui'
+
+import type { DatasetReadApi as Dataset } from '../generated/api.schemas'
+import { DATASET_ITEM_VERSIONS_PER_PAGE, DatasetTab, aiObservabilityDatasetLogic } from './aiObservabilityDatasetLogic'
+import { DatasetDetailsModal } from './DatasetDetailsModal'
+import { DatasetItemModal } from './DatasetItemModal'
+import { DatasetItems } from './DatasetItems'
+
+export function DatasetContents({ dataset }: { dataset: Dataset }): JSX.Element {
+ const {
+ activeTab,
+ isDatasetItemModalOpen,
+ selectedDatasetItem,
+ selectedDatasetItemLoadError,
+ selectedDatasetItemDetailsLoading,
+ selectedDatasetItemVersions,
+ datasetItemVersionsLoading,
+ datasetItemVersionsLoadError,
+ datasetItemVersionsPage,
+ restoringDatasetItemVersion,
+ archivingDatasetItemId,
+ canEditDataset,
+ canManageDataset,
+ isHistoricalRevision,
+ filters,
+ datasetExportLoading,
+ datasetExportLoadError,
+ datasetExportDisabledReason,
+ } = useValues(aiObservabilityDatasetLogic)
+ const {
+ closeModalAndRefetchDatasetItems,
+ loadDatasetItemDetails,
+ restoreDatasetItem,
+ restoreDatasetItemVersion,
+ loadDatasetItemVersions,
+ setFilters,
+ setActiveTab,
+ exportDataset,
+ } = useActions(aiObservabilityDatasetLogic)
+ const { searchParams } = useValues(router)
+ const selectedItemReadOnly = !canEditDataset || !!selectedDatasetItem?.archived
+ const selectedItemReadOnlyReason = isHistoricalRevision
+ ? `You're viewing revision ${filters.revision}. Select Latest to edit items.`
+ : dataset.archived
+ ? 'This dataset is archived. Unarchive it to edit items.'
+ : selectedDatasetItem?.archived
+ ? 'This item is archived. Unarchive it to edit.'
+ : !canManageDataset
+ ? 'You have view-only access to this dataset.'
+ : undefined
+
+ return (
+ <>
+ {isHistoricalRevision && (
+ <LemonBanner
+ type="info"
+ action={{
+ children: 'View latest',
+ onClick: () => setFilters({ revision: null, page: 1 }, false),
+ }}
+ >
+ This is a read-only snapshot of the dataset.
+ </LemonBanner>
+ )}
+ {datasetExportLoadError && (
+ <LemonBanner
+ type="error"
+ action={{
+ children: 'Try again',
+ onClick: () => exportDataset(filters.revision ?? undefined),
+ loading: datasetExportLoading,
+ disabledReason: datasetExportDisabledReason,
+ }}
+ >
+ {datasetExportLoadError.detail || "Couldn't add the dataset to exports. Try again."}
+ </LemonBanner>
+ )}
+ <DatasetItems dataset={dataset} />
+ <DatasetDetailsModal
+ dataset={dataset}
+ isOpen={activeTab === DatasetTab.Metadata && !isDatasetItemModalOpen}
+ onClose={() => setActiveTab(DatasetTab.Items)}
+ />
+
+ <DatasetItemModal
+ isOpen={isDatasetItemModalOpen}
+ onClose={closeModalAndRefetchDatasetItems}
+ partialDatasetItem={selectedDatasetItem}
+ datasetId={dataset.id}
+ displayBulkCreationButton
+ readOnly={selectedItemReadOnly}
+ readOnlyReason={selectedItemReadOnlyReason}
+ loading={selectedDatasetItemDetailsLoading}
+ loadError={selectedDatasetItemLoadError}
+ versions={selectedDatasetItemVersions.results}
+ versionsLoading={datasetItemVersionsLoading}
+ versionsLoadError={datasetItemVersionsLoadError}
+ versionsCount={selectedDatasetItemVersions.count}
+ versionsPage={datasetItemVersionsPage}
+ versionsPageSize={DATASET_ITEM_VERSIONS_PER_PAGE}
+ canRestoreVersions={canEditDataset && !selectedDatasetItem?.archived}
+ restoringVersion={restoringDatasetItemVersion}
+ onRestoreVersion={restoreDatasetItemVersion}
+ onVersionsPageChange={
+ selectedDatasetItem
+ ? (page) => loadDatasetItemVersions({ itemId: selectedDatasetItem.id, page })
+ : undefined
+ }
+ onRetryVersions={
+ selectedDatasetItem
+ ? () =>
+ loadDatasetItemVersions({
+ itemId: selectedDatasetItem.id,
+ page: datasetItemVersionsPage,
+ })
+ : undefined
+ }
+ onRetry={
+ searchParams.item
+ ? () =>
+ loadDatasetItemDetails({
+ itemId: String(searchParams.item),
+ revision: filters.revision ?? undefined,
+ })
+ : undefined
+ }
+ onUnarchive={
+ selectedDatasetItem?.archived && canManageDataset && !dataset.archived && !isHistoricalRevision
+ ? () => restoreDatasetItem(selectedDatasetItem.id, selectedDatasetItem.version)
+ : undefined
+ }
+ unarchiving={archivingDatasetItemId === selectedDatasetItem?.id}
+ />
+ </>
+ )
+}
diff --git a/products/ai_observability/frontend/datasets/DatasetDetailsModal.tsx b/products/ai_observability/frontend/datasets/DatasetDetailsModal.tsx
new file mode 100644
index 000000000000..32775e45beac
--- /dev/null
+++ b/products/ai_observability/frontend/datasets/DatasetDetailsModal.tsx
@@ -0,0 +1,43 @@
+import { LemonModal } from '@posthog/lemon-ui'
+
+import { HighlightedJSONViewer } from 'lib/components/HighlightedJSONViewer'
+import { isObject } from 'lib/utils/guards'
+
+import type { DatasetReadApi } from '../generated/api.schemas'
+
+export function DatasetDetailsModal({
+ dataset,
+ isOpen,
+ onClose,
+}: {
+ dataset: DatasetReadApi
+ isOpen: boolean
+ onClose: () => void
+}): JSX.Element {
+ return (
+ <LemonModal title="Dataset details" isOpen={isOpen} onClose={onClose} width="40rem">
+ <div className="flex flex-col gap-4">
+ <div>
+ <h4 className="mb-1">Name</h4>
+ <p className="m-0 break-words">{dataset.name}</p>
+ </div>
+ {dataset.description && (
+ <div>
+ <h4 className="mb-1">Description</h4>
+ <p className="m-0 break-words">{dataset.description}</p>
+ </div>
+ )}
+ <div>
+ <h4 className="mb-1">Metadata</h4>
+ {isObject(dataset.metadata) && Object.keys(dataset.metadata).length > 0 ? (
+ <div className="overflow-x-auto">
+ <HighlightedJSONViewer src={dataset.metadata} />
+ </div>
+ ) : (
+ <p className="text-muted m-0">No metadata.</p>
+ )}
+ </div>
+ </div>
+ </LemonModal>
+ )
+}
diff --git a/products/ai_observability/frontend/datasets/DatasetItems.tsx b/products/ai_observability/frontend/datasets/DatasetItems.tsx
new file mode 100644
index 000000000000..e802bf47dc4f
--- /dev/null
+++ b/products/ai_observability/frontend/datasets/DatasetItems.tsx
@@ -0,0 +1,293 @@
+import { useActions, useValues } from 'kea'
+import { router } from 'kea-router'
+
+import { IconDownload, IconPlusSmall } from '@posthog/icons'
+import {
+ LemonBanner,
+ LemonButton,
+ LemonSegmentedButton,
+ LemonSelect,
+ LemonTag,
+ Link,
+ ProfilePicture,
+} from '@posthog/lemon-ui'
+
+import { More } from 'lib/lemon-ui/LemonButton/More'
+import { LemonTable, LemonTableColumn, LemonTableColumns } from 'lib/lemon-ui/LemonTable'
+import { createdAtColumn, updatedAtColumn } from 'lib/lemon-ui/LemonTable/columnUtils'
+import { urls } from 'scenes/urls'
+
+import { UserBasicType } from '~/types'
+
+import type { DatasetItemReadApi as DatasetItem, DatasetReadApi as Dataset } from '../generated/api.schemas'
+import { truncateValue } from '../utils'
+import { aiObservabilityDatasetLogic } from './aiObservabilityDatasetLogic'
+import { JSONColumn } from './JSONColumn'
+import { RefreshButton } from './RefreshButton'
+
+export function DatasetItems({ dataset }: { dataset: Dataset }): JSX.Element {
+ const {
+ archivingDatasetItemId,
+ datasetItems,
+ datasetLoading,
+ datasetItemsLoading,
+ datasetItemsLoadError,
+ datasetRevisionsLoading,
+ datasetRevisionsLoadError,
+ pagination,
+ filters,
+ revisionOptions,
+ canEditDataset,
+ canManageDataset,
+ isHistoricalRevision,
+ isArchivingDataset,
+ datasetExportLoading,
+ datasetExportDisabledReason,
+ } = useValues(aiObservabilityDatasetLogic)
+ const {
+ archiveDatasetItem,
+ restoreDatasetItem,
+ loadDatasetItems,
+ loadDatasetRevisions,
+ refreshDataset,
+ setFilters,
+ restoreDataset,
+ exportDataset,
+ triggerDatasetItemModal,
+ } = useActions(aiObservabilityDatasetLogic)
+ const { searchParams } = useValues(router)
+ const canChangeDatasetStatus = canManageDataset && !isHistoricalRevision
+
+ const columns: LemonTableColumns<DatasetItem> = [
+ {
+ title: 'ID',
+ dataIndex: 'id',
+ key: 'id',
+ width: '10%',
+ render: function renderID(_, item) {
+ return (
+ <Link to={urls.aiObservabilityDataset(dataset.id, { ...searchParams, item: item.id })}>
+ {truncateValue(item.id)}
+ </Link>
+ )
+ },
+ },
+ {
+ title: 'Trace',
+ dataIndex: 'source_trace_id',
+ key: 'source_trace_id',
+ width: '10%',
+ render: function renderSourceTraceId(_, item) {
+ if (!item.source_trace_id || !item.source_timestamp) {
+ return <span>-</span>
+ }
+
+ return (
+ <Link
+ to={urls.aiObservabilityTrace(item.source_trace_id, {
+ event: item.source_event_id ?? undefined,
+ timestamp: item.source_timestamp,
+ })}
+ >
+ {truncateValue(item.source_trace_id)}
+ </Link>
+ )
+ },
+ },
+ {
+ title: 'Input',
+ dataIndex: 'input',
+ key: 'input',
+ width: '20%',
+ render: function renderInput(_, item) {
+ return <JSONColumn>{item.input}</JSONColumn>
+ },
+ },
+ {
+ title: 'Expected output',
+ dataIndex: 'expected_output',
+ key: 'expected_output',
+ width: '20%',
+ render: function renderExpectedOutput(_, item) {
+ return <JSONColumn>{item.expected_output}</JSONColumn>
+ },
+ },
+ {
+ title: 'Source output',
+ dataIndex: 'source_output',
+ key: 'source_output',
+ width: '20%',
+ render: function renderSourceOutput(_, item) {
+ return <JSONColumn>{item.source_output}</JSONColumn>
+ },
+ },
+ {
+ title: 'Metadata',
+ dataIndex: 'metadata',
+ key: 'metadata',
+ width: '25%',
+ render: function renderMetadata(_, item) {
+ return <JSONColumn>{item.metadata}</JSONColumn>
+ },
+ },
+ {
+ title: 'Created by',
+ dataIndex: 'created_by',
+ render: function renderCreatedBy(_, item) {
+ const { created_by } = item
+ return (
+ <div className="flex flex-row items-center flex-nowrap">
+ {created_by && <ProfilePicture user={created_by as UserBasicType} size="md" showName />}
+ </div>
+ )
+ },
+ },
+ {
+ title: 'Status',
+ key: 'status',
+ render: function renderStatus(_, item) {
+ return (
+ <LemonTag size="small" type={item.archived ? 'muted' : 'success'}>
+ {item.archived ? 'Archived' : 'Active'}
+ </LemonTag>
+ )
+ },
+ },
+ createdAtColumn<DatasetItem>() as LemonTableColumn<DatasetItem, keyof DatasetItem | undefined>,
+ updatedAtColumn<DatasetItem>() as LemonTableColumn<DatasetItem, keyof DatasetItem | undefined>,
+ {
+ width: 0,
+ render: function renderMore(_, item) {
+ return (
+ <More
+ overlay={
+ <>
+ <LemonButton
+ to={urls.aiObservabilityDataset(dataset.id, { ...searchParams, item: item.id })}
+ data-attr={`dataset-item-${item.id}-dropdown-edit`}
+ fullWidth
+ >
+ {canEditDataset && !item.archived ? 'Edit' : 'View'}
+ </LemonButton>
+
+ {canManageDataset && !dataset.archived && !isHistoricalRevision && (
+ <LemonButton
+ status={item.archived ? undefined : 'danger'}
+ onClick={() =>
+ item.archived
+ ? restoreDatasetItem(item.id, item.version)
+ : archiveDatasetItem(item.id, item.version)
+ }
+ loading={archivingDatasetItemId === item.id}
+ disabledReason={
+ archivingDatasetItemId && archivingDatasetItemId !== item.id
+ ? 'Another dataset item is being archived'
+ : undefined
+ }
+ data-attr={`dataset-item-${item.id}-dropdown-${
+ item.archived ? 'unarchive' : 'archive'
+ }`}
+ fullWidth
+ >
+ {item.archived ? 'Unarchive' : 'Archive'}
+ </LemonButton>
+ )}
+ </>
+ }
+ />
+ )
+ },
+ },
+ ]
+
+ return (
+ <div className="flex flex-col gap-3">
+ <div className="flex justify-between items-center gap-2 flex-wrap">
+ <div className="flex items-center gap-2 flex-wrap">
+ <LemonSegmentedButton
+ value={filters.archived ? 'archived' : 'active'}
+ onChange={(value) => setFilters({ archived: value === 'archived', page: 1 }, false)}
+ options={[
+ { value: 'active', label: 'Active' },
+ { value: 'archived', label: 'Archived' },
+ ]}
+ size="small"
+ data-attr="dataset-items-status-filter"
+ />
+ <LemonSelect
+ value={filters.revision === null ? 'latest' : String(filters.revision)}
+ onChange={(value) =>
+ setFilters({ revision: value === 'latest' ? null : Number(value), page: 1 }, false)
+ }
+ options={revisionOptions}
+ loading={datasetRevisionsLoading}
+ size="small"
+ data-attr="dataset-revision-select"
+ />
+ </div>
+ <div className="flex items-center gap-2 flex-wrap">
+ <RefreshButton
+ onClick={refreshDataset}
+ isRefreshing={datasetItemsLoading || datasetLoading || datasetRevisionsLoading}
+ />
+ <LemonButton
+ type="secondary"
+ onClick={() => exportDataset(filters.revision ?? undefined)}
+ loading={datasetExportLoading}
+ disabledReason={datasetExportDisabledReason}
+ data-attr="export-dataset"
+ icon={<IconDownload />}
+ size="small"
+ >
+ Export
+ </LemonButton>
+ {canEditDataset && (
+ <LemonButton
+ type="primary"
+ onClick={() => triggerDatasetItemModal(true)}
+ data-attr="add-dataset-item"
+ icon={<IconPlusSmall />}
+ size="small"
+ >
+ Add item
+ </LemonButton>
+ )}
+ {dataset.archived && canChangeDatasetStatus && (
+ <LemonButton
+ type="primary"
+ onClick={restoreDataset}
+ loading={isArchivingDataset}
+ data-attr="unarchive-dataset"
+ size="small"
+ >
+ Unarchive
+ </LemonButton>
+ )}
+ </div>
+ </div>
+
+ {datasetItemsLoadError && (
+ <LemonBanner type="error" action={{ children: 'Try again', onClick: () => loadDatasetItems(false) }}>
+ {datasetItemsLoadError.detail || "Couldn't load dataset items. Try again."}
+ </LemonBanner>
+ )}
+
+ {datasetRevisionsLoadError && (
+ <LemonBanner type="error" action={{ children: 'Try again', onClick: loadDatasetRevisions }}>
+ {datasetRevisionsLoadError.detail || "Couldn't load dataset revisions. Try again."}
+ </LemonBanner>
+ )}
+
+ <LemonTable
+ size="small"
+ loading={datasetItemsLoading}
+ columns={columns}
+ dataSource={datasetItems?.results || []}
+ rowKey="id"
+ pagination={pagination}
+ nouns={['dataset item', 'dataset items']}
+ emptyState="No items in this dataset yet."
+ />
+ </div>
+ )
+}
diff --git a/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.test.ts b/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.test.ts
index ea27bb97eaa6..8bd6d7e26529 100644
--- a/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.test.ts
+++ b/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.test.ts
@@ -15,7 +15,12 @@ import type {
DatasetItemReadApi as DatasetItem,
DatasetReadApi as Dataset,
} from '../generated/api.schemas'
-import { DatasetFormValues, DatasetLogicProps, aiObservabilityDatasetLogic } from './aiObservabilityDatasetLogic'
+import {
+ DatasetFormValues,
+ DatasetLogicProps,
+ DatasetTab,
+ aiObservabilityDatasetLogic,
+} from './aiObservabilityDatasetLogic'
import { aiObservabilityDatasetsLogic } from './aiObservabilityDatasetsLogic'
import { datasetsApi } from './datasetsApi'
import { EMPTY_JSON } from './utils'
@@ -172,7 +177,7 @@ describe('aiObservabilityDatasetLogic', () => {
})
})
- it('can create a new dataset with non-empty metadata', async () => {
+ it('creates a dataset inside Offline evals and preserves the return filters', async () => {
const formValues: DatasetFormValues = {
name: 'New Dataset',
description: 'New description',
@@ -181,6 +186,7 @@ describe('aiObservabilityDatasetLogic', () => {
mockDatasetsApi.createDataset.mockResolvedValue(mockDataset)
const routerReplaceSpy = jest.spyOn(router.actions, 'replace')
+ router.actions.push(urls.aiObservabilityDataset('new'), { search: 'quality', datasets_page: '2' })
await expectLogic(logic, () => {
logic.actions.setDatasetFormValues(formValues)
@@ -193,7 +199,12 @@ describe('aiObservabilityDatasetLogic', () => {
metadata: { test: 'value' },
})
expect(lemonToast.success).toHaveBeenCalledWith('Dataset created successfully')
- expect(routerReplaceSpy).toHaveBeenCalledWith(urls.aiObservabilityDataset(mockDataset.id))
+ expect(routerReplaceSpy).toHaveBeenCalledWith(
+ `/ai-evals/evaluations/offline/datasets/${mockDataset.id}?search=quality&datasets_page=2`
+ )
+ expect(logic.values.breadcrumbs[0].path).toBe(
+ '/ai-evals/evaluations/offline/experiments?tab=datasets&search=quality&datasets_page=2'
+ )
expect(logic.values.dataset).toEqual(mockDataset)
expect(logic.values.isEditingDataset).toBe(false)
})
@@ -1193,15 +1204,36 @@ describe('aiObservabilityDatasetLogic', () => {
})
})
- it('sets active tab from URL parameters via urlToAction', async () => {
+ it('opens and closes dataset details through URL state without reloading items or losing filters', async () => {
const datasetUrl = urls.aiObservabilityDataset(mockDataset.id)
await expectLogic(logic, () => {
- router.actions.push(datasetUrl, { tab: 'metadata', page: '1' })
+ router.actions.push(datasetUrl, { tab: 'metadata', search: 'quality' }, { section: 'items' })
}).toFinishAllListeners()
- // Verify tab is set from URL
expect(logic.values.activeTab).toBe('metadata')
+ mockDatasetsApi.listItems.mockClear()
+
+ await expectLogic(logic, () => {
+ logic.actions.setActiveTab(DatasetTab.Items)
+ }).toFinishAllListeners()
+
+ expect(router.values.searchParams).toEqual({ search: 'quality' })
+ expect(router.values.hashParams).toEqual({ section: 'items' })
+ expect(logic.values.activeTab).toBe('items')
+
+ await expectLogic(logic, () => {
+ logic.actions.setActiveTab(DatasetTab.Metadata)
+ }).toFinishAllListeners()
+
+ expect(router.values.searchParams).toEqual({ search: 'quality', tab: 'metadata' })
+
+ await expectLogic(logic, () => {
+ router.actions.push(datasetUrl, { search: 'quality' })
+ }).toFinishAllListeners()
+
+ expect(logic.values.activeTab).toBe('items')
+ expect(mockDatasetsApi.listItems).not.toHaveBeenCalled()
})
it('closes modal and clears state when closeModalAndRefetchDatasetItems is called', async () => {
diff --git a/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.ts b/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.ts
index 71faa18390a9..40df3b272edd 100644
--- a/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.ts
+++ b/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.ts
@@ -41,7 +41,7 @@ import type {
PaginatedDatasetRevisionReadListApi,
} from '../generated/api.schemas'
import { truncateValue } from '../utils'
-import { aiObservabilityDatasetsLogic, getDatasetListUrl } from './aiObservabilityDatasetsLogic'
+import { aiObservabilityDatasetsLogic, getDatasetDetailUrl, getDatasetListUrl } from './aiObservabilityDatasetsLogic'
import { datasetsApi } from './datasetsApi'
import { EMPTY_JSON, isStringJsonObject, parseJsonObject, prettifyJson } from './utils'
@@ -866,7 +866,7 @@ export const aiObservabilityDatasetLogic = kea<aiObservabilityDatasetLogicType>(
metadata: parseJsonObject(formValues.metadata),
})
lemonToast.success('Dataset created successfully')
- router.actions.replace(urls.aiObservabilityDataset(savedDataset.id))
+ router.actions.replace(getDatasetDetailUrl(savedDataset.id, router.values.searchParams))
void actions.addProductIntent({
product_type: ProductKey.LLM_DATASETS,
@@ -1247,12 +1247,6 @@ export const aiObservabilityDatasetLogic = kea<aiObservabilityDatasetLogicType>(
}
},
- setActiveTab: ({ tab }) => {
- if (tab === DatasetTab.Items && props.datasetId !== 'new') {
- actions.loadDatasetItems(true)
- }
- },
-
closeModalAndRefetchDatasetItems: async ({ refetchDatasetItems }) => {
if (refetchDatasetItems) {
await asyncActions.refreshAfterDatasetItemMutation()
@@ -1262,12 +1256,9 @@ export const aiObservabilityDatasetLogic = kea<aiObservabilityDatasetLogicType>(
urlToAction(({ actions, values }) => ({
[urls.aiObservabilityDataset(':id')]: (_, searchParams) => {
- if (
- searchParams.tab &&
- Object.values(DatasetTab).includes(searchParams.tab as DatasetTab) &&
- searchParams.tab !== values.activeTab
- ) {
- actions.setActiveTab(searchParams.tab as DatasetTab)
+ const activeTab = searchParams.tab === DatasetTab.Metadata ? DatasetTab.Metadata : DatasetTab.Items
+ if (activeTab !== values.activeTab) {
+ actions.setActiveTab(activeTab)
}
const newFilters = cleanFilters(searchParams)
@@ -1291,6 +1282,14 @@ export const aiObservabilityDatasetLogic = kea<aiObservabilityDatasetLogicType>(
})),
trackedActionToUrl(({ values }) => ({
+ setActiveTab: () => [
+ router.values.location.pathname,
+ {
+ ...router.values.searchParams,
+ tab: values.activeTab === DatasetTab.Metadata ? DatasetTab.Metadata : undefined,
+ },
+ router.values.hashParams,
+ ],
setFilters: () => {
const nextSearchParams = {
...router.values.searchParams,
diff --git a/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.test.ts b/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.test.ts
index e320ce347539..b948a52f776a 100644
--- a/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.test.ts
+++ b/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.test.ts
@@ -1,12 +1,15 @@
import { router } from 'kea-router'
import { expectLogic } from 'kea-test-utils'
+import { FEATURE_FLAGS } from 'lib/constants'
import { lemonToast } from 'lib/lemon-ui/LemonToast/LemonToast'
+import { featureFlagLogic } from 'lib/logic/featureFlagLogic'
import { urls } from 'scenes/urls'
import { initKeaTests } from '~/test/init'
import type { DatasetReadApi as Dataset } from '../generated/api.schemas'
+import { offlineEvaluationsLogic } from '../offline-evaluations/offlineEvaluationsLogic'
import {
DATASETS_PER_PAGE,
aiObservabilityDatasetsLogic,
@@ -96,24 +99,41 @@ describe('aiObservabilityDatasetsLogic', () => {
})
expect(getDatasetDetailUrl(mockDataset1.id, searchParams)).toContain('datasets_page=2')
expect(getDatasetListUrl(searchParams)).toContain('datasets_page=2')
+ expect(getDatasetListUrl(searchParams)).toContain('tab=datasets')
expect(getDatasetDetailUrl(mockDataset1.id, searchParams)).not.toContain('item=')
})
- it('shares tag filters in the URL and restores them when returning to datasets', async () => {
+ it('shares dataset tag filters in the URL without replacing them with experiment filters', async () => {
+ featureFlagLogic.mount()
+ featureFlagLogic.actions.setFeatureFlags([], {
+ [FEATURE_FLAGS.LLM_ANALYTICS_DATASETS]: true,
+ [FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS]: true,
+ })
+ const tabs = offlineEvaluationsLogic()
+ tabs.mount()
router.actions.push(urls.aiObservabilityDatasets())
const logic = aiObservabilityDatasetsLogic()
logic.mount()
await expectLogic(logic, () =>
logic.actions.setFilters({ tags: ['quality', 'a,b'], tag_match: 'all' }, true, false)
).toFinishAllListeners()
- expect(router.values.searchParams).toMatchObject({ tags: ['quality', 'a,b'], tag_match: 'all' })
+ expect(router.values.searchParams).toMatchObject({
+ tab: 'datasets',
+ tags: ['quality', 'a,b'],
+ tag_match: 'all',
+ })
expect(mockDatasetsApi.listDatasets).toHaveBeenLastCalledWith(
expect.objectContaining({ tags: '["quality","a,b"]', tag_match: 'all' })
)
- router.actions.push(urls.aiObservabilityScorers())
- router.actions.push(urls.aiObservabilityDatasets())
+ router.actions.push(urls.aiObservabilityOfflineEvaluations(), {
+ tab: 'experiments',
+ search: 'experiment-name',
+ tags: ['experiment-only'],
+ tag_match: 'any',
+ })
+ tabs.actions.setActiveTab('datasets')
await expectLogic(logic).toFinishAllListeners()
- expect(logic.values.filters.tags).toEqual(['quality', 'a,b'])
+ expect(logic.values.filters).toMatchObject({ search: '', tags: ['quality', 'a,b'], tag_match: 'all' })
expect(router.values.searchParams.tags).toEqual(['quality', 'a,b'])
router.actions.push(urls.aiObservabilityDatasets(), { tags: [], tag_match: 'any' })
await expectLogic(logic).toFinishAllListeners()
diff --git a/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.ts b/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.ts
index 4ccd90becfa0..ef18baf9db37 100644
--- a/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.ts
+++ b/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.ts
@@ -337,6 +337,7 @@ export const aiObservabilityDatasetsLogic = kea<aiObservabilityDatasetsLogicType
const { page, ...pagedSearchOrderParams } = cleanPagedSearchOrderParams(values.filters)
const nextValues = {
...pagedSearchOrderParams,
+ tab: 'datasets',
datasets_page: page,
dataset_status: values.filters.archived ? 'archived' : undefined,
tags: values.filters.tags,
@@ -344,7 +345,12 @@ export const aiObservabilityDatasetsLogic = kea<aiObservabilityDatasetsLogicType
}
const urlValues = cleanFilters(router.values.searchParams)
if (!objectsEqual(values.filters, urlValues)) {
- return [urls.aiObservabilityDatasets(), nextValues, {}, { replace: true }]
+ return [
+ urls.aiObservabilityOfflineEvaluations(),
+ nextValues,
+ router.values.hashParams,
+ { replace: true },
+ ]
}
}
return {
@@ -353,7 +359,10 @@ export const aiObservabilityDatasetsLogic = kea<aiObservabilityDatasetsLogicType
}),
urlToAction(({ actions, values }) => ({
- [urls.aiObservabilityDatasets()]: (_, searchParams, __, { method }) => {
+ [urls.aiObservabilityOfflineEvaluations()]: (_, searchParams, __, { method }) => {
+ if (searchParams.tab !== 'datasets') {
+ return
+ }
const newFilters = cleanFilters({ ...searchParams, ...restoreTagFilters('datasets', searchParams) })
if (!objectsEqual(values.filters, newFilters)) {
actions.setFilters(newFilters, false)
diff --git a/products/ai_observability/frontend/datasets/saveToDatasetButtonLogic.test.ts b/products/ai_observability/frontend/datasets/saveToDatasetButtonLogic.test.ts
index 83288d875389..cd712c5a2b90 100644
--- a/products/ai_observability/frontend/datasets/saveToDatasetButtonLogic.test.ts
+++ b/products/ai_observability/frontend/datasets/saveToDatasetButtonLogic.test.ts
@@ -448,7 +448,9 @@ describe('saveToDatasetButtonLogic', () => {
viewAction()
- expect(router.actions.push).toHaveBeenCalledWith('/ai-evals/datasets/test-dataset-1')
+ expect(router.actions.push).toHaveBeenCalledWith(
+ '/ai-evals/evaluations/offline/datasets/test-dataset-1'
+ )
})
})
diff --git a/products/ai_observability/frontend/evaluations/EvaluationsTabs.tsx b/products/ai_observability/frontend/evaluations/EvaluationsTabs.tsx
index d6a9c0cfacf3..35bf3dfa7414 100644
--- a/products/ai_observability/frontend/evaluations/EvaluationsTabs.tsx
+++ b/products/ai_observability/frontend/evaluations/EvaluationsTabs.tsx
@@ -37,10 +37,16 @@ export function EvaluationsTabs({
...accessTab(Scene.AIObservabilityEvaluation, urls.aiObservabilityEvaluations()),
'data-attr': 'evaluations-tab',
},
- !!featureFlags[FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS] && {
+ (!!featureFlags[FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS] ||
+ !!featureFlags[FEATURE_FLAGS.LLM_ANALYTICS_DATASETS]) && {
key: 'offline-evals',
label: 'Offline evals',
- ...accessTab(Scene.AIObservabilityOfflineExperiments, urls.aiObservabilityOfflineEvaluations()),
+ ...accessTab(
+ Scene.AIObservabilityOfflineExperiments,
+ featureFlags[FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS]
+ ? urls.aiObservabilityOfflineEvaluations()
+ : urls.aiObservabilityDatasets()
+ ),
'data-attr': 'offline-evals-tab',
},
{
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsScene.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsScene.tsx
new file mode 100644
index 000000000000..64bb3d55ddde
--- /dev/null
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsScene.tsx
@@ -0,0 +1,19 @@
+import type { ReactNode } from 'react'
+
+import { SceneContent } from '~/layout/scenes/components/SceneContent'
+import { SceneTitleSection } from '~/layout/scenes/components/SceneTitleSection'
+
+import { EvaluationsTabs } from '../evaluations/EvaluationsTabs'
+
+export function OfflineEvaluationsScene({ children }: { children: ReactNode }): JSX.Element {
+ return (
+ <SceneContent>
+ <SceneTitleSection
+ name="Evaluations"
+ description="Manage datasets, follow score trends, and inspect offline experiment results."
+ resourceType={{ type: 'llm_evaluations' }}
+ />
+ <EvaluationsTabs activeTab="offline-evals">{children}</EvaluationsTabs>
+ </SceneContent>
+ )
+}
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsTabs.stories.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsTabs.stories.tsx
new file mode 100644
index 000000000000..b4cd0ef68298
--- /dev/null
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsTabs.stories.tsx
@@ -0,0 +1,49 @@
+import type { Meta, StoryObj } from '@storybook/react'
+import { within } from '@testing-library/dom'
+
+import { FEATURE_FLAGS } from 'lib/constants'
+import { urls } from 'scenes/urls'
+
+import { OfflineEvaluationsTabs } from './OfflineEvaluationsTabs'
+import { OfflineExperimentsScene } from './OfflineExperimentsScene'
+
+const meta: Meta<typeof OfflineEvaluationsTabs> = {
+ title: 'Scenes-App/AI observability/Offline experiments/Navigation',
+ component: OfflineEvaluationsTabs,
+ args: { activeTab: 'datasets' },
+ parameters: {
+ pageUrl: urls.aiObservabilityDataset('example-dataset'),
+ featureFlags: [FEATURE_FLAGS.LLM_ANALYTICS_DATASETS, FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS],
+ },
+}
+export default meta
+
+type Story = StoryObj<typeof OfflineEvaluationsTabs>
+
+export const DatasetDetail: Story = {}
+export const DatasetsOnly: Story = {
+ parameters: {
+ featureFlags: {
+ [FEATURE_FLAGS.LLM_ANALYTICS_DATASETS]: true,
+ [FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS]: false,
+ },
+ },
+}
+export const DatasetsUnavailable: Story = {
+ render: () => <OfflineExperimentsScene />,
+ parameters: {
+ pageUrl: urls.aiObservabilityDatasets(),
+ featureFlags: {
+ [FEATURE_FLAGS.LLM_ANALYTICS_DATASETS]: false,
+ [FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS]: true,
+ },
+ },
+ play: async ({ canvasElement }) => {
+ const canvas = within(canvasElement)
+ await canvas.findByText('Datasets are coming soon.')
+ const datasetsTab = canvas.getByRole('button', { name: 'Datasets' })
+ if (!datasetsTab.hasAttribute('disabled') && datasetsTab.getAttribute('aria-disabled') !== 'true') {
+ throw new Error('Datasets must stay disabled when the feature flag is off')
+ }
+ },
+}
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsTabs.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsTabs.tsx
new file mode 100644
index 000000000000..a1a7adadb933
--- /dev/null
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsTabs.tsx
@@ -0,0 +1,52 @@
+import { useActions, useValues } from 'kea'
+
+import { LemonSegmentedButton, LemonTag } from '@posthog/lemon-ui'
+
+import { getProductAccessDisabledReason } from 'lib/utils/accessControlUtils'
+import { Scene } from 'scenes/sceneTypes'
+
+import { OfflineEvaluationsTab, offlineEvaluationsLogic } from './offlineEvaluationsLogic'
+
+export function OfflineEvaluationsTabs({ activeTab }: { activeTab: OfflineEvaluationsTab }): JSX.Element {
+ const { datasetsEnabled, offlineEvaluationsEnabled } = useValues(offlineEvaluationsLogic)
+ const { setActiveTab } = useActions(offlineEvaluationsLogic)
+
+ return (
+ <LemonSegmentedButton
+ size="small"
+ value={activeTab}
+ onChange={setActiveTab}
+ options={[
+ {
+ value: 'experiments',
+ label: (
+ <span className="flex items-center gap-1">
+ <span>Experiments</span>
+ {!offlineEvaluationsEnabled && (
+ <LemonTag type="muted" size="small">
+ Coming soon
+ </LemonTag>
+ )}
+ </span>
+ ),
+ disabledReason: !offlineEvaluationsEnabled
+ ? 'Offline experiments are coming soon.'
+ : getProductAccessDisabledReason({
+ sceneKey: Scene.AIObservabilityOfflineExperiment,
+ displayLabel: 'experiments',
+ }),
+ },
+ {
+ value: 'datasets',
+ label: 'Datasets',
+ disabledReason: !datasetsEnabled
+ ? 'Datasets are coming soon.'
+ : getProductAccessDisabledReason({
+ sceneKey: Scene.AIObservabilityDatasets,
+ displayLabel: 'datasets',
+ }),
+ },
+ ]}
+ />
+ )
+}
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentScene.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentScene.tsx
index a0ddd147c5d6..796643fa8ee7 100644
--- a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentScene.tsx
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentScene.tsx
@@ -3,12 +3,11 @@ import { useValues } from 'kea'
import { SceneExport } from 'scenes/sceneTypes'
import { teamLogic } from 'scenes/teamLogic'
-import { SceneContent } from '~/layout/scenes/components/SceneContent'
-import { SceneTitleSection } from '~/layout/scenes/components/SceneTitleSection'
import { ProductKey } from '~/queries/schema/schema-general'
-import { EvaluationsTabs } from '../evaluations/EvaluationsTabs'
import { OfflineEvaluationsGate } from './OfflineEvaluationsGate'
+import { OfflineEvaluationsScene } from './OfflineEvaluationsScene'
+import { OfflineEvaluationsTabs } from './OfflineEvaluationsTabs'
import { OfflineExperimentContent } from './OfflineExperimentContent'
export const scene: SceneExport<{ experimentId: string }> = {
@@ -20,19 +19,19 @@ export const scene: SceneExport<{ experimentId: string }> = {
export function OfflineExperimentScene({ experimentId }: { experimentId: string }): JSX.Element {
const { currentTeamId } = useValues(teamLogic)
return (
- <SceneContent>
- <SceneTitleSection name="Offline experiment" resourceType={{ type: 'evaluation' }} />
- <EvaluationsTabs activeTab="offline-evals">
- <OfflineEvaluationsGate isAvailable={!!currentTeamId}>
- {currentTeamId && (
- <OfflineExperimentContent
- key={`${currentTeamId}:${experimentId}`}
- teamId={currentTeamId}
- experimentId={experimentId}
- />
- )}
- </OfflineEvaluationsGate>
- </EvaluationsTabs>
- </SceneContent>
+ <OfflineEvaluationsScene>
+ <div>
+ <OfflineEvaluationsTabs activeTab="experiments" />
+ </div>
+ <OfflineEvaluationsGate isAvailable={!!currentTeamId}>
+ {currentTeamId && (
+ <OfflineExperimentContent
+ key={`${currentTeamId}:${experimentId}`}
+ teamId={currentTeamId}
+ experimentId={experimentId}
+ />
+ )}
+ </OfflineEvaluationsGate>
+ </OfflineEvaluationsScene>
)
}
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.stories.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.stories.tsx
index 612f260e7db0..1b310b1372a1 100644
--- a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.stories.tsx
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.stories.tsx
@@ -1,5 +1,5 @@
import type { Meta, StoryObj } from '@storybook/react'
-import { fireEvent, waitFor } from '@testing-library/dom'
+import { fireEvent, waitFor, within } from '@testing-library/dom'
import { playHoverAtFraction } from '@posthog/quill-charts/story-helpers'
@@ -58,7 +58,35 @@ const meta: Meta<typeof OfflineExperimentsOverview> = {
export default meta
type Story = StoryObj<typeof OfflineExperimentsOverview>
-export const RecentExperiments: Story = {}
+export const RecentExperiments: Story = {
+ play: async ({ canvasElement }) => {
+ const canvas = within(canvasElement)
+ await canvas.findByRole('heading', { name: 'Score trends' })
+ const expandButton = canvas.queryByRole('button', { name: 'Expand score trends' })
+ if (expandButton) {
+ fireEvent.click(expandButton)
+ }
+ const chart = await waitFor(() => {
+ const chart = canvasElement.querySelector('[data-attr="offline-score-trend"]')
+ if (!chart) {
+ throw new Error('Score charts have not loaded')
+ }
+ return chart
+ })
+ fireEvent.click(canvas.getByRole('button', { name: 'Collapse score trends' }))
+ await waitFor(() => {
+ if (!chart.isConnected || !chart.closest('[hidden]')) {
+ throw new Error('Collapsing score trends must hide the charts without unmounting them')
+ }
+ })
+ fireEvent.click(canvas.getByRole('button', { name: 'Expand score trends' }))
+ await waitFor(() => {
+ if (!chart.isConnected || chart.closest('[hidden]')) {
+ throw new Error('Expanding score trends must show the already mounted charts')
+ }
+ })
+ },
+}
export const PartialHistory: Story = {
parameters: { pageUrl: `${urls.aiObservabilityOfflineEvaluations()}?scores=${overviewScorers[0].id}` },
decorators: [
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.tsx
index cfb2bbde679f..5662550e4434 100644
--- a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.tsx
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.tsx
@@ -1,6 +1,7 @@
import { useActions, useValues } from 'kea'
+import type { ReactNode } from 'react'
-import { IconRefresh, IconSearch } from '@posthog/icons'
+import { IconChevronRight, IconRefresh, IconSearch } from '@posthog/icons'
import {
LemonBanner,
LemonButton,
@@ -32,7 +33,10 @@ import { offlineRunSourceLabel } from './offlineOverviewState'
import { OfflineOverviewTrend } from './OfflineOverviewTrend'
import { OfflineScoreChooser } from './OfflineScoreChooser'
-export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps): JSX.Element {
+export function OfflineExperimentsOverview({
+ navigation,
+ ...props
+}: OfflineExperimentsLogicProps & { navigation?: ReactNode }): JSX.Element {
const logic = offlineExperimentsLogic(props)
const {
experiments,
@@ -48,8 +52,11 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):
dateRange,
cursorStack,
refreshKey,
+ scoreTrendsExpanded,
+ scoreTrendsInitialized,
} = useValues(logic)
- const { setFilters, openChooser, nextPage, previousPage, refresh, loadOfflineSuggestedScorers } = useActions(logic)
+ const { setFilters, openChooser, nextPage, previousPage, refresh, loadOfflineSuggestedScorers, toggleScoreTrends } =
+ useActions(logic)
const columns: LemonTableColumns<OfflineExperimentReadApi> = [
{
title: 'Experiment',
@@ -60,7 +67,7 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):
<Tooltip title={`${experiment.name} · ${experiment.id}`}>
<Link
to={urls.aiObservabilityOfflineEvaluationExperiment(experiment.id)}
- className="block font-semibold truncate"
+ className="block font-semibold truncate @max-[50rem]:max-w-40"
data-attr="offline-experiment-open"
>
{experiment.name}
@@ -74,6 +81,7 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):
width: '19%',
render: (_, experiment) => (
<ResourceTags
+ compact
entity="experiments"
resourceId={experiment.id}
tags={experiment.tags}
@@ -86,12 +94,13 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):
title: 'Author',
key: 'created_by',
width: '18%',
- className: '[&_.profile-package]:max-w-full [&_.profile-name]:truncate',
+ className:
+ '[&_.profile-package]:max-w-full @max-[50rem]:[&_.profile-package]:max-w-24 [&_.profile-name]:truncate',
render: (_, experiment) =>
experiment.created_by ? (
<ProfilePicture user={experiment.created_by as UserBasicType} size="md" showName />
) : (
- <span className="text-muted">No author</span>
+ <span className="text-muted whitespace-nowrap">No author</span>
),
},
{
@@ -155,17 +164,28 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):
]
if (hasExperiments === null && !experimentsError) {
- return <LemonSkeleton className="h-80" />
+ return (
+ <div className="space-y-4">
+ {navigation}
+ <LemonSkeleton className="h-80" />
+ </div>
+ )
}
if (hasExperiments === false && !experimentsLoading && !experimentsError) {
- return <OfflineExperimentsEmptyState />
+ return (
+ <div className="space-y-4">
+ {navigation}
+ <OfflineExperimentsEmptyState />
+ </div>
+ )
}
return (
<div className="@container space-y-6 min-w-0">
<div aria-label="Experiment filters" className="flex flex-wrap items-center gap-2">
+ {navigation}
<DateFilter
- size="medium"
+ size="small"
dateFrom={filters.date_from || '-30d'}
dateTo={filters.date_to || null}
onChange={(dateFrom, dateTo) =>
@@ -175,6 +195,7 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):
showCustomRelativeRange
/>
<LemonSelect
+ size="small"
value={filters.run_source || ''}
onChange={(run_source) => setFilters({ run_source })}
data-attr="offline-experiments-source-filter"
@@ -187,6 +208,7 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):
]}
/>
<LemonSelect
+ size="small"
value={filters.statuses || ''}
onChange={(statuses) => setFilters({ statuses })}
data-attr="offline-experiments-status-filter"
@@ -197,11 +219,11 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):
{ value: 'failed', label: 'Failed' },
]}
/>
- <ResourceTagFilter filters={cleanTagFilters(filters)} onChange={setFilters} />
+ <ResourceTagFilter size="small" filters={cleanTagFilters(filters)} onChange={setFilters} />
<div data-attr="offline-experiments-author-filter">
<MemberSelect
value={filters.created_by ?? null}
- size="medium"
+ size="small"
defaultLabel="Any author"
extraOptions={[
{
@@ -213,7 +235,7 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):
>
{filters.created_by_is_null
? () => (
- <LemonButton type="secondary" size="medium">
+ <LemonButton type="secondary" size="small">
No author
</LemonButton>
)
@@ -222,6 +244,7 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):
</div>
<LemonButton
type="secondary"
+ size="small"
icon={<IconRefresh />}
onClick={refresh}
loading={experimentsLoading}
@@ -234,52 +257,81 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):
<section aria-label="Score trends" className="space-y-3">
<div className="flex flex-wrap justify-between items-center gap-2">
<div>
- <h2 className="mb-0">Score trends</h2>
- <p className="text-muted text-xs mb-0">
- Scores across the selected experiments. Uploading and failed runs may have partial results.
- </p>
+ <div className="flex items-center gap-1">
+ <h2 className="mb-0">Score trends</h2>
+ <LemonButton
+ icon={<IconChevronRight className={scoreTrendsExpanded ? 'rotate-90' : undefined} />}
+ size="xsmall"
+ className="!bg-transparent"
+ onClick={toggleScoreTrends}
+ aria-label={scoreTrendsExpanded ? 'Collapse score trends' : 'Expand score trends'}
+ aria-expanded={scoreTrendsExpanded}
+ aria-controls="offline-score-trends"
+ />
+ </div>
+ {scoreTrendsExpanded && (
+ <p className="text-muted text-xs mb-0">
+ Scores across the selected experiments. Uploading and failed runs may have partial
+ results.
+ </p>
+ )}
</div>
- <LemonButton type="secondary" size="small" onClick={openChooser} data-attr="offline-choose-scores">
- Choose scores
- </LemonButton>
+ {scoreTrendsExpanded && (
+ <LemonButton
+ type="secondary"
+ size="small"
+ onClick={openChooser}
+ data-attr="offline-choose-scores"
+ >
+ Choose scores
+ </LemonButton>
+ )}
</div>
- {!dateRange ? (
- <LemonBanner type="error">Choose a valid date range.</LemonBanner>
- ) : scorerIds === null || (!scorerIds.length && suggestedScorersLoading) ? (
- <LemonSkeleton className="h-40" />
- ) : scorerIds.length > 0 ? (
- <div className="grid grid-cols-1 @min-[56rem]:grid-cols-3 gap-3">
- {scorerIds.map((scorerId, index) => (
- <OfflineOverviewTrend
- key={scorerId}
- overviewLogic={logic}
- overviewId={logic.key}
- xDomain={trendXDomain}
- colorOffset={index}
- teamId={props.teamId}
- timezone={props.timezone}
- scorerId={scorerId}
- dateFrom={dateRange.dateFrom}
- dateTo={dateRange.dateTo}
- refreshKey={refreshKey}
- filters={trendFilters}
- />
- ))}
+ {scoreTrendsInitialized && (
+ <div id="offline-score-trends" hidden={!scoreTrendsExpanded}>
+ {!dateRange ? (
+ <LemonBanner type="error">Choose a valid date range.</LemonBanner>
+ ) : scorerIds === null || (!scorerIds.length && suggestedScorersLoading) ? (
+ <LemonSkeleton className="h-40" />
+ ) : scorerIds.length > 0 ? (
+ <div className="grid grid-cols-1 @min-[56rem]:grid-cols-3 gap-3">
+ {scorerIds.map((scorerId, index) => (
+ <OfflineOverviewTrend
+ key={scorerId}
+ overviewLogic={logic}
+ overviewId={logic.key}
+ xDomain={trendXDomain}
+ colorOffset={index}
+ teamId={props.teamId}
+ timezone={props.timezone}
+ scorerId={scorerId}
+ dateFrom={dateRange.dateFrom}
+ dateTo={dateRange.dateTo}
+ refreshKey={refreshKey}
+ filters={trendFilters}
+ />
+ ))}
+ </div>
+ ) : suggestedScorersError ? (
+ <LemonBanner
+ type="error"
+ action={{ children: 'Try again', onClick: loadOfflineSuggestedScorers }}
+ >
+ Could not load suggested scores. Try again or choose scores yourself.
+ </LemonBanner>
+ ) : (
+ <LemonBanner type="info" action={{ children: 'Choose scores', onClick: openChooser }}>
+ Choose recurring scores to follow their results over time.
+ </LemonBanner>
+ )}
</div>
- ) : suggestedScorersError ? (
- <LemonBanner type="error" action={{ children: 'Try again', onClick: loadOfflineSuggestedScorers }}>
- Could not load suggested scores. Try again or choose scores yourself.
- </LemonBanner>
- ) : (
- <LemonBanner type="info" action={{ children: 'Choose scores', onClick: openChooser }}>
- Choose recurring scores to follow their results over time.
- </LemonBanner>
)}
</section>
<section aria-label="Recent experiments" className="space-y-3">
<div className="flex flex-wrap justify-between items-center gap-2">
<h2 className="mb-0">Recent experiments</h2>
<LemonInput
+ size="small"
type="search"
prefix={<IconSearch />}
value={filters.search || ''}
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsScene.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsScene.tsx
index c819344d3990..5cce95001c04 100644
--- a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsScene.tsx
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsScene.tsx
@@ -1,15 +1,18 @@
import { useValues } from 'kea'
-import { SceneExport } from 'scenes/sceneTypes'
+import { LemonBanner } from '@posthog/lemon-ui'
+
+import { getProductAccessDisabledReason } from 'lib/utils/accessControlUtils'
+import { Scene, SceneExport } from 'scenes/sceneTypes'
import { teamLogic } from 'scenes/teamLogic'
import { userLogic } from 'scenes/userLogic'
-import { SceneContent } from '~/layout/scenes/components/SceneContent'
-import { SceneTitleSection } from '~/layout/scenes/components/SceneTitleSection'
import { ProductKey } from '~/queries/schema/schema-general'
-import { EvaluationsTabs } from '../evaluations/EvaluationsTabs'
-import { OfflineEvaluationsGate } from './OfflineEvaluationsGate'
+import { AIObservabilityDatasetsScene } from '../datasets/AIObservabilityDatasetsScene'
+import { offlineEvaluationsLogic } from './offlineEvaluationsLogic'
+import { OfflineEvaluationsScene } from './OfflineEvaluationsScene'
+import { OfflineEvaluationsTabs } from './OfflineEvaluationsTabs'
import { OfflineExperimentsOverview } from './OfflineExperimentsOverview'
export const scene: SceneExport = { component: OfflineExperimentsScene, productKey: ProductKey.AI_OBSERVABILITY }
@@ -17,28 +20,46 @@ export const scene: SceneExport = { component: OfflineExperimentsScene, productK
export function OfflineExperimentsScene(): JSX.Element {
const { currentTeam } = useValues(teamLogic)
const { user } = useValues(userLogic)
+ const { activeTab, datasetsEnabled, offlineEvaluationsEnabled } = useValues(offlineEvaluationsLogic)
+ const experimentsDisabledReason = !offlineEvaluationsEnabled
+ ? 'Offline experiments are coming soon.'
+ : getProductAccessDisabledReason({
+ sceneKey: Scene.AIObservabilityOfflineExperiment,
+ displayLabel: 'experiments',
+ })
+ const datasetsDisabledReason = !datasetsEnabled
+ ? 'Datasets are coming soon.'
+ : getProductAccessDisabledReason({
+ sceneKey: Scene.AIObservabilityDatasets,
+ displayLabel: 'datasets',
+ })
+ const activeTabDisabledReason =
+ activeTab === 'datasets'
+ ? datasetsDisabledReason
+ : experimentsDisabledReason ||
+ (!currentTeam || !user ? 'Offline evaluations are not available in this project.' : undefined)
+ const navigation = <OfflineEvaluationsTabs activeTab={activeTab} />
return (
- <SceneContent>
- <SceneTitleSection
- name="Evaluations"
- description="Follow score trends and inspect offline experiment results."
- resourceType={{ type: 'llm_evaluations' }}
- />
- <EvaluationsTabs activeTab="offline-evals">
- <OfflineEvaluationsGate
- isAvailable={!!currentTeam && !!user}
- unavailableMessage="Offline evaluations are not available in this project."
- >
- {currentTeam && user && (
- <OfflineExperimentsOverview
- key={`${currentTeam.id}:${user.id}`}
- teamId={currentTeam.id}
- userId={user.id}
- timezone={currentTeam.timezone}
- />
- )}
- </OfflineEvaluationsGate>
- </EvaluationsTabs>
- </SceneContent>
+ <OfflineEvaluationsScene>
+ {activeTabDisabledReason ? (
+ <>
+ {navigation}
+ <LemonBanner type="info">{activeTabDisabledReason}</LemonBanner>
+ </>
+ ) : activeTab === 'datasets' ? (
+ <AIObservabilityDatasetsScene navigation={navigation} />
+ ) : (
+ currentTeam &&
+ user && (
+ <OfflineExperimentsOverview
+ key={`${currentTeam.id}:${user.id}`}
+ teamId={currentTeam.id}
+ userId={user.id}
+ timezone={currentTeam.timezone}
+ navigation={navigation}
+ />
+ )
+ )}
+ </OfflineEvaluationsScene>
)
}
diff --git a/products/ai_observability/frontend/offline-evaluations/offlineEvaluationsLogic.ts b/products/ai_observability/frontend/offline-evaluations/offlineEvaluationsLogic.ts
new file mode 100644
index 000000000000..a3fa4df0f811
--- /dev/null
+++ b/products/ai_observability/frontend/offline-evaluations/offlineEvaluationsLogic.ts
@@ -0,0 +1,128 @@
+import { MakeLogicType, actions, afterMount, connect, kea, listeners, path, reducers, selectors } from 'kea'
+import { actionToUrl, router, urlToAction } from 'kea-router'
+
+import { FEATURE_FLAGS } from 'lib/constants'
+import { featureFlagLogic, type FeatureFlagsSet } from 'lib/logic/featureFlagLogic'
+import { removeProjectIdIfPresent } from 'lib/utils/kea-router'
+import { urls } from 'scenes/urls'
+
+import { getDatasetListUrl } from '../datasets/aiObservabilityDatasetsLogic'
+
+export type OfflineEvaluationsTab = 'experiments' | 'datasets'
+
+// Generated by kea-typegen. Update if you're an agent, ignore if you're human.
+export interface offlineEvaluationsLogicValues {
+ featureFlags: FeatureFlagsSet // featureFlagLogic
+ receivedFeatureFlags: boolean // featureFlagLogic
+ activeTab: OfflineEvaluationsTab
+ datasetsEnabled: boolean
+ datasetsOnly: boolean
+ offlineEvaluationsEnabled: boolean
+ selectedTab: OfflineEvaluationsTab
+}
+
+// Generated by kea-typegen. Update if you're an agent, ignore if you're human.
+export interface offlineEvaluationsLogicActions {
+ ensureAvailableTab: () => {
+ value: true
+ }
+ setActiveTab: (tab: OfflineEvaluationsTab) => {
+ tab: OfflineEvaluationsTab
+ }
+ syncTabFromUrl: (tab: OfflineEvaluationsTab) => {
+ tab: OfflineEvaluationsTab
+ }
+}
+
+// Generated by kea-typegen. Update if you're an agent, ignore if you're human.
+export interface offlineEvaluationsLogicMeta {
+ __keaTypeGenInternalSelectorTypes: {
+ offlineEvaluationsEnabled: (featureFlags: FeatureFlagsSet) => boolean
+ datasetsEnabled: (featureFlags: FeatureFlagsSet) => boolean
+ datasetsOnly: (datasetsEnabled: boolean, offlineEvaluationsEnabled: boolean) => boolean
+ activeTab: (selectedTab: OfflineEvaluationsTab, datasetsOnly: boolean) => OfflineEvaluationsTab
+ }
+}
+
+export type offlineEvaluationsLogicType = MakeLogicType<
+ offlineEvaluationsLogicValues,
+ offlineEvaluationsLogicActions,
+ Record<string, any>,
+ offlineEvaluationsLogicMeta
+>
+
+export const offlineEvaluationsLogic = kea<offlineEvaluationsLogicType>([
+ path(['products', 'ai_observability', 'offlineEvaluationsLogic']),
+ connect({ values: [featureFlagLogic, ['featureFlags', 'receivedFeatureFlags']] }),
+ actions({
+ setActiveTab: (tab: OfflineEvaluationsTab) => ({ tab }),
+ syncTabFromUrl: (tab: OfflineEvaluationsTab) => ({ tab }),
+ ensureAvailableTab: true,
+ }),
+ reducers({
+ selectedTab: [
+ 'experiments' as OfflineEvaluationsTab,
+ {
+ setActiveTab: (_, { tab }) => tab,
+ syncTabFromUrl: (_, { tab }) => tab,
+ },
+ ],
+ }),
+ selectors({
+ offlineEvaluationsEnabled: [
+ (s) => [s.featureFlags],
+ (featureFlags: FeatureFlagsSet): boolean =>
+ !!featureFlags[FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS],
+ ],
+ datasetsEnabled: [
+ (s) => [s.featureFlags],
+ (featureFlags: FeatureFlagsSet): boolean => !!featureFlags[FEATURE_FLAGS.LLM_ANALYTICS_DATASETS],
+ ],
+ datasetsOnly: [
+ (s) => [s.datasetsEnabled, s.offlineEvaluationsEnabled],
+ (datasetsEnabled: boolean, offlineEvaluationsEnabled: boolean): boolean =>
+ datasetsEnabled && !offlineEvaluationsEnabled,
+ ],
+ activeTab: [
+ (s) => [s.selectedTab, s.datasetsOnly],
+ (selectedTab: OfflineEvaluationsTab, datasetsOnly: boolean): OfflineEvaluationsTab =>
+ datasetsOnly ? 'datasets' : selectedTab,
+ ],
+ }),
+ listeners(({ actions, values }) => ({
+ ensureAvailableTab: () => {
+ if (
+ removeProjectIdIfPresent(router.values.location.pathname) !== urls.aiObservabilityOfflineEvaluations()
+ ) {
+ return
+ }
+ actions.syncTabFromUrl(router.values.searchParams.tab === 'datasets' ? 'datasets' : 'experiments')
+ if (values.receivedFeatureFlags && values.datasetsOnly && router.values.searchParams.tab !== 'datasets') {
+ router.actions.replace(
+ router.values.location.pathname,
+ { ...router.values.searchParams, tab: 'datasets' },
+ router.values.hashParams
+ )
+ }
+ },
+ [featureFlagLogic.actionTypes.setFeatureFlags]: () => actions.ensureAvailableTab(),
+ })),
+ actionToUrl(({ values }) => ({
+ setActiveTab: ({ tab }) => [
+ values.datasetsOnly || tab === 'datasets'
+ ? getDatasetListUrl(
+ removeProjectIdIfPresent(router.values.location.pathname) ===
+ urls.aiObservabilityOfflineEvaluations() && router.values.searchParams.tab !== 'datasets'
+ ? {}
+ : router.values.searchParams
+ )
+ : `${urls.aiObservabilityOfflineEvaluations()}?tab=experiments`,
+ {},
+ router.values.hashParams,
+ ],
+ })),
+ urlToAction(({ actions }) => ({
+ [urls.aiObservabilityOfflineEvaluations()]: () => actions.ensureAvailableTab(),
+ })),
+ afterMount(({ actions }) => actions.ensureAvailableTab()),
+])
diff --git a/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.test.ts b/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.test.ts
index 31848ff476e6..1fb047db36f5 100644
--- a/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.test.ts
+++ b/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.test.ts
@@ -1,11 +1,15 @@
import { combineUrl, router } from 'kea-router'
+import { FEATURE_FLAGS } from 'lib/constants'
+import { featureFlagLogic } from 'lib/logic/featureFlagLogic'
+import { removeProjectIdIfPresent } from 'lib/utils/kea-router'
import { urls } from 'scenes/urls'
import { initKeaTests } from '~/test/init'
import * as api from '../generated/api'
import type { OfflineExperimentPageApi } from '../generated/api.schemas'
+import { offlineEvaluationsLogic } from './offlineEvaluationsLogic'
import { offlineExperimentsLogic } from './offlineExperimentsLogic'
import { overviewExperiments, overviewScorers, overviewHistory } from './offlineOverviewFixtures'
import { readOfflineScorerPreferences, saveOfflineScorerPreferences } from './offlineOverviewState'
@@ -53,6 +57,113 @@ describe('offlineExperimentsLogic', () => {
afterEach(() => jest.useRealTimers())
+ it.each(['new', 'dataset-1'])(
+ 'keeps the selected sub-tab and destination when opening dataset %s',
+ async (datasetId) => {
+ featureFlagLogic.mount()
+ featureFlagLogic.actions.setFeatureFlags([], {
+ [FEATURE_FLAGS.LLM_ANALYTICS_DATASETS]: true,
+ [FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS]: true,
+ })
+ const tabs = offlineEvaluationsLogic()
+ tabs.mount()
+ expect(tabs.values.activeTab).toBe('experiments')
+ const logic = offlineExperimentsLogic(props)
+ logic.mount()
+ tabs.actions.setActiveTab('datasets')
+ await jest.advanceTimersByTimeAsync(400)
+ expect(tabs.values.activeTab).toBe('datasets')
+ expect(router.values.searchParams.tab).toBe('datasets')
+ router.actions.push(urls.aiObservabilityDataset(datasetId), { search: 'regression' })
+ expect(tabs.values.activeTab).toBe('datasets')
+ logic.actions.suggestScorerIds([overviewScorers[0].id])
+ expect(removeProjectIdIfPresent(router.values.location.pathname)).toBe(
+ urls.aiObservabilityDataset(datasetId)
+ )
+ expect(router.values.searchParams).toEqual({ search: 'regression' })
+ tabs.actions.setActiveTab('datasets')
+ expect(router.values.searchParams).toMatchObject({ tab: 'datasets', search: 'regression' })
+ router.actions.push(urls.aiObservabilityOfflineEvaluations(), { tab: 'experiments' })
+ expect(tabs.values.activeTab).toBe('experiments')
+ expect(router.values.searchParams.tab).toBe('experiments')
+ router.actions.push(urls.aiObservabilityOfflineEvaluations(), { tab: 'unknown' })
+ expect(tabs.values.activeTab).toBe('experiments')
+ }
+ )
+
+ it.each([undefined, 'experiments', 'datasets'])(
+ 'opens datasets and preserves filters when offline evals are disabled and tab is %s',
+ (tab) => {
+ featureFlagLogic.mount()
+ featureFlagLogic.actions.setFeatureFlags([], {
+ [FEATURE_FLAGS.LLM_ANALYTICS_DATASETS]: true,
+ [FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS]: false,
+ })
+ const filters = { tab, search: 'regression', dataset_status: 'archived', datasets_page: 2 }
+ router.actions.push(urls.aiObservabilityOfflineEvaluations(), filters, { panel: 'datasets' })
+ const tabs = offlineEvaluationsLogic()
+ tabs.mount()
+
+ expect(tabs.values.activeTab).toBe('datasets')
+ expect(router.values.searchParams).toEqual({ ...filters, tab: 'datasets' })
+ expect(router.values.hashParams).toEqual({ panel: 'datasets' })
+ expect(router.values.lastMethod).toBe(tab === 'datasets' ? 'PUSH' : 'REPLACE')
+
+ router.actions.push(urls.aiObservabilityOfflineEvaluations(), { tab: 'experiments' })
+ expect(router.values.searchParams.tab).toBe('datasets')
+ tabs.actions.setActiveTab('experiments')
+ expect(tabs.values.activeTab).toBe('datasets')
+ expect(router.values.searchParams.tab).toBe('datasets')
+ }
+ )
+
+ it('redirects after feature flags arrive without redirecting unrelated pages', () => {
+ const tabs = offlineEvaluationsLogic()
+ tabs.mount()
+ expect(router.values.searchParams.tab).toBeUndefined()
+ featureFlagLogic.actions.setFeatureFlags([], { [FEATURE_FLAGS.LLM_ANALYTICS_DATASETS]: true })
+ expect(tabs.values.activeTab).toBe('datasets')
+ expect(router.values.searchParams.tab).toBe('datasets')
+
+ router.actions.push(urls.aiObservabilityScorers())
+ const pathname = router.values.location.pathname
+ featureFlagLogic.actions.setFeatureFlags([], { [FEATURE_FLAGS.LLM_ANALYTICS_DATASETS]: true })
+ expect(router.values.location.pathname).toBe(pathname)
+ expect(router.values.searchParams.tab).toBeUndefined()
+ })
+
+ it('remembers collapsed score trends and defers score requests until the first expansion', async () => {
+ const logic = offlineExperimentsLogic(props)
+ const unmount = logic.mount()
+ expect(logic.values.scoreTrendsExpanded).toBe(true)
+ logic.actions.toggleScoreTrends()
+ unmount()
+ router.actions.replace(urls.aiObservabilityOfflineEvaluations(), { scores: '' })
+ jest.clearAllMocks()
+ const remounted = offlineExperimentsLogic(props)
+ remounted.mount()
+ expect(remounted.values.scoreTrendsExpanded).toBe(false)
+ await jest.advanceTimersByTimeAsync(400)
+ expect(list).toHaveBeenCalledTimes(1)
+ expect(api.aiObservabilityOfflineExperimentsScorerSummariesList).not.toHaveBeenCalled()
+ expect(api.llmAnalyticsScoreDefinitionsList).not.toHaveBeenCalled()
+
+ remounted.actions.setFilters({ run_source: 'ci' })
+ remounted.actions.refresh()
+ await jest.advanceTimersByTimeAsync(400)
+ expect(api.aiObservabilityOfflineExperimentsScorerSummariesList).not.toHaveBeenCalled()
+ expect(api.llmAnalyticsScoreDefinitionsList).not.toHaveBeenCalled()
+
+ remounted.actions.toggleScoreTrends()
+ await jest.advanceTimersByTimeAsync(400)
+ expect(api.aiObservabilityOfflineExperimentsScorerSummariesList).toHaveBeenCalledTimes(1)
+ expect(remounted.values.scorerIds).toEqual(overviewScorers.map(({ id }) => id))
+ remounted.actions.toggleScoreTrends()
+ remounted.actions.toggleScoreTrends()
+ await jest.advanceTimersByTimeAsync(400)
+ expect(api.aiObservabilityOfflineExperimentsScorerSummariesList).toHaveBeenCalledTimes(1)
+ })
+
it('shares experiment tag filters with trends and restores them from the URL', async () => {
const logic = offlineExperimentsLogic(props)
logic.mount()
diff --git a/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.ts b/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.ts
index f5941af6f903..b880fb26d040 100644
--- a/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.ts
+++ b/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.ts
@@ -15,6 +15,7 @@ import { loaders } from 'kea-loaders'
import { actionToUrl, router, urlToAction } from 'kea-router'
import { dayjs } from 'lib/dayjs'
+import { removeProjectIdIfPresent } from 'lib/utils/kea-router'
import { urls } from 'scenes/urls'
import { cleanTagFilters, restoreTagFilters, saveTagFilters } from '../components/resourceTagFilters'
@@ -70,6 +71,8 @@ export interface offlineExperimentsLogicValues {
} | null
refreshKey: number
resolvedNow: string
+ scoreTrendsExpanded: boolean
+ scoreTrendsInitialized: boolean
scorerIds: string[] | null
scorerOffset: number
scorerOptions: PaginatedScoreDefinitionListApi | null
@@ -95,6 +98,9 @@ export interface offlineExperimentsLogicActions {
hydrateUrl: (search: Record<string, unknown>) => {
search: Record<string, unknown>
}
+ initializeScoreTrends: () => {
+ value: true
+ }
loadOfflineExperiments: (_?: any) => any
loadOfflineExperimentsFailure: (
error: string,
@@ -186,6 +192,9 @@ export interface offlineExperimentsLogicActions {
suggestScorerIds: (scorerIds: string[]) => {
scorerIds: string[]
}
+ toggleScoreTrends: () => {
+ value: true
+ }
}
// Generated by kea-typegen. Update if you're an agent, ignore if you're human.
@@ -218,6 +227,8 @@ export const offlineExperimentsLogic: LogicWrapper<offlineExperimentsLogicType>
key(({ teamId, userId }) => `${teamId}:${userId}`),
path((key) => ['products', 'ai_observability', 'offlineExperimentsLogic', key]),
actions({
+ toggleScoreTrends: true,
+ initializeScoreTrends: true,
setHoveredTrend: (chartId: string, timestamp: number) => ({ chartId, timestamp }),
clearHoveredTrend: (chartId: string) => ({ chartId }),
setFilters: (filters: Partial<OfflineExperimentFilters>) => ({ filters }),
@@ -340,6 +351,8 @@ export const offlineExperimentsLogic: LogicWrapper<offlineExperimentsLogicType>
],
})),
reducers(({ props }) => ({
+ scoreTrendsExpanded: [true, { persist: true }, { toggleScoreTrends: (expanded) => !expanded }],
+ scoreTrendsInitialized: [false, { initializeScoreTrends: () => true }],
filters: [
{ date_from: '-30d' } as OfflineExperimentFilters,
{
@@ -514,17 +527,32 @@ export const offlineExperimentsLogic: LogicWrapper<offlineExperimentsLogicType>
],
}),
listeners(({ actions, values, props }) => ({
+ toggleScoreTrends: () => {
+ if (values.scoreTrendsExpanded && !values.scoreTrendsInitialized) {
+ actions.initializeScoreTrends()
+ }
+ },
+ initializeScoreTrends: () => {
+ if (!values.scorerIds?.length && values.dateRange) {
+ actions.loadOfflineSuggestedScorers()
+ }
+ },
setFilters: () => {
saveTagFilters('experiments', cleanTagFilters(values.filters), props.teamId, props.userId)
actions.loadOfflineExperiments()
- if (!values.scorerIds?.length && values.dateRange && values.suggestedScorers === null) {
+ if (
+ values.scoreTrendsInitialized &&
+ !values.scorerIds?.length &&
+ values.dateRange &&
+ values.suggestedScorers === null
+ ) {
actions.loadOfflineSuggestedScorers()
}
},
hydrateUrl: () => {
saveTagFilters('experiments', cleanTagFilters(values.filters), props.teamId, props.userId)
actions.loadOfflineExperiments()
- if (!values.scorerIds?.length && values.dateRange) {
+ if (values.scoreTrendsInitialized && !values.scorerIds?.length && values.dateRange) {
if (values.suggestedScorers) {
actions.suggestScorerIds(values.suggestedScorers)
} else {
@@ -538,7 +566,7 @@ export const offlineExperimentsLogic: LogicWrapper<offlineExperimentsLogicType>
previousPage: () => actions.loadOfflineExperiments(),
refresh: () => {
actions.loadOfflineExperiments()
- if (!values.scorerIds?.length && values.dateRange) {
+ if (values.scoreTrendsInitialized && !values.scorerIds?.length && values.dateRange) {
actions.loadOfflineSuggestedScorers()
}
},
@@ -565,23 +593,35 @@ export const offlineExperimentsLogic: LogicWrapper<offlineExperimentsLogicType>
},
setScorerIds: ({ scorerIds }) => {
saveOfflineScorerPreferences(props.userId, props.teamId, scorerIds)
- if (!scorerIds.length && values.dateRange) {
+ if (values.scoreTrendsInitialized && !scorerIds.length && values.dateRange) {
actions.loadOfflineSuggestedScorers()
}
},
})),
actionToUrl(({ values }) => {
- const location = (): [string, Record<string, unknown>, Record<string, unknown>, { replace: boolean }] => [
- urls.aiObservabilityOfflineEvaluations(),
- {
- ...offlineFiltersToUrl(values.filters),
- scores: values.scorerIds?.join(','),
- cursor_stack: values.cursorStack.length ? values.cursorStack : undefined,
- list_now: values.cursorStack.length ? values.resolvedNow : undefined,
- },
- {},
- { replace: true },
- ]
+ const location = ():
+ | [string, Record<string, unknown>, Record<string, unknown>, { replace: boolean }]
+ | undefined => {
+ if (
+ removeProjectIdIfPresent(router.values.location.pathname) !==
+ urls.aiObservabilityOfflineEvaluations() ||
+ router.values.searchParams.tab === 'datasets'
+ ) {
+ return
+ }
+ return [
+ urls.aiObservabilityOfflineEvaluations(),
+ {
+ tab: router.values.searchParams.tab === 'experiments' ? 'experiments' : undefined,
+ ...offlineFiltersToUrl(values.filters),
+ scores: values.scorerIds?.join(','),
+ cursor_stack: values.cursorStack.length ? values.cursorStack : undefined,
+ list_now: values.cursorStack.length ? values.resolvedNow : undefined,
+ },
+ {},
+ { replace: true },
+ ]
+ }
return {
setFilters: location,
hydrateUrl: location,
@@ -594,10 +634,15 @@ export const offlineExperimentsLogic: LogicWrapper<offlineExperimentsLogicType>
}),
urlToAction(({ actions }) => ({
[urls.aiObservabilityOfflineEvaluations()]: (_, search, __, { method }) => {
- if (method !== 'REPLACE') {
+ if (method !== 'REPLACE' && search.tab !== 'datasets') {
actions.hydrateUrl(search)
}
},
})),
- afterMount(({ actions }) => actions.hydrateUrl(router.values.searchParams)),
+ afterMount(({ actions, values }) => {
+ actions.hydrateUrl(router.values.searchParams)
+ if (values.scoreTrendsExpanded) {
+ actions.initializeScoreTrends()
+ }
+ }),
])
diff --git a/products/ai_observability/manifest.tsx b/products/ai_observability/manifest.tsx
index 114916250d3f..54a261a09a5f 100644
--- a/products/ai_observability/manifest.tsx
+++ b/products/ai_observability/manifest.tsx
@@ -193,8 +193,7 @@ export const manifest: ProductManifest = {
'/ai-observability/clusters': ['AIObservabilityClusters', 'aiObservabilityClusters'],
'/ai-observability/clusters/:runId': ['AIObservabilityClusters', 'aiObservabilityClusters'],
[AI_OBSERVABILITY_CLUSTER_URL_PATTERN]: ['AIObservabilityCluster', 'aiObservabilityCluster'],
- '/ai-evals/datasets': ['AIObservabilityDatasets', 'aiObservabilityDatasets'],
- '/ai-evals/datasets/:id': ['AIObservabilityDataset', 'aiObservabilityDataset'],
+ '/ai-evals/evaluations/offline/datasets/:id': ['AIObservabilityDataset', 'aiObservabilityDataset'],
'/ai-evals/taggers': ['AIObservabilityTags', 'aiObservabilityTags'],
'/ai-evals/taggers/:id': ['AIObservabilityTag', 'aiObservabilityTag'],
'/ai-evals/evaluations': ['AIObservabilityEvaluations', 'aiObservabilityEvaluations'],
@@ -238,6 +237,10 @@ export const manifest: ProductManifest = {
},
'/ai-evals': (_params, searchParams, hashParams) =>
combineUrl(urls.aiObservabilityEvaluations(), searchParams, hashParams).url,
+ '/ai-evals/datasets': (_params, searchParams, hashParams) =>
+ combineUrl(urls.aiObservabilityDatasets(), { ...searchParams, tab: 'datasets' }, hashParams).url,
+ '/ai-evals/datasets/:id': (params, searchParams, hashParams) =>
+ combineUrl(urls.aiObservabilityDataset(params.id), searchParams, hashParams).url,
'/ai-evals/evaluations/offline': (_params, searchParams, hashParams) =>
combineUrl(urls.aiObservabilityOfflineEvaluations(), searchParams, hashParams).url,
'/ai-evals/tags': (_params, searchParams, hashParams) =>
@@ -297,7 +300,7 @@ export const manifest: ProductManifest = {
'/llm-analytics/clusters/:runId/:clusterId': (params, searchParams, hashParams) =>
combineUrl(urls.aiObservabilityCluster(params.runId, params.clusterId), searchParams, hashParams).url,
'/llm-analytics/datasets': (_params, searchParams, hashParams) =>
- combineUrl(urls.aiObservabilityDatasets(), searchParams, hashParams).url,
+ combineUrl(urls.aiObservabilityDatasets(), { ...searchParams, tab: 'datasets' }, hashParams).url,
'/llm-analytics/datasets/:id': (params, searchParams, hashParams) =>
combineUrl(urls.aiObservabilityDataset(params.id), searchParams, hashParams).url,
'/llm-analytics/tags': (_params, searchParams, hashParams) =>
@@ -405,9 +408,9 @@ export const manifest: ProductManifest = {
return `/ai-observability/sessions/${id}${stringifiedParams ? `?${stringifiedParams}` : ''}`
},
aiObservabilityPlayground: (): string => '/ai-observability/playground',
- aiObservabilityDatasets: (): string => '/ai-evals/datasets',
+ aiObservabilityDatasets: (): string => '/ai-evals/evaluations/offline/experiments?tab=datasets',
aiObservabilityDataset: (id: string, params?: { item?: string }): string =>
- combineUrl(`/ai-evals/datasets/${id}`, params).url,
+ combineUrl(`/ai-evals/evaluations/offline/datasets/${id}`, params).url,
aiObservabilityTags: (): string => '/ai-evals/taggers',
aiObservabilityTag: (id: string): string => `/ai-evals/taggers/${id}`,
aiObservabilityEvaluations: (): string => '/ai-evals/evaluations',
@@ -482,34 +485,22 @@ export const manifest: ProductManifest = {
searchKeywords: ['topics'],
sceneKey: 'AIObservabilityClusters',
},
- {
- path: 'Datasets',
- intents: [ProductKey.LLM_DATASETS],
- category: ProductItemCategory.AI_ENGINEERING,
- type: 'llm_datasets',
- iconType: 'llm_datasets' as FileSystemIconType,
- iconColor: ['var(--color-product-llm-datasets-light)', 'var(--color-product-llm-datasets-dark)'],
- href: urls.aiObservabilityDatasets(),
- searchKeywords: ['test sets', 'golden set'],
- flag: FEATURE_FLAGS.LLM_ANALYTICS_DATASETS,
- tags: ['beta'],
- sceneKey: 'AIObservabilityDatasets',
- },
{
path: 'Evaluations',
- intents: [ProductKey.LLM_EVALUATIONS],
+ intents: [ProductKey.LLM_EVALUATIONS, ProductKey.LLM_DATASETS],
category: ProductItemCategory.AI_ENGINEERING,
type: 'llm_evaluations',
iconType: 'llm_evaluations' as FileSystemIconType,
iconColor: ['var(--color-product-llm-evaluations-light)', 'var(--color-product-llm-evaluations-dark)'],
href: urls.aiObservabilityEvaluations(),
- searchKeywords: ['llm judge', 'llm as a judge', 'grading'],
+ searchKeywords: ['llm judge', 'llm as a judge', 'grading', 'datasets', 'test sets', 'golden set'],
searchTabs: [
{
name: 'Offline evals',
href: urls.aiObservabilityOfflineEvaluations(),
flag: FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS,
},
+ { name: 'Datasets', href: urls.aiObservabilityDatasets(), flag: FEATURE_FLAGS.LLM_ANALYTICS_DATASETS },
{ name: 'Scorers', href: urls.aiObservabilityScorers() },
],
sceneKey: 'AIObservabilityEvaluations',
diff --git a/services/mcp/src/tools/links/app-url-manifest.json b/services/mcp/src/tools/links/app-url-manifest.json
index adeaaac15990..c891160bdfe4 100644
--- a/services/mcp/src/tools/links/app-url-manifest.json
+++ b/services/mcp/src/tools/links/app-url-manifest.json
@@ -70,12 +70,12 @@
"scope": "project"
},
"aiObservabilityDataset": {
- "template": "/ai-evals/datasets/{id}",
+ "template": "/ai-evals/evaluations/offline/datasets/{id}",
"params": ["id"],
"scope": "project"
},
"aiObservabilityDatasets": {
- "template": "/ai-evals/datasets",
+ "template": "/ai-evals/evaluations/offline/experiments?tab=datasets",
"params": [],
"scope": "project"
},
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/generate-app-url.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/generate-app-url.json
index 79845971479d..cf3d494ec3d4 100644
--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/generate-app-url.json
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/generate-app-url.json
@@ -13,7 +13,7 @@
"type": "object"
},
"url": {
- "description": "A path template copied verbatim from the catalog below (e.g. `/persons/{uuid}`). Its `{placeholders}` are filled from `params`. These slugs come from PostHog's canonical route table, so they are always correct — never pass a path that is not in this list.\n\n/account-connected/{kind}\n/account/credential-review\n/account/social-connected\n/activity-logs\n/activity/{tab}\n/agentic/account-mismatch\n/agentic/authorize\n/ai\n/ai-enrichment\n/ai-evals/datasets\n/ai-evals/datasets/{id}\n/ai-evals/evaluations\n/ai-evals/evaluations/offline/experiments\n/ai-evals/evaluations/offline/experiments/{experimentId}\n/ai-evals/evaluations/scorers\n/ai-evals/evaluations/scorers/{scorerId}\n/ai-evals/evaluations/scorers/{scorerId}/offline\n/ai-evals/evaluations/templates\n/ai-evals/evaluations/{id}\n/ai-evals/taggers\n/ai-evals/taggers/{id}\n/ai-observability/clusters\n/ai-observability/clusters/{runId}/{clusterId}\n/ai-observability/dashboard\n/ai-observability/errors\n/ai-observability/generations\n/ai-observability/playground\n/ai-observability/reviews\n/ai-observability/self-driving\n/ai-observability/sentiment\n/ai-observability/sessions\n/ai-observability/sessions/{id}\n/ai-observability/tools\n/ai-observability/traces\n/ai-observability/traces/{id}\n/ai-observability/users\n/ai/history\n/alerts\n/analytics\n/analytics/all\n/analytics/mine\n/analytics/product-analytics\n/analytics/spaces\n/approvals/{id}\n/autoresearch\n/autoresearch/new\n/autoresearch/{id}\n/bi\n/bi/new\n/bi/{insightShortId}\n/billing/authorization_status\n/broadcasts\n/broadcasts/new\n/broadcasts/{id}\n/business-knowledge\n/business-knowledge/playground\n/business-knowledge/settings\n/business-knowledge/{id}\n/canvas\n/canvases\n/canvases/new\n/canvases/{id}\n/cli/authorize\n/cli/live\n/code-review\n/code/canvas/{channelId}/{dashboardId}\n/code/channel/{channelId}\n/code/loop/{loopId}\n/code/task/{taskId}\n/cohorts\n/cohorts/{id}\n/cohorts/{id}/calculation-history\n/connect/vercel/link\n/coupons/{campaign}\n/create-organization\n/cross-project-dashboards\n/cross-project-dashboards/{id}\n/customer_analytics\n/customer_analytics/accounts\n/customer_analytics/accounts/by-external-id/{externalId}\n/customer_analytics/accounts/{accountId}\n/customer_analytics/announcements\n/customer_analytics/configuration\n/customer_analytics/dashboard\n/customer_analytics/feature-requests\n/customer_analytics/feed\n/customer_analytics/journeys\n/customer_analytics/journeys/new\n/customer_analytics/journeys/templates\n/customer_analytics/journeys/{id}/edit\n/customer_analytics/notes\n/customer_analytics/tasks\n/dashboard\n/dashboard/templates/{templateId}/copy-to-project\n/dashboard/{id}\n/dashboard/{id}/sharing\n/dashboard/{id}/subscriptions\n/dashboard/{id}/subscriptions/{subscriptionId}\n/dashboard/{id}/tiles/{tileId}\n/data-catalog\n/data-catalog/metrics/{name}\n/data-management/actions\n/data-management/actions/new\n/data-management/actions/new/\n/data-management/actions/{id}\n/data-management/annotations\n/data-management/annotations/{id}\n/data-management/core-events\n/data-management/database\n/data-management/destinations\n/data-management/event-filtering\n/data-management/events\n/data-management/events/{id}\n/data-management/events/{id}/edit\n/data-management/history\n/data-management/ingestion-warnings\n/data-management/ingestion-warnings-v2\n/data-management/managed-viewsets\n/data-management/materialized-columns\n/data-management/properties\n/data-management/properties/{id}\n/data-management/properties/{id}/edit\n/data-management/revenue\n/data-management/schema\n/data-management/sources\n/data-management/sources/{id}/schemas\n/data-management/sources/{sourceId}/schemas/{schemaId}\n/data-management/transformations\n/data-management/variables\n/data-management/variables/{id}\n/data-management/variables/{id}/edit\n/data-management/warehouse-destinations\n/data-management/warehouse-properties\n/data-ops\n/data-warehouse/connect\n/data-warehouse/new-source\n/debug\n/debug/hog\n/debug/precompute\n/early_access_features\n/early_access_features/{id}\n/embedded/{token}\n/endpoints\n/endpoints/{name}\n/engineering-analytics/authors\n/engineering-analytics/authors/{handle}\n/engineering-analytics/deploys\n/engineering-analytics/overview\n/engineering-analytics/pull-requests\n/engineering-analytics/repos/{repoOwner}/{repoName}/actions/runs/{runId}\n/engineering-analytics/repos/{repoOwner}/{repoName}/actions/workflows/{workflowName}\n/engineering-analytics/repos/{repoOwner}/{repoName}/pull-requests/{number}\n/engineering-analytics/repos/{repoOwner}/{repoName}/pull-requests/{number}/ci-explorer\n/engineering-analytics/teams\n/engineering-analytics/teams/{ownerTeam}\n/engineering-analytics/tests\n/engineering-analytics/workflows\n/error-tracking\n/error-tracking/alerts/new/{templateId}\n/error-tracking/alerts/{id}\n/error-tracking/fingerprint/{fingerprint}\n/error-tracking/{id}\n/etl\n/events/{id}/{timestamp}\n/experiments\n/experiments/shared-metrics\n/experiments/shared-metrics/{id}\n/experiments/staff\n/experiments/{id}\n/exports\n/feature-flags\n/feature-flags/new\n/feature-flags/staff\n/feature-flags/staff/cohorts\n/feature-flags/templates\n/feature-flags/{id}\n/files\n/functions/new/{templateId}\n/functions/{id}\n/games/368hedgehogs\n/games/flappyhog\n/games/shipit\n/groups/{groupTypeIndex}\n/groups/{groupTypeIndex}/new\n/groups/{groupTypeIndex}/{groupKey}\n/health\n/health/alerts\n/health/pipeline-status\n/health/sdk-health\n/health/{category}\n/heatmaps\n/heatmaps/new\n/heatmaps/recording\n/heatmaps/{id}\n/home\n/identity-matching\n/inbox\n/inbox/reports/triage\n/inbox/scout-trials\n/inbox/scouts/findings\n/inbox/scouts/runs\n/inbox/scouts/scratchpad\n/inbox/scouts/{skillName}\n/inbox/{tab}/{reportId}\n/insights\n/insights/new\n/insights/quick-start\n/insights/{id}\n/insights/{id}/edit\n/insights/{id}/sharing\n/insights/{id}/subscriptions\n/insights/{id}/subscriptions/{subscriptionId}\n/insights/{insightShortId}/alerts\n/instance/async_migrations\n/instance/async_migrations/future\n/instance/async_migrations/settings\n/instance/dead_letter_queue\n/instance/kafka_inspector\n/instance/metrics\n/instance/settings\n/instance/staff_users\n/instance/status\n/integrations/stripe/confirm-install\n/integrations/vercel/link-error\n/integrations/{kind}/callback\n/integrations/{slug}\n/legal\n/legal/new/{type}\n/login\n/login/2fa\n/login/2fa_setup\n/logs\n/logs/alerts/{alertId}/notifications/{hogFunctionId}\n/logs/alerts/{id}\n/logs/drop-rules/new\n/logs/drop-rules/{id}\n/logs/retention-rules/new\n/logs/retention-rules/{id}\n/managed_migrations\n/managed_migrations/new\n/marketing\n/mcp-analytics\n/mcp-analytics/activity\n/mcp-analytics/dashboard\n/mcp-analytics/intent-clustering\n/mcp-analytics/missing-capabilities\n/mcp-analytics/notifications\n/mcp-analytics/sessions\n/mcp-analytics/tool-quality\n/mcp-analytics/tool-quality/{toolName}\n/mcp-registry\n/mcp-servers\n/mcp-servers/agent/{id}\n/mcp-servers/member/{id}\n/mcp-servers/server/{id}\n/mcp-servers/{tab}\n/metrics\n/metrics/dashboard-review\n/models\n/models/{id}\n/move-to-cloud\n/my-tickets\n/notebooks\n/notebooks/widgets/{widgetId}\n/notebooks/{shortId}\n/oauth/authorize\n/onboarding\n/onboarding/simple\n/organization-deactivated\n/organization-pending-deletion\n/organization/billing\n/organization/billing/overview\n/organization/billing/real-time-usage\n/organization/confirm-creation\n/organization/create-project\n/organization/legal\n/organization/startups\n/person/{id}\n/persons\n/persons/{uuid}\n/pipeline/batch-exports/new/{service}\n/pipeline/batch-exports/{id}\n/pipeline/new/\n/pipeline/plugins/{id}\n/platform-alerts\n/platform-alerts/{id}\n/preflight\n/product_tours\n/product_tours/{id}\n/project-pending-deletion\n/prompt-management/prompts\n/prompt-management/prompts/{name}\n/pulse\n/replay-vision\n/replay-vision/new/goal\n/replay-vision/observations/{observationId}\n/replay-vision/{id}/budget\n/replay-vision/{id}/configure\n/replay-vision/{id}/details\n/replay-vision/{id}/goal\n/replay-vision/{id}/overview\n/replay-vision/{id}/triggers\n/replay/file-playback\n/replay/home\n/replay/kiosk\n/replay/playlists/{id}\n/replay/settings\n/replay/{id}\n/reset\n/reset/{userUuid}/{token}\n/reset_2fa/{userUuid}/{token}\n/resource-transfer/{resourceKind}/{resourceId}\n/sessions/{id}\n/settings/environment-approvals\n/settings/organization-authentication/{feature}/{configId}\n/settings/project\n/settings/user-feature-previews\n/shared/{token}\n/shared_dashboard/{shareToken}\n/signup\n/signup/{id}\n/site/{url}\n/skills\n/skills/community\n/skills/{categoryTab}\n/skills/{name}\n/slack-task-context\n/spaces\n/spaces/new\n/spaces/{id}\n/spaces/{id}/canvases\n/spaces/{id}/new\n/spaces/{id}/settings\n/sql\n/stamphog\n/stamphog/digests\n/stamphog/install/callback\n/stamphog/runs\n/startups\n/streamlit-apps\n/streamlit-apps/new\n/streamlit-apps/{id}\n/streamlit-apps/{id}/edit\n/subscriptions\n/subscriptions/new\n/subscriptions/{id}\n/subscriptions/{id}/edit\n/support\n/support/settings\n/support/tickets\n/support/tickets/{ticketId}\n/surveys\n/surveys/form/new\n/surveys/guided/new\n/surveys/{id}\n/tasks\n/tasks/new\n/tasks/{taskId}\n/terminal\n/themes/custom-css\n/toolbar\n/tools\n/tracing\n/tracing/retention-rules/new\n/tracing/retention-rules/{id}\n/unsubscribe\n/uptime-monitoring\n/verify_email\n/visual-review\n/visual-review/repos/{repoOwner}/{repoName}/flakiness\n/visual-review/repos/{repoOwner}/{repoName}/runs\n/visual-review/repos/{repoOwner}/{repoName}/runs/{runId}\n/visual-review/repos/{repoOwner}/{repoName}/snapshots\n/visual-review/repos/{repoOwner}/{repoName}/{runType}/snapshots/{identifier}\n/visual-review/runs/{runId}\n/visual-review/settings\n/web\n/web-scripts\n/web-scripts/new\n/web/agents\n/web/bots\n/web/content-autopilot\n/web/health\n/web/live\n/web/marketing\n/web/page-performance\n/web/page-reports\n/web/recap\n/web/session-attribution-explorer\n/web/web-vitals\n/welcome\n/wizard/runs\n/workflows\n/workflows/library/messages/{id}\n/workflows/library/templates/new\n/workflows/library/templates/{id}\n/workflows/new/workflow\n/workflows/{id}/{tab}",
+ "description": "A path template copied verbatim from the catalog below (e.g. `/persons/{uuid}`). Its `{placeholders}` are filled from `params`. These slugs come from PostHog's canonical route table, so they are always correct — never pass a path that is not in this list.\n\n/account-connected/{kind}\n/account/credential-review\n/account/social-connected\n/activity-logs\n/activity/{tab}\n/agentic/account-mismatch\n/agentic/authorize\n/ai\n/ai-enrichment\n/ai-evals/evaluations\n/ai-evals/evaluations/offline/datasets/{id}\n/ai-evals/evaluations/offline/experiments\n/ai-evals/evaluations/offline/experiments/{experimentId}\n/ai-evals/evaluations/offline/experiments?tab=datasets\n/ai-evals/evaluations/scorers\n/ai-evals/evaluations/scorers/{scorerId}\n/ai-evals/evaluations/scorers/{scorerId}/offline\n/ai-evals/evaluations/templates\n/ai-evals/evaluations/{id}\n/ai-evals/taggers\n/ai-evals/taggers/{id}\n/ai-observability/clusters\n/ai-observability/clusters/{runId}/{clusterId}\n/ai-observability/dashboard\n/ai-observability/errors\n/ai-observability/generations\n/ai-observability/playground\n/ai-observability/reviews\n/ai-observability/self-driving\n/ai-observability/sentiment\n/ai-observability/sessions\n/ai-observability/sessions/{id}\n/ai-observability/tools\n/ai-observability/traces\n/ai-observability/traces/{id}\n/ai-observability/users\n/ai/history\n/alerts\n/analytics\n/analytics/all\n/analytics/mine\n/analytics/product-analytics\n/analytics/spaces\n/approvals/{id}\n/autoresearch\n/autoresearch/new\n/autoresearch/{id}\n/bi\n/bi/new\n/bi/{insightShortId}\n/billing/authorization_status\n/broadcasts\n/broadcasts/new\n/broadcasts/{id}\n/business-knowledge\n/business-knowledge/playground\n/business-knowledge/settings\n/business-knowledge/{id}\n/canvas\n/canvases\n/canvases/new\n/canvases/{id}\n/cli/authorize\n/cli/live\n/code-review\n/code/canvas/{channelId}/{dashboardId}\n/code/channel/{channelId}\n/code/loop/{loopId}\n/code/task/{taskId}\n/cohorts\n/cohorts/{id}\n/cohorts/{id}/calculation-history\n/connect/vercel/link\n/coupons/{campaign}\n/create-organization\n/cross-project-dashboards\n/cross-project-dashboards/{id}\n/customer_analytics\n/customer_analytics/accounts\n/customer_analytics/accounts/by-external-id/{externalId}\n/customer_analytics/accounts/{accountId}\n/customer_analytics/announcements\n/customer_analytics/configuration\n/customer_analytics/dashboard\n/customer_analytics/feature-requests\n/customer_analytics/feed\n/customer_analytics/journeys\n/customer_analytics/journeys/new\n/customer_analytics/journeys/templates\n/customer_analytics/journeys/{id}/edit\n/customer_analytics/notes\n/customer_analytics/tasks\n/dashboard\n/dashboard/templates/{templateId}/copy-to-project\n/dashboard/{id}\n/dashboard/{id}/sharing\n/dashboard/{id}/subscriptions\n/dashboard/{id}/subscriptions/{subscriptionId}\n/dashboard/{id}/tiles/{tileId}\n/data-catalog\n/data-catalog/metrics/{name}\n/data-management/actions\n/data-management/actions/new\n/data-management/actions/new/\n/data-management/actions/{id}\n/data-management/annotations\n/data-management/annotations/{id}\n/data-management/core-events\n/data-management/database\n/data-management/destinations\n/data-management/event-filtering\n/data-management/events\n/data-management/events/{id}\n/data-management/events/{id}/edit\n/data-management/history\n/data-management/ingestion-warnings\n/data-management/ingestion-warnings-v2\n/data-management/managed-viewsets\n/data-management/materialized-columns\n/data-management/properties\n/data-management/properties/{id}\n/data-management/properties/{id}/edit\n/data-management/revenue\n/data-management/schema\n/data-management/sources\n/data-management/sources/{id}/schemas\n/data-management/sources/{sourceId}/schemas/{schemaId}\n/data-management/transformations\n/data-management/variables\n/data-management/variables/{id}\n/data-management/variables/{id}/edit\n/data-management/warehouse-destinations\n/data-management/warehouse-properties\n/data-ops\n/data-warehouse/connect\n/data-warehouse/new-source\n/debug\n/debug/hog\n/debug/precompute\n/early_access_features\n/early_access_features/{id}\n/embedded/{token}\n/endpoints\n/endpoints/{name}\n/engineering-analytics/authors\n/engineering-analytics/authors/{handle}\n/engineering-analytics/deploys\n/engineering-analytics/overview\n/engineering-analytics/pull-requests\n/engineering-analytics/repos/{repoOwner}/{repoName}/actions/runs/{runId}\n/engineering-analytics/repos/{repoOwner}/{repoName}/actions/workflows/{workflowName}\n/engineering-analytics/repos/{repoOwner}/{repoName}/pull-requests/{number}\n/engineering-analytics/repos/{repoOwner}/{repoName}/pull-requests/{number}/ci-explorer\n/engineering-analytics/teams\n/engineering-analytics/teams/{ownerTeam}\n/engineering-analytics/tests\n/engineering-analytics/workflows\n/error-tracking\n/error-tracking/alerts/new/{templateId}\n/error-tracking/alerts/{id}\n/error-tracking/fingerprint/{fingerprint}\n/error-tracking/{id}\n/etl\n/events/{id}/{timestamp}\n/experiments\n/experiments/shared-metrics\n/experiments/shared-metrics/{id}\n/experiments/staff\n/experiments/{id}\n/exports\n/feature-flags\n/feature-flags/new\n/feature-flags/staff\n/feature-flags/staff/cohorts\n/feature-flags/templates\n/feature-flags/{id}\n/files\n/functions/new/{templateId}\n/functions/{id}\n/games/368hedgehogs\n/games/flappyhog\n/games/shipit\n/groups/{groupTypeIndex}\n/groups/{groupTypeIndex}/new\n/groups/{groupTypeIndex}/{groupKey}\n/health\n/health/alerts\n/health/pipeline-status\n/health/sdk-health\n/health/{category}\n/heatmaps\n/heatmaps/new\n/heatmaps/recording\n/heatmaps/{id}\n/home\n/identity-matching\n/inbox\n/inbox/reports/triage\n/inbox/scout-trials\n/inbox/scouts/findings\n/inbox/scouts/runs\n/inbox/scouts/scratchpad\n/inbox/scouts/{skillName}\n/inbox/{tab}/{reportId}\n/insights\n/insights/new\n/insights/quick-start\n/insights/{id}\n/insights/{id}/edit\n/insights/{id}/sharing\n/insights/{id}/subscriptions\n/insights/{id}/subscriptions/{subscriptionId}\n/insights/{insightShortId}/alerts\n/instance/async_migrations\n/instance/async_migrations/future\n/instance/async_migrations/settings\n/instance/dead_letter_queue\n/instance/kafka_inspector\n/instance/metrics\n/instance/settings\n/instance/staff_users\n/instance/status\n/integrations/stripe/confirm-install\n/integrations/vercel/link-error\n/integrations/{kind}/callback\n/integrations/{slug}\n/legal\n/legal/new/{type}\n/login\n/login/2fa\n/login/2fa_setup\n/logs\n/logs/alerts/{alertId}/notifications/{hogFunctionId}\n/logs/alerts/{id}\n/logs/drop-rules/new\n/logs/drop-rules/{id}\n/logs/retention-rules/new\n/logs/retention-rules/{id}\n/managed_migrations\n/managed_migrations/new\n/marketing\n/mcp-analytics\n/mcp-analytics/activity\n/mcp-analytics/dashboard\n/mcp-analytics/intent-clustering\n/mcp-analytics/missing-capabilities\n/mcp-analytics/notifications\n/mcp-analytics/sessions\n/mcp-analytics/tool-quality\n/mcp-analytics/tool-quality/{toolName}\n/mcp-registry\n/mcp-servers\n/mcp-servers/agent/{id}\n/mcp-servers/member/{id}\n/mcp-servers/server/{id}\n/mcp-servers/{tab}\n/metrics\n/metrics/dashboard-review\n/models\n/models/{id}\n/move-to-cloud\n/my-tickets\n/notebooks\n/notebooks/widgets/{widgetId}\n/notebooks/{shortId}\n/oauth/authorize\n/onboarding\n/onboarding/simple\n/organization-deactivated\n/organization-pending-deletion\n/organization/billing\n/organization/billing/overview\n/organization/billing/real-time-usage\n/organization/confirm-creation\n/organization/create-project\n/organization/legal\n/organization/startups\n/person/{id}\n/persons\n/persons/{uuid}\n/pipeline/batch-exports/new/{service}\n/pipeline/batch-exports/{id}\n/pipeline/new/\n/pipeline/plugins/{id}\n/platform-alerts\n/platform-alerts/{id}\n/preflight\n/product_tours\n/product_tours/{id}\n/project-pending-deletion\n/prompt-management/prompts\n/prompt-management/prompts/{name}\n/pulse\n/replay-vision\n/replay-vision/new/goal\n/replay-vision/observations/{observationId}\n/replay-vision/{id}/budget\n/replay-vision/{id}/configure\n/replay-vision/{id}/details\n/replay-vision/{id}/goal\n/replay-vision/{id}/overview\n/replay-vision/{id}/triggers\n/replay/file-playback\n/replay/home\n/replay/kiosk\n/replay/playlists/{id}\n/replay/settings\n/replay/{id}\n/reset\n/reset/{userUuid}/{token}\n/reset_2fa/{userUuid}/{token}\n/resource-transfer/{resourceKind}/{resourceId}\n/sessions/{id}\n/settings/environment-approvals\n/settings/organization-authentication/{feature}/{configId}\n/settings/project\n/settings/user-feature-previews\n/shared/{token}\n/shared_dashboard/{shareToken}\n/signup\n/signup/{id}\n/site/{url}\n/skills\n/skills/community\n/skills/{categoryTab}\n/skills/{name}\n/slack-task-context\n/spaces\n/spaces/new\n/spaces/{id}\n/spaces/{id}/canvases\n/spaces/{id}/new\n/spaces/{id}/settings\n/sql\n/stamphog\n/stamphog/digests\n/stamphog/install/callback\n/stamphog/runs\n/startups\n/streamlit-apps\n/streamlit-apps/new\n/streamlit-apps/{id}\n/streamlit-apps/{id}/edit\n/subscriptions\n/subscriptions/new\n/subscriptions/{id}\n/subscriptions/{id}/edit\n/support\n/support/settings\n/support/tickets\n/support/tickets/{ticketId}\n/surveys\n/surveys/form/new\n/surveys/guided/new\n/surveys/{id}\n/tasks\n/tasks/new\n/tasks/{taskId}\n/terminal\n/themes/custom-css\n/toolbar\n/tools\n/tracing\n/tracing/retention-rules/new\n/tracing/retention-rules/{id}\n/unsubscribe\n/uptime-monitoring\n/verify_email\n/visual-review\n/visual-review/repos/{repoOwner}/{repoName}/flakiness\n/visual-review/repos/{repoOwner}/{repoName}/runs\n/visual-review/repos/{repoOwner}/{repoName}/runs/{runId}\n/visual-review/repos/{repoOwner}/{repoName}/snapshots\n/visual-review/repos/{repoOwner}/{repoName}/{runType}/snapshots/{identifier}\n/visual-review/runs/{runId}\n/visual-review/settings\n/web\n/web-scripts\n/web-scripts/new\n/web/agents\n/web/bots\n/web/content-autopilot\n/web/health\n/web/live\n/web/marketing\n/web/page-performance\n/web/page-reports\n/web/recap\n/web/session-attribution-explorer\n/web/web-vitals\n/welcome\n/wizard/runs\n/workflows\n/workflows/library/messages/{id}\n/workflows/library/templates/new\n/workflows/library/templates/{id}\n/workflows/new/workflow\n/workflows/{id}/{tab}",
"type": "string"
}
},
diff --git a/services/mcp/tests/unit/generate-app-url.test.ts b/services/mcp/tests/unit/generate-app-url.test.ts
index 59d78685e082..4edb7a849d9c 100644
--- a/services/mcp/tests/unit/generate-app-url.test.ts
+++ b/services/mcp/tests/unit/generate-app-url.test.ts
@@ -59,6 +59,17 @@ describe('generate-app-url', () => {
['/feature_flags/{id}', { id: '42' }, '/feature-flags/42'],
['/feature-flags/{id}', { id: '42' }, '/feature-flags/42'],
['/feature_flags', {}, '/feature-flags'],
+ [
+ '/ai-evals/evaluations/offline/experiments?tab=datasets',
+ {},
+ '/ai-evals/evaluations/offline/experiments?tab=datasets',
+ ],
+ ['/ai-evals/evaluations/offline/experiments', {}, '/ai-evals/evaluations/offline/experiments'],
+ [
+ '/ai-evals/evaluations/offline/datasets/{id}',
+ { id: 'example-dataset' },
+ '/ai-evals/evaluations/offline/datasets/example-dataset',
+ ],
])('builds the canonical app link for the template %s', async (url, params, path) => {
const result = await generateAppUrlHandler(ctx, { url, params })
expect(result.url).toBe(`https://us.posthog.com/project/354703${path}`)