:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(aio): move datasets into offline evaluations (#121572)
Radu Raicea35cfd4e69ae8
diff --git a/docs/internal/ai-offline-evaluation-reporting.md b/docs/internal/ai-offline-evaluation-reporting.mdindex 068a31b1c7fa..c6165771a7bb 100644--- a/docs/internal/ai-offline-evaluation-reporting.md+++ b/docs/internal/ai-offline-evaluation-reporting.md@@ -553,6 +553,23 @@ Opening those resources requires their own permissions.## Inspecting offline resultsOpen **Evaluations → Offline evals** to see the newest experiments and chosen score trends.+When offline evaluations are enabled, **Experiments** is selected by default. **Datasets** opens the list on the same page+with `tab=datasets` in the query string. MCP-generated dataset-list links retain this tab selector.+Existing `/ai-evals/datasets` and `/llm-analytics/datasets`+links redirect there, preserving filters and pagination. Dataset creation and detail pages live at+`/ai-evals/evaluations/offline/datasets/:id`, within the Offline evals navigation. Both legacy dataset+detail routes redirect there, preserving item deep links, revisions, tabs, filters, and hashes.+The Evaluations heading and navigation stay in place on dataset creation and detail pages. Dataset items appear+directly in the detail view; **Details** opens the dataset metadata without replacing the items table.+The back arrow beside the dataset name returns to the list and preserves its filters, including after creating a dataset.+Creation and editing actions appear below the dataset form. Switching from Experiments to Datasets clears+experiment search and filters and restores the saved dataset tag selection.+Projects with Datasets enabled but without offline evaluations open **Datasets** automatically,+including from experiment-list links. **Experiments** stays visible with a disabled **Coming soon** label.+When Datasets is disabled, its tab stays disabled and direct list links show an availability message.+The chevron beside **Score trends** collapses or expands the charts. It starts expanded and remembers+your choice in local storage. Charts load on the first expansion if the page opens collapsed, and+keep their data when collapsed and expanded again. Refresh or a filter change updates loaded charts.A shared time range, run source, and upload state filter applies to both the score charts and the experiment list.The overview starts with the last 30 days and all upload states. Uploading and failed runs can show partial score summaries.Projects without experiments show setup steps; a filter with no matching experiments keeps the overview available.diff --git a/frontend/bin/build-app-url-manifest.mjs b/frontend/bin/build-app-url-manifest.mjsindex 265997dabd2a..b99e94a8eb0f 100644--- a/frontend/bin/build-app-url-manifest.mjs+++ b/frontend/bin/build-app-url-manifest.mjs@@ -63,6 +63,8 @@ const EXCLUDED_HELPERS = new Set(['absolute', 'default', 'project', 'currentProj// `tracingOperation` renders its missing-parameter error, and `dashboardTemplates` opens the// dashboard list without its templates modal.const EXCLUDED_BUILDERS = new Set(['tracingOperation', 'dashboardTemplates'])+// Dropping these builders' fixed query strings opens a different destination.+const BUILDERS_WITH_REQUIRED_QUERY = new Set(['aiObservabilityDatasets'])// Flag-gated surfaces for PostHog staff only: a link an agent hands a customer would land on a scene that refuses them.const INTERNAL_BUILDERS = new Set(['decisionPlayground', 'library', 'todayReport'])@@ -245,7 +247,7 @@ function readSignature(fn) {// Invoke with the fewest sentinel args that yield a clean path — drops optional trailing params and// hits guarded default branches (`replay()` -> `/replay/home`) instead of erroring on a sentinel.-function probeBuilder(fn, signature) {+function probeBuilder(fn, signature, preserveQuery = false) {const maxArgs = signature.kind === 'object' ? 0 : signature.names.lengthfor (let count = 0; count <= maxArgs; count++) {const args = signature.kind === 'object' ? [{}] : signature.names.slice(0, count).map(sentinel)@@ -258,7 +260,8 @@ function probeBuilder(fn, signature) {if (typeof raw !== 'string') {continue}- const pathOnly = raw.split('?')[0].split('#')[0]+ const withoutHash = raw.split('#')[0]+ const pathOnly = withoutHash.split('?')[0]if (!pathOnly.startsWith('/') ||pathOnly === '/' ||@@ -267,7 +270,7 @@ function probeBuilder(fn, signature) {) {continue}- return pathOnly+ return preserveQuery ? withoutHash : pathOnly}return null}@@ -320,7 +323,7 @@ function buildManifest(urls) {excluded.push({ name, reason: 'could not parse signature' })continue}- const probePath = probeBuilder(fn, signature)+ const probePath = probeBuilder(fn, signature, BUILDERS_WITH_REQUIRED_QUERY.has(name))if (!probePath) {excluded.push({ name, reason: 'no clean path' })continuediff --git a/frontend/snapshots.yml b/frontend/snapshots.ymlindex 5abc309df5d4..a5b59aa6120c 100644--- a/frontend/snapshots.yml+++ b/frontend/snapshots.yml@@ -4509,9 +4509,9 @@ snapshots:layout-products-and-files--products-starred-collapsed--light:hash: v1.k794b7964.44533a12c58c83a253be3424bd24a6eae261feec47d3411bd896ac21cf584525.8lDvM03JlIuE1ki0q3wpcBen63AqJnQiiGqmFu87yk4layout-products-and-files--search--dark:- hash: v1.k794b7964.6695fe610e3f2c6defc39393d92d7075ee02c494f49c840a6aacb90f521bfe1f.6pIULWHpgkRwZOKuJOF4mNBxJlVemTXRwX0f-wbvnXI+ hash: v1.k794b7964.e1fe572d5b4b897dcf4e0ad0b1ffdec60b9bf74f5061b4f49bc6e89b10d0baa4.sfgp48RdHVp8bSgi-ZqJIBOQHghQCA3koUdq5CxwTUglayout-products-and-files--search--light:- hash: v1.k794b7964.2f1fc9d1cd3b0c4141c564ba244b0cf17599ff92a95fbea4937f5a7c05fd5040.hwD6HSMsaa7NWobGJRS5hjX89po3O5KU10hOKbE-Zbs+ hash: v1.k794b7964.1f6e05d594715431db27c7c8c5c55d60bf0aae1ce6ec8b88b79183749d865870.Rg_tcvdTUI6VRn6u3XBDxkYmNSVAXw0C-iyN8XtV6wElayout-products-product-tooltip--dark--dark:hash: v1.k794b7964.52ba01b050e579d7c600df08497a2dfab2dba9cf670553e032afce3819d6d00d.PNNtTamfHv5qRTsFnRwxhlFEaIteMOgYQgAab2QRRj0layout-products-product-tooltip--dark--light:@@ -8822,6 +8822,26 @@ snapshots:hash: v1.k794b7964.6a8f9cfbd5283164176a115eb6aa1ac80fe67c9f7a5fc44ba3c08ef04ff575be._962ei0cP3XkOKCVtwajtH83Pjlx2Vdaaj9bUI3qsP4scenes-app-ai-observability-conversation-display--tools--light:hash: v1.k794b7964.cc6f7770155f4b720ff0597c3dd5deff699a267cfbdd8fe76a131cb51740215d.lKCAOsQBSoBkK46EHSYdGmWxAi4CQ4wUrx2J1iRF_f0+ scenes-app-ai-observability-dataset--datasets-only-narrow--dark:+ hash: v1.k794b7964.9bea36774533c6f125e958f3469a59b8080d36b9ff5546bb92770ee5789db6ce.NLbIs2qFi3yuLycJ1To_LN5AdGNrWH25ayWiPQMsUmc+ scenes-app-ai-observability-dataset--datasets-only-narrow--light:+ hash: v1.k794b7964.c3caaf1aac1bdfe99eca9ebe496276c2f6cf43ce915731c33be5a31be4559a12.aXsTGkDbE3QxmaPfNA5i0Ii5mhLkyjvZoDQF3pHXA7g+ scenes-app-ai-observability-dataset--details--dark:+ hash: v1.k794b7964.1122d600365d38303883c8e6d986964fe7ccbd096922d18024722364a618779e.NSoJGML-0Zw9WarRka5NuBy1dItmCNgsiRRZCA5uKYA+ scenes-app-ai-observability-dataset--details--light:+ hash: v1.k794b7964.1fbe29482ffd475101b36f6ac411b80a7bfbf25c6486b970463727aa976f08e7.ARBMFFBiA_F7UvW7eyBy5IbGasKLGw5yBAFTTk_CQ1o+ scenes-app-ai-observability-dataset--items--dark:+ hash: v1.k794b7964.e1c2f9a6a56d543640d473a518dd09e37f61160d9086c174b412eb5bd293ae42.29BCUSzH5UfdEWOViSV7jxXsfuocRv0Rd7bA3O-A4Q0+ scenes-app-ai-observability-dataset--items--light:+ hash: v1.k794b7964.1c015595c8578810412d647be5d2024cc10f89550afcad9b5718eb03034d1694.KdlzqK5EDz_RBcSlVhL_3dh0Qw6Q41cWdOqnAeG9U28+ scenes-app-ai-observability-dataset--new-dataset--dark:+ hash: v1.k794b7964.7e1118fe1a3df1cbcce546cd7abe512a0263df5554566ed4828d68a5a7a8b474.wXttiynU_OFB8vlU5bXmyrHZIelPEw07Q0Fp61UCa0U+ scenes-app-ai-observability-dataset--new-dataset--light:+ hash: v1.k794b7964.224de55acdf166af1f369d618e308efe3a3aa46ae965737dc05a6204b2e5b375.7U2SsbfISxaMVddaKRLuHiWNt9Qj-6eLotsB6JKtaAM+ scenes-app-ai-observability-dataset--read-only--dark:+ hash: v1.k794b7964.682c3453692dd77c305690f91a7a77ddbe2de1b323aadea84174cdee39b97239.7UWvSw1xUNRBYr2VWzbKN22wfuhShLBrsxvpywsjbSs+ scenes-app-ai-observability-dataset--read-only--light:+ hash: v1.k794b7964.8365628e228326828fe8768e604dde02ea03e30af59047759367ec263cc4d21d.7A820WLz__G-1QESZjLMCWMb6ZJ1e06o20bttjsjnMoscenes-app-ai-observability-evaluation-explanation--reasoning--dark:hash: v1.k794b7964.df59798a73898fed2019aaa67ba440611c743d7bab0ff9c282db9eaeec4602dd.RDTleoCWmufVyJ6bte4BtB45JV51kbNvUbRZAa8lvUoscenes-app-ai-observability-evaluation-explanation--reasoning--light:@@ -8886,14 +8906,26 @@ snapshots:hash: v1.k794b7964.501bc38da197ae45e045d236336316a18cb78d533ee3008c678c25afe6f3624a.It6rPCGENocYlEoHpN4OPbTNCP4EOKtWe3DmtpiP8CMscenes-app-ai-observability-offline-experiments-experiment--request-failure--light:hash: v1.k794b7964.a31d57c72b94117a24a7a8948a68c6959b4fd00cde6d24aa3cdd206e1ca12eb7.392D8AhKK3JzKggndN4ojaEVt_vsHWraj2vVAWxz5NI+ scenes-app-ai-observability-offline-experiments-navigation--dataset-detail--dark:+ hash: v1.k794b7964.b52ff6e7af233221985da0bd9f25d650d605a85f4689e9bdd254c2ddfb625a7b.ar4RTmAEuV5gpBCtbtPCBwaP_kF2-6X2TFbRzU05Vo0+ scenes-app-ai-observability-offline-experiments-navigation--dataset-detail--light:+ hash: v1.k794b7964.7b1a2bbdc0ca9f1644bf91d1b6c8bbb6757abc39b8664602d91d2ddf8d92ba71.XlVuI4J64mWHM7DonkiOKD8bxPhcUIyxDQ62Z3NnVcU+ scenes-app-ai-observability-offline-experiments-navigation--datasets-only--dark:+ hash: v1.k794b7964.cfe60d1bb96149e5c07f38be4ff3c70dbb41b7e70c230a9ef127b121618326af.f7b7gNJVtuTFiIiGUMpXsbhB9uCrtd_OpC_--TfrTrs+ scenes-app-ai-observability-offline-experiments-navigation--datasets-only--light:+ hash: v1.k794b7964.35b96b7629c48614494653876b9e6f09df4d8a94a6857e7bb1542af21c73fd01.DmMuGfWJT-L59Z5Zy_Xl-GFe7Yn0C2w7H2biCvSBmp0+ scenes-app-ai-observability-offline-experiments-navigation--datasets-unavailable--dark:+ hash: v1.k794b7964.2a9db18dd5a878696f7b1af18c6a3608da26907d0f5eb65f278b9860d0a29a56.LqQr1nb1bEcpIIvExJLIuIgSGs4Aua4QFigCFIlT-ig+ scenes-app-ai-observability-offline-experiments-navigation--datasets-unavailable--light:+ hash: v1.k794b7964.dcfb6dc4d1b0290142c394df9165c1d1bac1972fa1179ebf1d4e1221f5f59354.biHIGb-srQvpY-WtpRgH3alC-AFxpd-J3cJOcgegL5oscenes-app-ai-observability-offline-experiments-overview--all-time-synchronized-hover--dark:- hash: v1.k794b7964.fb3ce13091c4c7f3c908e142c2b4e79064192344b38097db465eda6c4cf79a3b.0PHKo_RVtCnJr9EefOBcHXNht8MHl_FoBKHcqd9ol-Y+ hash: v1.k794b7964.d592fa62f27081449d44b3c03494292be8a043e255da1d70a02b9581221d41a6.hVExLS77L3Jez2RZ2180Yes9ksKzGjOGfoMXsP2mO1wscenes-app-ai-observability-offline-experiments-overview--all-time-synchronized-hover--light:- hash: v1.k794b7964.3df907417177586aa0f059463468cbab2be1d8c569fbfc349b0e00cddd7dea63.B9QMiTZkEOtrF1hQIUwQc25TjgyNMLgK3uRlboFL2UI+ hash: v1.k794b7964.5b8b807f13c80ffd34c7c4ec6e86529e8c3b2079a0aae53f07085c53a47da970.95QtsJLf8OrKvsEnaISXHWtG_It-Q38o1-5fuhvFxRMscenes-app-ai-observability-offline-experiments-overview--default-scorers--dark:- hash: v1.k794b7964.fa4c20dac4b64c4e62be12fb9f263082fe0ece58b4c5f3392c6fb19c5e403a59.yw0Qflw3sth1GPGHq6TOq4_fdSvLLHsN30lffaOu4uw+ hash: v1.k794b7964.73de5e6dff8607fbad95d027b0b2856095e68f28922faa4f06315f6338c8012f.iH_6hZJBzDtbyRJu-cLJduev5kRJYJf7MPCFhd_q4ooscenes-app-ai-observability-offline-experiments-overview--default-scorers--light:- hash: v1.k794b7964.fc5e13d18d4567e67d59f1e4e2d330dcecddb178178f77555998dee7bdaac38f.qodNHeE8aFki2ZeCyyGaY_J-Q9P1fo3j2k2n80SoaVk+ hash: v1.k794b7964.726dc1aa28e3c1b1075ee590b0f24bd8acd6f62efc5bbfad63afbba5d0ba6e0a.u16euuIPiAp98GLWkoqQz4E2i4kGWjiuRZEgpugNcC0scenes-app-ai-observability-offline-experiments-overview--empty--dark:hash: v1.k794b7964.977ade824abb5791ed84c5b6d125bc9589d0787b69daea3efeb6d0fe91f45f70.czJW_b0HTFrXWZGbmMZfAs1UD-NkCfVNLy36XZiTH7Yscenes-app-ai-observability-offline-experiments-overview--empty--light:@@ -8903,53 +8935,53 @@ snapshots:scenes-app-ai-observability-offline-experiments-overview--empty-narrow--light:hash: v1.k794b7964.754079aa9ab7ee416590c9bc8f94fa93f59b7769e98d4e63400883bf4d23ffae.hLRMtJxEnlm36ceer7qs_5s0s0euWyZB0BAqWPZz8c0scenes-app-ai-observability-offline-experiments-overview--filtered-trends--dark:- hash: v1.k794b7964.b364f69b2d316e73b525d40ae1d3e7fbae7ee1f9013e869125d8346005736145.ytOrnuZ9UQOyKRmTwQhFrIOFDlDDJn9oUzzsAGKA5Po+ hash: v1.k794b7964.3c78d5fe30a6ac258d072c3c285ee5f89bf1216f371d8d567207278d6976cd75.Lj5t_bkyrialU7kZJ3OCIREFPSJHuQ7KjxXmoBS5WEIscenes-app-ai-observability-offline-experiments-overview--filtered-trends--light:- hash: v1.k794b7964.377ad2181d398e9a57a1d7fac8fc7fb2237fc680c3c7c5563c5973c2d39f34a5.6hIWHqTofDepWwl5hoetAWimrE1eEatrRlEJwxUG_oE+ hash: v1.k794b7964.812fede378b45377df67ce7a81b6a4f8ebda638209ff66b2f340b4275e3e745e.I6zJjCdwYoP-wj4Ix6ysSL3AXq7Pflfb-9q8KEgi90gscenes-app-ai-observability-offline-experiments-overview--list-error--dark:- hash: v1.k794b7964.8b831fef9bfedf72e76e149939a00bf4a76900156748d3bafb93f1d3200f61d6.CUFTGjUs_D9pgPOcndn6bS7R25bmn-9mLEb9b4PDi4o+ hash: v1.k794b7964.8b08646679e7e9bc99f28b7964580747c49d3499add523991ecd0c37999a5730.DRysc1W_RehJmLU8rb-CBI6FdtvC6LZ4WxiHULuUsbMscenes-app-ai-observability-offline-experiments-overview--list-error--light:- hash: v1.k794b7964.fa8e8ecc24f4173a2267281305834cffe9f7eb099785fb88d0830aa165544ed2.j-uksYVVEc6BUX0THSvf1i60NT5xraYlyAP2m8byBgQ+ hash: v1.k794b7964.f334897ee8fefcc511aa2b2c9110ab86087d6b8635e654e004f87ba36182b3f2.4ARq9cFYhHpXg4hwzj5Qo5HzyM_8Vm37M3RX2wUzxuAscenes-app-ai-observability-offline-experiments-overview--multiple-versions--dark:- hash: v1.k794b7964.84a57ee92f9203aa1869bee3548823fb04a7288edec4eaffb8cc294206bb6290.rG4uZItUyrl_vHXhYQY-Ctz_8_jlXYrmALwq89ln5dE+ hash: v1.k794b7964.e19b98e1146e62df304fb74a78609f238061bf04b0eb5a0ecbdd60c3e08706db.scNCgHa-qolVB6l0ISpAbWQ2OysYbr-aagU7JP4bjhYscenes-app-ai-observability-offline-experiments-overview--multiple-versions--light:- hash: v1.k794b7964.ebc934d093a8e63a084fe2d3e5cb9ad3871728bb40672f6cf61539975338ac16.dZzCqNHkhveuOCds4g7xVYzXzoGFMWJ7Dx46IMrxaDY+ hash: v1.k794b7964.3aa4c8b9363dc26ddc19a2fe796fda8a6bf0f7333e2ff27095de47f0e7504160.jwtwbP8ON-5hP8Bl9TEryP5AuF0GXOL5PL1pSdybbNYscenes-app-ai-observability-offline-experiments-overview--multiple-versions-narrow--dark:- hash: v1.k794b7964.030b960d331a7b7bea03eb21353c286d47a6268a333f940f8433ad624942dd28.WNcOt3ZcTgD9RG5goeQrngfLjg4IPg_xrlGQqtbA9P8+ hash: v1.k794b7964.f8ef3b6bb3ad6d5cc1607c2464b0c24f6092dfdd1a36e5371841f96d0862ffde.phw7DAjJ0YPTz-TlOwBt_tWGhoxQ2Xayx7tpP46HEtsscenes-app-ai-observability-offline-experiments-overview--multiple-versions-narrow--light:- hash: v1.k794b7964.423be802a7cf08f619cbcb4cb44a8a340f376f8ed37517a6a21b9c51f5f806ee.tvZnyRoE2qTCtCf1uFDX9gX8UYXqIQMGTlySrXoXrv4+ hash: v1.k794b7964.c95966719f13fabaa65824dff4241e4ff45e3f672cdda558f8856074ee393d30.yLUEYWkQdofVB_u3RiCPns1GWYX0pHmNQ5FV_2S_SiQscenes-app-ai-observability-offline-experiments-overview--narrow--dark:- hash: v1.k794b7964.18ed7d7ec785488c7ab89a9e96c1ccd5b275354f306ecb34ae2fe5e432052176.xM7r2_DGEEvvWion1un3ajeDOXQjNS9SOo3ftnVC4mE+ hash: v1.k794b7964.076b09003e2f95dc77677a233d6aed6b3cac091eb9dc1082e5c66c3dab103daa.8GpuwmX5FXnNibzd6F9j1JHkzQY18Euc0dBy21UZWIAscenes-app-ai-observability-offline-experiments-overview--narrow--light:- hash: v1.k794b7964.1179772dcf2581f883409bea23a188fe79829a2886dd436a602f166534331a1f.2JhsYQ8qLMUOeSuRKMqsSsf2tXBMu77U0qoOHwP7g70+ hash: v1.k794b7964.cf41bb77c212ae4f3902b07cdd1b427e28dfc35c7ef7d97a13c74284c250f671.poKtRWEEhip6WUeBDRiZOHhk454BV5PqeKR_xKg31yoscenes-app-ai-observability-offline-experiments-overview--no-matching-experiments--dark:- hash: v1.k794b7964.1fcbfbaeaadd5301cc28702cbf6c93dc47f2fe56318e9c087303b0af08455def.9P2-61KkzzeyH14B56FzMWousQ3HXkJazoiEBMr6b3A+ hash: v1.k794b7964.b9a6e968b876bc9cb4c7419a7207a855c02bf59260bce65f3e1d6a4cf43f8162.D-a81APnf9lM8eqv7yefQZe4LFolm_3V2ESmXW4b8jsscenes-app-ai-observability-offline-experiments-overview--no-matching-experiments--light:- hash: v1.k794b7964.ef2304ac785a197aa675c9e826062246bb19cb244a12047629d26dcbd375a3cc.P9LAmLcHzhobEftRQ30_-UJJGpnLGA9-Z_th_nwd73Q+ hash: v1.k794b7964.a46726aef52244de1229d9c340e86b1b2dec844f804f0259a3a2355a9d0132b6.rrzNUBXeCapNsY6HZ0_Dty-kR8DEMF3ICGgugfM1cOsscenes-app-ai-observability-offline-experiments-overview--partial-history--dark:- hash: v1.k794b7964.dfc8dfdcbc8b02b1593e54be9125973c174ac67cb4c5fde46f119fcbdce70d93.gWtzSWjSRhkQeb5LVEF_qLMpAQGHk4qOXPo6YvN7cNM+ hash: v1.k794b7964.ed2754c3530dbc22dd79a579206750a9a9ca25e7e02671af70ecfda2d52621cc.q2C2PvHpo2u4er1wViaYpchyoRMXUhuUpftRlIyp9tIscenes-app-ai-observability-offline-experiments-overview--partial-history--light:- hash: v1.k794b7964.7d61c4c030e2ac777b8eac0c612e36d23f9cb701bb09764986957c834e8160d3.uOegrRMSXAU7ry1PHSKRdlWrrskTbFcXEnGkaJcgULM+ hash: v1.k794b7964.302ca88f07d59cdabb3dd8b286b13ff39c03b318a35bca99405683c8e3789d64.-Jd-kXQ46NBmqRsF90-2Y23EI8pzdxvbU9FADQWauYQscenes-app-ai-observability-offline-experiments-overview--partial-history-narrow--dark:- hash: v1.k794b7964.263decd33e83edddaaa509224c0ab69be4cab9d0f1f0bb2aa6e42ec89b685417.sACn0pHOVuNcLWjOXOSdVZXvVOprawYthGBMuGinn3c+ hash: v1.k794b7964.4c450840d8d01ddb2eb4817e1fea7f7fdbd288e93ea3c375555b4ceb1ae08311.jwIcdD5ZnyQniw64ArTCvFuX43hlRtSDbHAIU6qowFUscenes-app-ai-observability-offline-experiments-overview--partial-history-narrow--light:- hash: v1.k794b7964.a02e0465d32c0184cca031c780901fd142ce28b32d792065811caf8eed6c8f21.RU-d8SUARzAYGDeVLBN8Hn1XeabYC0gaLz2BfNN3odU+ hash: v1.k794b7964.5b69ff2c107ab0450df0d7495d19da2d19293d8fe3b453224845365f8b402f3f.UkyFocgNR0N1T0QWFTOa6HcQYW4BdRMv5_qSzgjjFmIscenes-app-ai-observability-offline-experiments-overview--recent-experiments--dark:- hash: v1.k794b7964.fa4c20dac4b64c4e62be12fb9f263082fe0ece58b4c5f3392c6fb19c5e403a59.qSvAMWBW7apGQFrYPcJoNARPkAgjlSg94f8WuUuWek8+ hash: v1.k794b7964.73de5e6dff8607fbad95d027b0b2856095e68f28922faa4f06315f6338c8012f.YbQpwN5pANDtCZDWgkZ8z9-XfMnI8dB5_HsfibRZ1z0scenes-app-ai-observability-offline-experiments-overview--recent-experiments--light:- hash: v1.k794b7964.fc5e13d18d4567e67d59f1e4e2d330dcecddb178178f77555998dee7bdaac38f.ium6vEX51-vJpa_q1nCmYTwF67NJ68RsV-mLLX4AVkE+ hash: v1.k794b7964.726dc1aa28e3c1b1075ee590b0f24bd8acd6f62efc5bbfad63afbba5d0ba6e0a.wGg0Or9ZpTIpeXaLA386097O4aeCtkFCyB8O4kj-Y6gscenes-app-ai-observability-offline-experiments-overview--score-error--dark:- hash: v1.k794b7964.bc28fdc2405851f5b61cfedfa355a27d090fb84a4e496d9f2fc549eaa173435f.S1IV4V9DdF-gbm2AKqw4-4bc9gz3-eVSXk2sMG8Zs9Y+ hash: v1.k794b7964.689beee8e1c93df52ab1400230008ed48f2828cf75d9992961c81c99816b7fd1._LEb7qSea9grjGiQ8KguiqaT6g270Z8Il2C2ABMqHaIscenes-app-ai-observability-offline-experiments-overview--score-error--light:- hash: v1.k794b7964.dece407dd02844758e8950c45573df48c523954726b16521214138d4a10f38d7.LaZL9pX6seylgIWgDtukSLSffO92cydY_RRpFKgEJHw+ hash: v1.k794b7964.3e158e8aaa911450d7e23670499ae4475a9dc63631c33163a0126867e493983f.nudFomjOot039FZfsD0h2aexX1D7lizpFjF0-mzhF98scenes-app-ai-observability-offline-experiments-overview--synchronized-hover--dark:- hash: v1.k794b7964.f82a77c70a10f644911e40efbbd9e5049ab6ddaa708ecdc1c0463b7b554ffb5b.MvIo2T4NRSyRK8PhTkhObMSioTbx_3kXKE_wOCYEk9U+ hash: v1.k794b7964.668130859c652243deae419e7ed1859d817725f5309c93f05b870cf901aea6a9.SI6bFs6z8YKTICZfTS7AWmzizhfGYyb6TlxfRpzpcfYscenes-app-ai-observability-offline-experiments-overview--synchronized-hover--light:- hash: v1.k794b7964.76fa7ae08158f6597e0763661932c7f9ae895dc25805bc304c56e8e45a64f15f.QkSBo1cy4H4qsYFYadwaRIuZtl1k7KgiTku4bgcGtgE+ hash: v1.k794b7964.6f807533006b80fd5f76474da92184ef23a3732b79550d63253a56bd1bd87d90.lS1pIeWc5KqSo7MjkKRrU4LvjA4xbQDn4Szolkmi1m8scenes-app-ai-observability-offline-experiments-overview--synchronized-hover-narrow--dark:- hash: v1.k794b7964.56d79a197d4687fca8bcd4dcda8346f5e0914feb1db5466ec6e302599315c94d.HEivagJC0ow1_ZIhKVgJFkEo8CIv4ssSW4DEHXTqe8M+ hash: v1.k794b7964.d24b4ebb39da187de88b75cb4fc95646dd62e623baad293435d54393d5f6f500.uGmSH6SyeGMPuhz5VtsXsT7TcZ7e-LYsl_WAaMmgLQsscenes-app-ai-observability-offline-experiments-overview--synchronized-hover-narrow--light:- hash: v1.k794b7964.c34bcc78bfa8f503b5cabac0aee2b534ac814a7387219c007ddd24d636b87df3.b-l0k8WJngZByacU9IzDwQDBGU6gXMfFeOLtK9jMwM0+ hash: v1.k794b7964.deb49ddb9dd172730240051497fe2de1326e28b43dac6280075eb58d8d448dc1.X50r68eZ8nAkCo6w7QqWQs1OINlCknBraztf1U6tuhoscenes-app-ai-observability-promptreferencetags--label-and-version-references--dark:hash: v1.k794b7964.4cc699d7489063a64cf4ec1f930d11fc062eb0d9c9e46150f939503945bb8841.jKmH2ZmjZex98fImM00HE_xBBayalWKHtqXbVlhE_VMscenes-app-ai-observability-promptreferencetags--label-and-version-references--light:diff --git a/frontend/src/lib/components/ObjectTags/ObjectTags.tsx b/frontend/src/lib/components/ObjectTags/ObjectTags.tsxindex 4eb9f86cd809..1a8ab9ea95d0 100644--- a/frontend/src/lib/components/ObjectTags/ObjectTags.tsx+++ b/frontend/src/lib/components/ObjectTags/ObjectTags.tsx@@ -36,6 +36,7 @@ interface ObjectTagsPropsBase {* room surrounding table columns get.*/wrap?: boolean+ compact?: booleaneditorFullWidth?: boolean}@@ -82,6 +83,7 @@ export function ObjectTags({maxVisibleTags,showOverflowLabel = false,wrap = false,+ compact = false,editorFullWidth = false,}: ObjectTagsProps): JSX.Element {const objectTagId = useId()@@ -107,7 +109,8 @@ export function ObjectTags({style={style}className={clsx(className,- 'inline-flex flex-wrap gap-0.5 items-center',+ 'inline-flex gap-0.5 items-center',+ compact ? 'flex-nowrap min-w-0' : 'flex-wrap',wrap && 'min-w-0 max-w-full',editingTags && editorFullWidth && 'w-full min-w-0')}@@ -141,10 +144,17 @@ export function ObjectTags({key={index}type={COLOR_OVERRIDES[tag] || colorForString(tag)}onClick={onTagClick ? () => onTagClick(tag) : undefined}- className={wrap ? 'max-w-full' : undefined}+ size={compact ? 'small' : 'medium'}+ className={compact ? 'min-w-0' : wrap ? 'max-w-full' : undefined}wrap={wrap}>- {tag}+ {compact ? (+ <span className="truncate" title={tag}>+ {tag}+ </span>+ ) : (+ tag+ )}</LemonTag>)})}diff --git a/frontend/src/products.json b/frontend/src/products.jsonindex 4a059abf6973..a1fb76cb42a8 100644--- a/frontend/src/products.json+++ b/frontend/src/products.json@@ -21,19 +21,12 @@"type": "llm_clusters","intents": ["llm_clusters"]},- {- "path": "Datasets",- "category": "AI engineering",- "iconType": null,- "type": "llm_datasets",- "intents": ["llm_datasets"]- },{"path": "Evaluations","category": "AI engineering","iconType": null,"type": "llm_evaluations",- "intents": ["llm_evaluations"]+ "intents": ["llm_evaluations", "llm_datasets"]},{"path": "Taggers",diff --git a/frontend/src/products.tsx b/frontend/src/products.tsxindex 014fb2544c4c..598e8f784de6 100644--- a/frontend/src/products.tsx+++ b/frontend/src/products.tsx@@ -88,8 +88,7 @@ export const productRoutes: Record<string, [string, string]> = {'/ai-observability/clusters': ['AIObservabilityClusters', 'aiObservabilityClusters'],'/ai-observability/clusters/:runId': ['AIObservabilityClusters', 'aiObservabilityClusters'],[AI_OBSERVABILITY_CLUSTER_URL_PATTERN]: ['AIObservabilityCluster', 'aiObservabilityCluster'],- '/ai-evals/datasets': ['AIObservabilityDatasets', 'aiObservabilityDatasets'],- '/ai-evals/datasets/:id': ['AIObservabilityDataset', 'aiObservabilityDataset'],+ '/ai-evals/evaluations/offline/datasets/:id': ['AIObservabilityDataset', 'aiObservabilityDataset'],'/ai-evals/taggers': ['AIObservabilityTags', 'aiObservabilityTags'],'/ai-evals/taggers/:id': ['AIObservabilityTag', 'aiObservabilityTag'],'/ai-evals/evaluations': ['AIObservabilityEvaluations', 'aiObservabilityEvaluations'],@@ -411,6 +410,10 @@ export const productRedirects: Record<},'/ai-evals': (_params, searchParams, hashParams) =>combineUrl(urls.aiObservabilityEvaluations(), searchParams, hashParams).url,+ '/ai-evals/datasets': (_params, searchParams, hashParams) =>+ combineUrl(urls.aiObservabilityDatasets(), { ...searchParams, tab: 'datasets' }, hashParams).url,+ '/ai-evals/datasets/:id': (params, searchParams, hashParams) =>+ combineUrl(urls.aiObservabilityDataset(params.id), searchParams, hashParams).url,'/ai-evals/evaluations/offline': (_params, searchParams, hashParams) =>combineUrl(urls.aiObservabilityOfflineEvaluations(), searchParams, hashParams).url,'/ai-evals/tags': (_params, searchParams, hashParams) =>@@ -468,7 +471,7 @@ export const productRedirects: Record<'/llm-analytics/clusters/:runId/:clusterId': (params, searchParams, hashParams) =>combineUrl(urls.aiObservabilityCluster(params.runId, params.clusterId), searchParams, hashParams).url,'/llm-analytics/datasets': (_params, searchParams, hashParams) =>- combineUrl(urls.aiObservabilityDatasets(), searchParams, hashParams).url,+ combineUrl(urls.aiObservabilityDatasets(), { ...searchParams, tab: 'datasets' }, hashParams).url,'/llm-analytics/datasets/:id': (params, searchParams, hashParams) =>combineUrl(urls.aiObservabilityDataset(params.id), searchParams, hashParams).url,'/llm-analytics/tags': (_params, searchParams, hashParams) =>@@ -1379,13 +1382,13 @@ export const productUrls = {return `/ai-observability/sessions/${id}${stringifiedParams ? `?${stringifiedParams}` : ''}`},aiObservabilityPlayground: (): string => '/ai-observability/playground',- aiObservabilityDatasets: (): string => '/ai-evals/datasets',+ aiObservabilityDatasets: (): string => '/ai-evals/evaluations/offline/experiments?tab=datasets',aiObservabilityDataset: (id: string,params?: {item?: string}- ): string => combineUrl(`/ai-evals/datasets/${id}`, params).url,+ ): string => combineUrl(`/ai-evals/evaluations/offline/datasets/${id}`, params).url,aiObservabilityTags: (): string => '/ai-evals/taggers',aiObservabilityTag: (id: string): string => `/ai-evals/taggers/${id}`,aiObservabilityEvaluations: (): string => '/ai-evals/evaluations',@@ -2338,7 +2341,6 @@ export type ProductTreePath =| 'Dashboards'| 'Data catalog'| 'Data warehouse'- | 'Datasets'| 'Early access features'| 'Endpoints'| 'Engineering analytics'@@ -2604,42 +2606,6 @@ export const getTreeItemsProducts = (): FileSystemImport[] => ['WarehouseDestinations',],},- {- path: 'Datasets',- intents: [ProductKey.LLM_DATASETS],- category: ProductItemCategory.AI_ENGINEERING,- type: 'llm_datasets',- iconType: 'llm_datasets' as FileSystemIconType,- iconColor: ['var(--color-product-llm-datasets-light)', 'var(--color-product-llm-datasets-dark)'],- href: urls.aiObservabilityDatasets(),- searchKeywords: ['test sets', 'golden set'],- flag: FEATURE_FLAGS.LLM_ANALYTICS_DATASETS,- tags: ['beta'],- sceneKey: 'AIObservabilityDatasets',- sceneKeys: [- 'AIObservability',- 'AIObservabilityTrace',- 'AIObservabilitySession',- 'AIObservabilityUsers',- 'AIObservabilityPlayground',- 'AIObservabilityDatasets',- 'AIObservabilityDataset',- 'AIObservabilityEvaluations',- 'AIObservabilityScorers',- 'AIObservabilityScorer',- 'AIObservabilityOfflineExperiments',- 'AIObservabilityOfflineExperiment',- 'AIObservabilityOfflineScorerHistory',- 'AIObservabilityEvaluation',- 'AIObservabilityEvaluationTemplates',- 'AIObservabilityTags',- 'AIObservabilityTag',- 'AIObservabilityPrompts',- 'AIObservabilityPrompt',- 'AIObservabilityClusters',- 'AIObservabilityCluster',- ],- },{path: 'ETL',displayLabel: 'ELT',@@ -2735,19 +2701,20 @@ export const getTreeItemsProducts = (): FileSystemImport[] => [},{path: 'Evaluations',- intents: [ProductKey.LLM_EVALUATIONS],+ intents: [ProductKey.LLM_EVALUATIONS, ProductKey.LLM_DATASETS],category: ProductItemCategory.AI_ENGINEERING,type: 'llm_evaluations',iconType: 'llm_evaluations' as FileSystemIconType,iconColor: ['var(--color-product-llm-evaluations-light)', 'var(--color-product-llm-evaluations-dark)'],href: urls.aiObservabilityEvaluations(),- searchKeywords: ['llm judge', 'llm as a judge', 'grading'],+ searchKeywords: ['llm judge', 'llm as a judge', 'grading', 'datasets', 'test sets', 'golden set'],searchTabs: [{name: 'Offline evals',href: urls.aiObservabilityOfflineEvaluations(),flag: FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS,},+ { name: 'Datasets', href: urls.aiObservabilityDatasets(), flag: FEATURE_FLAGS.LLM_ANALYTICS_DATASETS },{ name: 'Scorers', href: urls.aiObservabilityScorers() },],sceneKey: 'AIObservabilityEvaluations',diff --git a/frontend/src/scenes/sceneLogic.test.tsx b/frontend/src/scenes/sceneLogic.test.tsxindex 55553757981a..55d22e6623e7 100644--- a/frontend/src/scenes/sceneLogic.test.tsx+++ b/frontend/src/scenes/sceneLogic.test.tsx@@ -67,6 +67,7 @@ const testScenes: Record<string, () => any> = {[Scene.AgenticAuthorize]: sceneImport,[Scene.Alerts]: sceneImport,[Scene.AIObservabilityEvaluations]: sceneImport,+ [Scene.AIObservabilityOfflineExperiments]: sceneImport,[Scene.Billing]: sceneImport,[Scene.DataManagement]: sceneImport,[Scene.OrganizationCreateFirst]: sceneImport,@@ -464,9 +465,11 @@ describe('sceneLogic', () => {})it.each([- [AccessControlLevel.Viewer, Scene.AIObservabilityEvaluations],- [AccessControlLevel.None, Scene.ErrorAccessDenied],- ])('gates the combined evaluations entry with scorer access %s', async (scorerAccess, expectedScene) => {+ [Scene.AIObservabilityEvaluations, AccessControlLevel.Viewer, Scene.AIObservabilityEvaluations],+ [Scene.AIObservabilityEvaluations, AccessControlLevel.None, Scene.ErrorAccessDenied],+ [Scene.AIObservabilityOfflineExperiments, AccessControlLevel.Viewer, Scene.AIObservabilityOfflineExperiments],+ [Scene.AIObservabilityOfflineExperiments, AccessControlLevel.None, Scene.ErrorAccessDenied],+ ])('gates %s with scorer and dataset access %s', async (targetScene, scorerAccess, expectedScene) => {const priorAppContext = window.POSTHOG_APP_CONTEXTtry {window.POSTHOG_APP_CONTEXT = {@@ -477,7 +480,7 @@ describe('sceneLogic', () => {[AccessControlResourceType.LlmAnalytics]: scorerAccess,},} as AppContext- logic.actions.setScene(Scene.AIObservabilityEvaluations, 'aiObservabilityEvaluations', {+ logic.actions.setScene(targetScene as Scene, targetScene, {params: {},searchParams: {},hashParams: {},diff --git a/frontend/src/scenes/sceneTypes.ts b/frontend/src/scenes/sceneTypes.tsindex 15e333fb1897..e211d49a48f8 100644--- a/frontend/src/scenes/sceneTypes.ts+++ b/frontend/src/scenes/sceneTypes.ts@@ -487,7 +487,10 @@ export const sceneToAccessControlResourceType: Partial<[Scene.AIObservabilityEvaluations]: [AccessControlResourceType.Evaluation, AccessControlResourceType.LlmAnalytics],[Scene.AIObservabilityScorers]: AccessControlResourceType.LlmAnalytics,[Scene.AIObservabilityScorer]: AccessControlResourceType.LlmAnalytics,- [Scene.AIObservabilityOfflineExperiments]: AccessControlResourceType.Evaluation,+ [Scene.AIObservabilityOfflineExperiments]: [+ AccessControlResourceType.Evaluation,+ AccessControlResourceType.LlmAnalytics,+ ],[Scene.AIObservabilityOfflineExperiment]: AccessControlResourceType.Evaluation,[Scene.AIObservabilityOfflineScorerHistory]: AccessControlResourceType.Evaluation,[Scene.AIObservabilityEvaluationTemplates]: AccessControlResourceType.Evaluation,diff --git a/frontend/src/styles/base.scss b/frontend/src/styles/base.scssindex f9f0d449a9c3..c1024f82aeac 100644--- a/frontend/src/styles/base.scss+++ b/frontend/src/styles/base.scss@@ -348,13 +348,13 @@--color-product-data-warehouse-dark: rgb(160 135 255);--color-product-data-catalog-light: rgb(101 78 196);--color-product-data-catalog-dark: rgb(139 118 232);- --color-product-llm-analytics-light: rgb(182 42 217);- --color-product-llm-analytics-dark: rgb(212 106 240);- --color-product-llm-clusters-light: rgb(108 93 238);+ --color-product-llm-analytics-light: rgb(99 102 241);+ --color-product-llm-analytics-dark: rgb(129 140 248);+ --color-product-llm-clusters-light: rgb(110 92 238);--color-product-llm-datasets-light: rgb(121 87 235);- --color-product-llm-evaluations-light: rgb(135 81 232);- --color-product-mcp-analytics-light: rgb(150 76 229);- --color-product-mcp-analytics-dark: rgb(181 124 249);+ --color-product-llm-evaluations-light: rgb(126 85 234);+ --color-product-mcp-analytics-light: rgb(142 78 231);+ --color-product-mcp-analytics-dark: rgb(173 125 248);--color-product-notebooks-light: rgb(239 68 68);--color-product-notebooks-dark: rgb(252 165 165);--color-product-tracing-light: rgb(6 182 212);@@ -383,13 +383,13 @@--color-product-annotations-dark: rgb(94 235 238);--color-product-warehouse-destinations-dark: rgb(249 115 22);--color-product-warehouse-destinations-light: rgb(194 65 12);- --color-product-llm-clusters-dark: rgb(133 128 248);+ --color-product-llm-clusters-dark: rgb(136 128 248);--color-product-llm-datasets-dark: rgb(149 127 248);- --color-product-llm-evaluations-dark: rgb(165 126 248);- --color-product-llm-playground-light: rgb(165 70 226);- --color-product-llm-playground-dark: rgb(198 123 249);- --color-product-llm-prompts-light: rgb(180 65 222);- --color-product-llm-prompts-dark: rgb(215 122 249);+ --color-product-llm-evaluations-dark: rgb(154 126 248);+ --color-product-llm-playground-light: rgb(160 72 227);+ --color-product-llm-playground-dark: rgb(192 124 249);+ --color-product-llm-prompts-light: rgb(178 66 222);+ --color-product-llm-prompts-dark: rgb(212 122 249);--color-product-llm-tags-light: rgb(196 60 218);--color-product-llm-tags-dark: rgb(232 121 249);--color-product-event-definitions-light: rgb(24 145 200);diff --git a/products/ai_observability/frontend/aiObservabilityLogic.test.ts b/products/ai_observability/frontend/aiObservabilityLogic.test.tsindex e2749eba070e..9251e2b558b1 100644--- a/products/ai_observability/frontend/aiObservabilityLogic.test.ts+++ b/products/ai_observability/frontend/aiObservabilityLogic.test.ts@@ -11,7 +11,7 @@ import { emptySceneParams } from 'scenes/scenes'import { Scene } from 'scenes/sceneTypes'import { urls } from 'scenes/urls'-import { productRedirects } from '~/products'+import { productRedirects, productRoutes } from '~/products'import { isEventsQuery, isTracesQuery } from '~/queries/utils'import { initKeaTests } from '~/test/init'import { PropertyFilterType, PropertyOperator } from '~/types'@@ -43,7 +43,7 @@ describe('LLM analytics URL split', () => {expect(urls.aiObservabilitySelfDriving()).toBe('/ai-observability/self-driving')expect(urls.aiObservabilityReviews()).toBe('/ai-observability/reviews')expect(urls.aiObservabilityTrace('trace-1')).toBe('/ai-observability/traces/trace-1')- expect(urls.aiObservabilityDatasets()).toBe('/ai-evals/datasets')+ expect(urls.aiObservabilityDatasets()).toBe('/ai-evals/evaluations/offline/experiments?tab=datasets')expect(urls.aiObservabilityTags()).toBe('/ai-evals/taggers')expect(urls.aiObservabilityEvaluations()).toBe('/ai-evals/evaluations')expect(urls.aiObservabilityScorers()).toBe('/ai-evals/evaluations/scorers')@@ -63,7 +63,7 @@ describe('LLM analytics URL split', () => {'/ai-observability/traces/trace-1?event=event-1')expect(redirectUrl('/llm-analytics/datasets/:id', { id: 'dataset-1' }, { item: 'item-1' })).toBe(- '/ai-evals/datasets/dataset-1?item=item-1'+ '/ai-evals/evaluations/offline/datasets/dataset-1?item=item-1')expect(redirectUrl('/llm-analytics/tags/:id', { id: 'tagger-1' })).toBe('/ai-evals/taggers/tagger-1')expect(redirectUrl('/llm-analytics/evaluations/:id', { id: 'evaluation-1' })).toBe(@@ -95,6 +95,60 @@ describe('LLM analytics URL split', () => {})+ it.each([+ ['/ai-evals/datasets/:id', 'new'],+ ['/ai-evals/datasets/:id', 'dataset-1'],+ ['/llm-analytics/datasets/:id', 'new'],+ ['/llm-analytics/datasets/:id', 'dataset-1'],+ ])('redirects dataset creation and detail bookmarks from %s (%s)', (path, id) => {+ const searchParams = {+ tab: 'metadata',+ item: 'item-1',+ revision: '3',+ item_status: 'archived',+ datasets_page: '2',+ search: 'quality',+ }+ const redirected = new URL(redirectUrl(path, { id }, searchParams, { section: 'items' }), 'https://example.com')+ expect(redirected.pathname).toBe(`/ai-evals/evaluations/offline/datasets/${id}`)+ expect(Object.fromEntries(redirected.searchParams)).toEqual(searchParams)+ expect(redirected.hash).toBe('#section=items')+ expect(productRoutes['/ai-evals/evaluations/offline/datasets/:id']).toEqual([+ 'AIObservabilityDataset',+ 'aiObservabilityDataset',+ ])+ })++ it.each(['/ai-evals/datasets', '/llm-analytics/datasets'])(+ 'redirects dataset bookmarks with their filters and hash: %s',+ (path) => {+ const redirected = new URL(+ redirectUrl(+ path,+ {},+ {+ tab: 'metadata',+ search: 'quality',+ dataset_status: 'archived',+ datasets_page: '2',+ tags: 'regression',+ },+ { section: 'list' }+ ),+ 'https://example.com'+ )+ expect(redirected.pathname).toBe(urls.aiObservabilityOfflineEvaluations())+ expect(Object.fromEntries(redirected.searchParams)).toEqual({+ tab: 'datasets',+ search: 'quality',+ dataset_status: 'archived',+ datasets_page: '2',+ tags: 'regression',+ })+ expect(redirected.hash).toBe('#section=list')+ }+ )+it('redirects AI observability settings to the project-level BYOK setting', () => {expect(redirectUrl('/ai-observability/settings')).toBe('/settings/project-ai-observability#ai-observability-byok'diff --git a/products/ai_observability/frontend/components/ResourceTagFilter.tsx b/products/ai_observability/frontend/components/ResourceTagFilter.tsxindex 835f25c954ca..1c8f50ca0588 100644--- a/products/ai_observability/frontend/components/ResourceTagFilter.tsx+++ b/products/ai_observability/frontend/components/ResourceTagFilter.tsx@@ -8,15 +8,17 @@ import type { ResourceTagFilters } from './resourceTagFilters'export function ResourceTagFilter({filters,onChange,+ size = 'medium',}: {filters: ResourceTagFiltersonChange: (filters: ResourceTagFilters) => void+ size?: 'small' | 'medium'}): JSX.Element {return (<div className="flex flex-wrap items-center gap-2"><TagSelectdefaultLabel="Tags"- size="medium"+ size={size}value={filters.tags}onChange={(tags) => {if (tags.length > 20) {@@ -29,6 +31,7 @@ export function ResourceTagFilter({/>{filters.tags.length > 1 && (<LemonSelect+ size={size}value={filters.tag_match}onChange={(tag_match) => onChange({ tags: filters.tags, tag_match })}options={[diff --git a/products/ai_observability/frontend/components/ResourceTags.tsx b/products/ai_observability/frontend/components/ResourceTags.tsxindex f204e5187f7f..74a1578bc7d1 100644--- a/products/ai_observability/frontend/components/ResourceTags.tsx+++ b/products/ai_observability/frontend/components/ResourceTags.tsx@@ -21,6 +21,7 @@ export function ResourceTags({onTagClick,userAccessLevel,disabledReason,+ compact = false,}: {entity: TaggedResourceresourceId: string@@ -29,18 +30,20 @@ export function ResourceTags({onTagClick: (tag: string) => voiduserAccessLevel?: AccessControlLeveldisabledReason?: string+ compact?: boolean}): JSX.Element {const { currentTeamId } = useValues(teamLogic)const logic = resourceTagsLogic({ entity, resourceId, tags, onSaved, teamId: currentTeamId ?? 0 })const { isOpen } = useValues(logic)const { open } = useActions(logic)return (- <div className="flex items-start gap-1 min-w-0 max-w-64">+ <div className="flex items-center gap-1 min-w-0 max-w-64"><ObjectTagstags={[...tags]}staticOnly- maxVisibleTags={2}- wrap+ maxVisibleTags={compact ? 1 : 2}+ compact={compact}+ wrap={!compact}onTagClick={onTagClick}data-attr="aio-resource-tags"/>diff --git a/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.stories.tsx b/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.stories.tsxnew file mode 100644index 000000000000..709afd51fa93--- /dev/null+++ b/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.stories.tsx@@ -0,0 +1,119 @@+import type { Meta, StoryObj } from '@storybook/react'+import { waitFor, within } from '@testing-library/dom'+import { BindLogic } from 'kea'+import { combineUrl } from 'kea-router'++import { FEATURE_FLAGS } from 'lib/constants'+import { urls } from 'scenes/urls'++import { mswDecorator } from '~/mocks/browser'++import type { DatasetItemReadApi, DatasetReadApi } from '../generated/api.schemas'+import { aiObservabilityDatasetLogic } from './aiObservabilityDatasetLogic'+import { AIObservabilityDatasetScene } from './AIObservabilityDatasetScene'++const dataset: DatasetReadApi = {+ id: 'example-dataset',+ name: 'Answer quality',+ description: 'Questions and expected answers for evaluating the support assistant.',+ metadata: { owner: 'support', language: 'en' },+ tags: [],+ archived: false,+ current_revision: 1,+ current_revision_id: 'example-revision',+ created_at: '2024-01-01T00:00:00Z',+ updated_at: '2024-01-01T00:00:00Z',+ created_by: null,+ team_id: 997,+ user_access_level: 'editor',+}++const item: DatasetItemReadApi = {+ id: 'example-item',+ dataset: dataset.id,+ client_item_id: null,+ version: 1,+ version_id: 'example-item-version',+ dataset_revision: 1,+ dataset_revision_id: 'example-revision',+ archived: false,+ input: { question: 'How do I reset my password?' },+ expected_output: { answer: 'Use the password reset link on the sign-in page.' },+ source_output: null,+ metadata: { category: 'account' },+ source_trace_id: null,+ source_timestamp: null,+ source_event_id: null,+ created_by: null,+ created_at: dataset.created_at,+ updated_at: dataset.created_at,+ version_created_at: dataset.created_at,+ version_created_by: null,+ team_id: 997,+}++const meta: Meta<typeof AIObservabilityDatasetScene> = {+ title: 'Scenes-App/AI observability/Dataset',+ component: AIObservabilityDatasetScene,+ parameters: {+ mockDate: '2024-01-02T12:00:00Z',+ pageUrl: urls.aiObservabilityDataset(dataset.id),+ featureFlags: [FEATURE_FLAGS.LLM_ANALYTICS_DATASETS, FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS],+ },+ decorators: [+ mswDecorator({+ get: {+ '/api/projects/:team/datasets/:id/': dataset,+ '/api/projects/:team/datasets/:id/revisions/': { count: 0, results: [], next: null, previous: null },+ '/api/projects/:team/dataset_items/': { count: 1, results: [item], next: null, previous: null },+ },+ }),+ (Story, { parameters }) => (+ <BindLogic logic={aiObservabilityDatasetLogic} props={{ datasetId: parameters.datasetId || dataset.id }}>+ <Story />+ </BindLogic>+ ),+ ],+ play: async ({ canvasElement }) => {+ await within(canvasElement).findByText(item.id)+ },+}+export default meta+type Story = StoryObj<typeof AIObservabilityDatasetScene>++export const Items: Story = {}+export const Details: Story = {+ parameters: { pageUrl: combineUrl(urls.aiObservabilityDataset(dataset.id), { tab: 'metadata' }).url },+}+export const NewDataset: Story = {+ parameters: { datasetId: 'new', pageUrl: urls.aiObservabilityDataset('new') },+ play: async ({ canvasElement }) => {+ await waitFor(() => {+ if (!canvasElement.querySelector('.monaco-editor')) {+ throw new Error('Waiting for the dataset metadata editor')+ }+ })+ },+}+export const DatasetsOnlyNarrow: Story = {+ parameters: {+ featureFlags: {+ [FEATURE_FLAGS.LLM_ANALYTICS_DATASETS]: true,+ [FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS]: false,+ },+ },+ decorators: [+ (Story) => (+ <div className="w-[520px]">+ <Story />+ </div>+ ),+ ],+}+export const ReadOnly: Story = {+ decorators: [+ mswDecorator({+ get: { '/api/projects/:team/datasets/:id/': { ...dataset, user_access_level: 'viewer' } },+ }),+ ],+}diff --git a/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.tsx b/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.tsxindex be08ebc1aecd..45b787a53cea 100644--- a/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.tsx+++ b/products/ai_observability/frontend/datasets/AIObservabilityDatasetScene.tsx@@ -1,49 +1,31 @@import { useActions, useValues } from 'kea'import { Form } from 'kea-forms'-import { combineUrl, router } from 'kea-router'+import { router } from 'kea-router'import { useEffect } from 'react'-import { IconArchive, IconDownload, IconPlusSmall } from '@posthog/icons'-import {- LemonBanner,- LemonButton,- LemonDivider,- LemonSegmentedButton,- LemonSelect,- LemonTab,- LemonTabs,- LemonTag,- Link,- ProfilePicture,-} from '@posthog/lemon-ui'+import { IconArchive, IconArrowLeft, IconDownload, IconInfo } from '@posthog/icons'+import { LemonBanner, LemonButton, LemonTag } from '@posthog/lemon-ui'import { Button } from '@posthog/quill'import { AccessControlAction } from 'lib/components/AccessControlAction'import { AccessDenied } from 'lib/components/AccessDenied'-import { HighlightedJSONViewer } from 'lib/components/HighlightedJSONViewer'import { NotFound } from 'lib/components/NotFound'import { SceneMenuBarFileItems } from 'lib/components/Scenes/SceneMenuBarFileItems'import { FEATURE_FLAGS } from 'lib/constants'import { useFeatureFlag } from 'lib/hooks/useFeatureFlag'-import { More } from 'lib/lemon-ui/LemonButton/More'import { LemonDialog } from 'lib/lemon-ui/LemonDialog'import { LemonSkeleton } from 'lib/lemon-ui/LemonSkeleton'-import { createdAtColumn, updatedAtColumn } from 'lib/lemon-ui/LemonTable/columnUtils'import { featureFlagLogic } from 'lib/logic/featureFlagLogic'import { ButtonPrimitive } from 'lib/ui/Button/ButtonPrimitives'import { toAccessControlLevel } from 'lib/utils/accessControlUtils'-import { isObject } from 'lib/utils/guards'import { SceneExport } from 'scenes/sceneTypes'-import { urls } from 'scenes/urls'-import { SceneContent } from '~/layout/scenes/components/SceneContent'import {SceneMenuBar,SceneMenuBarItem,SceneMenuBarMenu,SceneMenuBarSeparator,} from '~/layout/scenes/components/SceneMenuBar'-import { SceneTitleSection } from '~/layout/scenes/components/SceneTitleSection'import {ScenePanel,ScenePanelActionsSection,@@ -52,24 +34,15 @@ import {} from '~/layout/scenes/SceneLayout'import { SceneTextarea } from '~/lib/components/Scenes/SceneTextarea'import { SceneTextInput } from '~/lib/components/Scenes/SceneTextInput'-import { LemonTable, LemonTableColumn, LemonTableColumns } from '~/lib/lemon-ui/LemonTable'import { ProductKey } from '~/queries/schema/schema-general'-import { AccessControlLevel, AccessControlResourceType, type UserBasicType } from '~/types'+import { AccessControlLevel, AccessControlResourceType } from '~/types'-import type { DatasetItemReadApi as DatasetItem, DatasetReadApi as Dataset } from '../generated/api.schemas'-import { truncateValue } from '../utils'-import {- DATASET_ITEM_VERSIONS_PER_PAGE,- DatasetLogicProps,- DatasetTab,- isDataset,- aiObservabilityDatasetLogic,-} from './aiObservabilityDatasetLogic'+import { OfflineEvaluationsScene } from '../offline-evaluations/OfflineEvaluationsScene'+import { OfflineEvaluationsTabs } from '../offline-evaluations/OfflineEvaluationsTabs'+import { DatasetLogicProps, DatasetTab, isDataset, aiObservabilityDatasetLogic } from './aiObservabilityDatasetLogic'import { getDatasetListUrl } from './aiObservabilityDatasetsLogic'-import { DatasetItemModal } from './DatasetItemModal'+import { DatasetContents } from './DatasetContents'import { EditDatasetForm } from './EditDatasetForm'-import { JSONColumn } from './JSONColumn'-import { RefreshButton } from './RefreshButton'const RESOURCE_TYPE = 'dataset'@@ -110,6 +83,7 @@ export function AIObservabilityDatasetScene(): JSX.Element {exportDataset,setDatasetFormValue,onUnmount,+ setActiveTab,} = useActions(aiObservabilityDatasetLogic)const { searchParams } = useValues(router)const { featureFlags } = useValues(featureFlagLogic)@@ -156,113 +130,52 @@ export function AIObservabilityDatasetScene(): JSX.Element {useEffect(() => () => onUnmount(), [onUnmount])if (isDatasetMissing) {- return <NotFound object="dataset" />+ return (+ <OfflineEvaluationsScene>+ <NotFound object="dataset" />+ </OfflineEvaluationsScene>+ )}if (datasetLoadError?.status === 403) {- return <AccessDenied object="dataset" />+ return (+ <OfflineEvaluationsScene>+ <AccessDenied object="dataset" />+ </OfflineEvaluationsScene>+ )}if (datasetLoadError && !isDataset(dataset)) {return (- <SceneContent>+ <OfflineEvaluationsScene><LemonBanner type="error" action={{ children: 'Try again', onClick: loadDataset }}>{datasetLoadError.detail || "Couldn't load this dataset. Try again."}</LemonBanner>- </SceneContent>+ </OfflineEvaluationsScene>)}if (shouldDisplaySkeleton) {return (- <div className="flex flex-col gap-2">- <LemonSkeleton active className="h-4 w-2/5" />- <LemonSkeleton active className="h-4 w-full" />- <LemonSkeleton active className="h-4 w-full" />- <LemonSkeleton active className="h-4 w-3/5" />- </div>+ <OfflineEvaluationsScene>+ <div className="flex flex-col gap-2">+ <LemonSkeleton active className="h-4 w-2/5" />+ <LemonSkeleton active className="h-4 w-full" />+ <LemonSkeleton active className="h-4 w-full" />+ <LemonSkeleton active className="h-4 w-3/5" />+ </div>+ </OfflineEvaluationsScene>)}return (<Form id="dataset-form" formKey="datasetForm" logic={aiObservabilityDatasetLogic}>- <SceneContent>+ <OfflineEvaluationsScene>{datasetLoadError && (<LemonBanner type="error" action={{ children: 'Try again', onClick: loadDataset }}>{datasetLoadError.detail || "Couldn't refresh this dataset. Try again."}</LemonBanner>)}- <SceneTitleSection- name={datasetForm.name}- resourceType={{ type: 'llm_analytics' }}- isLoading={datasetLoading}- actions={- !shouldDisplaySkeleton && displayEditForm ? (- todayRailNav ? (- <>- <Button- variant="outline"- data-attr="cancel-dataset"- onClick={cancelEditing}- disabledReason={isDatasetFormSubmitting ? 'Saving…' : undefined}- >- Cancel- </Button>- <AccessControlAction- resourceType={AccessControlResourceType.LlmAnalytics}- minAccessLevel={AccessControlLevel.Editor}- userAccessLevel={- isDataset(dataset)- ? toAccessControlLevel(dataset.user_access_level)- : undefined- }- >- <Button- variant="primary"- data-attr="save-dataset"- onClick={submitDatasetForm}- loading={isDatasetFormSubmitting}- >- {isNewDataset ? 'Create dataset' : 'Save'}- </Button>- </AccessControlAction>- </>- ) : (- <>- <LemonButton- type="secondary"- data-attr="cancel-dataset"- onClick={cancelEditing}- disabledReason={isDatasetFormSubmitting ? 'Saving…' : undefined}- size="small"- >- Cancel- </LemonButton>- <AccessControlAction- resourceType={AccessControlResourceType.LlmAnalytics}- minAccessLevel={AccessControlLevel.Editor}- userAccessLevel={- isDataset(dataset)- ? toAccessControlLevel(dataset.user_access_level)- : undefined- }- >- <LemonButton- type="primary"- data-attr="save-dataset"- onClick={submitDatasetForm}- loading={isDatasetFormSubmitting}- size="small"- >- {isNewDataset ? 'Create dataset' : 'Save'}- </LemonButton>- </AccessControlAction>- </>- )- ) : undefined- }- />-{isDataset(dataset) && sceneMenuBarEnabled && (<SceneMenuBar><SceneMenuBarMenu label="File" dataAttr={`${RESOURCE_TYPE}-menubar-file`}>@@ -345,455 +258,127 @@ export function AIObservabilityDatasetScene(): JSX.Element {</ScenePanel>)}- {displayEditForm ? <EditDatasetForm /> : isDataset(dataset) ? <DatasetTabs dataset={dataset} /> : null}- </SceneContent>- </Form>- )-}--function DatasetTabs({ dataset }: { dataset: Dataset }): JSX.Element {- const {- activeTab,- isDatasetItemModalOpen,- selectedDatasetItem,- selectedDatasetItemLoadError,- selectedDatasetItemDetailsLoading,- selectedDatasetItemVersions,- datasetItemVersionsLoading,- datasetItemVersionsLoadError,- datasetItemVersionsPage,- restoringDatasetItemVersion,- archivingDatasetItemId,- canEditDataset,- canManageDataset,- isHistoricalRevision,- filters,- datasetExportLoading,- datasetExportLoadError,- datasetExportDisabledReason,- } = useValues(aiObservabilityDatasetLogic)- const {- closeModalAndRefetchDatasetItems,- loadDatasetItemDetails,- restoreDatasetItem,- restoreDatasetItemVersion,- loadDatasetItemVersions,- setFilters,- exportDataset,- editDataset,- } = useActions(aiObservabilityDatasetLogic)- const { searchParams } = useValues(router)- const selectedItemReadOnly = !canEditDataset || !!selectedDatasetItem?.archived- const selectedItemReadOnlyReason = isHistoricalRevision- ? `You're viewing revision ${filters.revision}. Select Latest to edit items.`- : dataset.archived- ? 'This dataset is archived. Unarchive it to edit items.'- : selectedDatasetItem?.archived- ? 'This item is archived. Unarchive it to edit.'- : !canManageDataset- ? 'You have view-only access to this dataset.'- : undefined-- const tabs: LemonTab<DatasetTab>[] = [- {- key: DatasetTab.Items,- label: 'Items',- content: <DatasetItems dataset={dataset} />,- link: combineUrl(urls.aiObservabilityDataset(dataset.id), { ...searchParams, tab: DatasetTab.Items }).url,- },- {- key: DatasetTab.Metadata,- label: 'Metadata',- content: <DatasetMetadata dataset={dataset} />,- link: combineUrl(urls.aiObservabilityDataset(dataset.id), { ...searchParams, tab: DatasetTab.Metadata })- .url,- },- ]-- return (- <>- <div className="flex flex-col gap-2">- <div className="flex items-center justify-between gap-2">- <div className="flex items-center gap-2">- <LemonTag type={dataset.archived ? 'muted' : 'success'}>- {dataset.archived ? 'Archived' : 'Active'}- </LemonTag>- {isHistoricalRevision && <LemonTag type="default">Revision {filters.revision}</LemonTag>}+ <div className="flex flex-wrap items-center gap-3">+ <OfflineEvaluationsTabs activeTab="datasets" />+ <div className="flex flex-1 basis-80 flex-wrap items-center gap-2">+ <nav aria-label="Dataset" className="flex min-w-0 flex-1 basis-40 items-center gap-2">+ <LemonButton+ size="small"+ icon={<IconArrowLeft />}+ aria-label="Back to datasets"+ tooltip="Back to datasets"+ data-attr="back-to-datasets"+ to={getDatasetListUrl(searchParams)}+ />+ <h2 className="text-base font-semibold m-0 truncate" title={datasetForm.name}>+ {isNewDataset ? 'New dataset' : (isDataset(dataset) && dataset.name) || 'Dataset'}+ </h2>+ {isDataset(dataset) && dataset.archived && (+ <LemonTag type="muted" size="small">+ Archived+ </LemonTag>+ )}+ </nav>+ <div className="flex flex-wrap items-center gap-2">+ {!displayEditForm && isDataset(dataset) ? (+ <>+ <LemonButton+ size="small"+ onClick={() => setActiveTab(DatasetTab.Metadata)}+ data-attr="dataset-details"+ icon={<IconInfo />}+ >+ Details+ </LemonButton>+ {canEditDataset && (+ <LemonButton+ type="secondary"+ size="small"+ onClick={() => editDataset(true)}+ data-attr="edit-dataset"+ >+ Edit+ </LemonButton>+ )}+ </>+ ) : null}+ </div></div>- {canEditDataset && (- <LemonButton- type="secondary"- onClick={() => editDataset(true)}- data-attr="edit-dataset"- size="small"- >- Edit- </LemonButton>- )}</div>- <p className="m-0">{dataset.description || <span className="italic">Description (optional)</span>}</p>- {isHistoricalRevision && (- <LemonBanner- type="info"- action={{- children: 'View latest',- onClick: () => setFilters({ revision: null, page: 1 }, false),- }}- >- This is a read-only snapshot of the dataset.- </LemonBanner>+ {!displayEditForm && isDataset(dataset) && dataset.description && (+ <p className="text-muted m-0">{dataset.description}</p>)}- {datasetExportLoadError && (- <LemonBanner- type="error"- action={{- children: 'Try again',- onClick: () => exportDataset(filters.revision ?? undefined),- loading: datasetExportLoading,- disabledReason: datasetExportDisabledReason,- }}- >- {datasetExportLoadError.detail || "Couldn't add the dataset to exports. Try again."}- </LemonBanner>- )}- </div>-- <LemonTabs activeKey={activeTab} data-attr="dataset-tabs" tabs={tabs} />-- <DatasetItemModal- isOpen={isDatasetItemModalOpen}- onClose={closeModalAndRefetchDatasetItems}- partialDatasetItem={selectedDatasetItem}- datasetId={dataset.id}- displayBulkCreationButton- readOnly={selectedItemReadOnly}- readOnlyReason={selectedItemReadOnlyReason}- loading={selectedDatasetItemDetailsLoading}- loadError={selectedDatasetItemLoadError}- versions={selectedDatasetItemVersions.results}- versionsLoading={datasetItemVersionsLoading}- versionsLoadError={datasetItemVersionsLoadError}- versionsCount={selectedDatasetItemVersions.count}- versionsPage={datasetItemVersionsPage}- versionsPageSize={DATASET_ITEM_VERSIONS_PER_PAGE}- canRestoreVersions={canEditDataset && !selectedDatasetItem?.archived}- restoringVersion={restoringDatasetItemVersion}- onRestoreVersion={restoreDatasetItemVersion}- onVersionsPageChange={- selectedDatasetItem- ? (page) => loadDatasetItemVersions({ itemId: selectedDatasetItem.id, page })- : undefined- }- onRetryVersions={- selectedDatasetItem- ? () =>- loadDatasetItemVersions({- itemId: selectedDatasetItem.id,- page: datasetItemVersionsPage,- })- : undefined- }- onRetry={- searchParams.item- ? () =>- loadDatasetItemDetails({- itemId: String(searchParams.item),- revision: filters.revision ?? undefined,- })- : undefined- }- onUnarchive={- selectedDatasetItem?.archived && canManageDataset && !dataset.archived && !isHistoricalRevision- ? () => restoreDatasetItem(selectedDatasetItem.id, selectedDatasetItem.version)- : undefined- }- unarchiving={archivingDatasetItemId === selectedDatasetItem?.id}- />- </>- )-}--function DatasetItems({ dataset }: { dataset: Dataset }): JSX.Element {- const {- archivingDatasetItemId,- datasetItems,- datasetLoading,- datasetItemsLoading,- datasetItemsLoadError,- datasetRevisionsLoading,- datasetRevisionsLoadError,- pagination,- filters,- revisionOptions,- canEditDataset,- canManageDataset,- isHistoricalRevision,- isArchivingDataset,- datasetExportLoading,- datasetExportDisabledReason,- } = useValues(aiObservabilityDatasetLogic)- const {- archiveDatasetItem,- restoreDatasetItem,- loadDatasetItems,- loadDatasetRevisions,- refreshDataset,- setFilters,- restoreDataset,- exportDataset,- triggerDatasetItemModal,- } = useActions(aiObservabilityDatasetLogic)- const { searchParams } = useValues(router)- const canChangeDatasetStatus = canManageDataset && !isHistoricalRevision-- const columns: LemonTableColumns<DatasetItem> = [- {- title: 'ID',- dataIndex: 'id',- key: 'id',- width: '10%',- render: function renderID(_, item) {- return (- <Link to={urls.aiObservabilityDataset(dataset.id, { ...searchParams, item: item.id })}>- {truncateValue(item.id)}- </Link>- )- },- },- {- title: 'Trace',- dataIndex: 'source_trace_id',- key: 'source_trace_id',- width: '10%',- render: function renderSourceTraceId(_, item) {- if (!item.source_trace_id || !item.source_timestamp) {- return <span>-</span>- }-- return (- <Link- to={urls.aiObservabilityTrace(item.source_trace_id, {- event: item.source_event_id ?? undefined,- timestamp: item.source_timestamp,- })}- >- {truncateValue(item.source_trace_id)}- </Link>- )- },- },- {- title: 'Input',- dataIndex: 'input',- key: 'input',- width: '20%',- render: function renderInput(_, item) {- return <JSONColumn>{item.input}</JSONColumn>- },- },- {- title: 'Expected output',- dataIndex: 'expected_output',- key: 'expected_output',- width: '20%',- render: function renderExpectedOutput(_, item) {- return <JSONColumn>{item.expected_output}</JSONColumn>- },- },- {- title: 'Source output',- dataIndex: 'source_output',- key: 'source_output',- width: '20%',- render: function renderSourceOutput(_, item) {- return <JSONColumn>{item.source_output}</JSONColumn>- },- },- {- title: 'Metadata',- dataIndex: 'metadata',- key: 'metadata',- width: '25%',- render: function renderMetadata(_, item) {- return <JSONColumn>{item.metadata}</JSONColumn>- },- },- {- title: 'Created by',- dataIndex: 'created_by',- render: function renderCreatedBy(_, item) {- const { created_by } = item- return (- <div className="flex flex-row items-center flex-nowrap">- {created_by && <ProfilePicture user={created_by as UserBasicType} size="md" showName />}- </div>- )- },- },- {- title: 'Status',- key: 'status',- render: function renderStatus(_, item) {- return (- <LemonTag type={item.archived ? 'muted' : 'success'}>- {item.archived ? 'Archived' : 'Active'}- </LemonTag>- )- },- },- createdAtColumn<DatasetItem>() as LemonTableColumn<DatasetItem, keyof DatasetItem | undefined>,- updatedAtColumn<DatasetItem>() as LemonTableColumn<DatasetItem, keyof DatasetItem | undefined>,- {- width: 0,- render: function renderMore(_, item) {- return (- <More- overlay={- <>- <LemonButton- to={urls.aiObservabilityDataset(dataset.id, { ...searchParams, item: item.id })}- data-attr={`dataset-item-${item.id}-dropdown-edit`}- fullWidth- >- {canEditDataset && !item.archived ? 'Edit' : 'View'}- </LemonButton>-- {canManageDataset && !dataset.archived && !isHistoricalRevision && (- <LemonButton- status={item.archived ? undefined : 'danger'}- onClick={() =>- item.archived- ? restoreDatasetItem(item.id, item.version)- : archiveDatasetItem(item.id, item.version)- }- loading={archivingDatasetItemId === item.id}- disabledReason={- archivingDatasetItemId && archivingDatasetItemId !== item.id- ? 'Another dataset item is being archived'+ {displayEditForm ? (+ <div className="flex max-w-160 flex-col gap-4">+ <EditDatasetForm />+ <div className="flex flex-wrap items-center gap-2">+ {todayRailNav ? (+ <>+ <Button+ variant="outline"+ data-attr="cancel-dataset"+ onClick={cancelEditing}+ disabledReason={isDatasetFormSubmitting ? 'Saving…' : undefined}+ >+ Cancel+ </Button>+ <AccessControlAction+ resourceType={AccessControlResourceType.LlmAnalytics}+ minAccessLevel={AccessControlLevel.Editor}+ userAccessLevel={+ isDataset(dataset)+ ? toAccessControlLevel(dataset.user_access_level): undefined}- data-attr={`dataset-item-${item.id}-dropdown-${- item.archived ? 'unarchive' : 'archive'- }`}- fullWidth>- {item.archived ? 'Unarchive' : 'Archive'}+ <Button+ variant="primary"+ data-attr="save-dataset"+ onClick={submitDatasetForm}+ loading={isDatasetFormSubmitting}+ >+ {isNewDataset ? 'Create dataset' : 'Save'}+ </Button>+ </AccessControlAction>+ </>+ ) : (+ <>+ <LemonButton+ type="secondary"+ data-attr="cancel-dataset"+ onClick={cancelEditing}+ disabledReason={isDatasetFormSubmitting ? 'Saving…' : undefined}+ size="small"+ >+ Cancel</LemonButton>- )}- </>- }- />- )- },- },- ]-- return (- <>- <div className="flex justify-between items-center gap-2 flex-wrap">- <div className="flex items-center gap-2 flex-wrap">- <LemonSegmentedButton- value={filters.archived ? 'archived' : 'active'}- onChange={(value) => setFilters({ archived: value === 'archived', page: 1 }, false)}- options={[- { value: 'active', label: 'Active' },- { value: 'archived', label: 'Archived' },- ]}- size="small"- data-attr="dataset-items-status-filter"- />- <LemonSelect- value={filters.revision === null ? 'latest' : String(filters.revision)}- onChange={(value) =>- setFilters({ revision: value === 'latest' ? null : Number(value), page: 1 }, false)- }- options={revisionOptions}- loading={datasetRevisionsLoading}- size="small"- data-attr="dataset-revision-select"- />- </div>- <div className="flex items-center gap-2 flex-wrap">- <RefreshButton- onClick={refreshDataset}- isRefreshing={datasetItemsLoading || datasetLoading || datasetRevisionsLoading}- />- <LemonButton- type="secondary"- onClick={() => exportDataset(filters.revision ?? undefined)}- loading={datasetExportLoading}- disabledReason={datasetExportDisabledReason}- data-attr="export-dataset"- icon={<IconDownload />}- size="small"- >- Export- </LemonButton>- {canEditDataset && (- <LemonButton- type="primary"- onClick={() => triggerDatasetItemModal(true)}- data-attr="add-dataset-item"- icon={<IconPlusSmall />}- size="small"- >- Add item- </LemonButton>- )}- {dataset.archived && canChangeDatasetStatus && (- <LemonButton- type="primary"- onClick={restoreDataset}- loading={isArchivingDataset}- data-attr="unarchive-dataset"- size="small"- >- Unarchive- </LemonButton>- )}- </div>- </div>-- {datasetItemsLoadError && (- <LemonBanner type="error" action={{ children: 'Try again', onClick: () => loadDatasetItems(false) }}>- {datasetItemsLoadError.detail || "Couldn't load dataset items. Try again."}- </LemonBanner>- )}-- {datasetRevisionsLoadError && (- <LemonBanner type="error" action={{ children: 'Try again', onClick: loadDatasetRevisions }}>- {datasetRevisionsLoadError.detail || "Couldn't load dataset revisions. Try again."}- </LemonBanner>- )}-- <LemonDivider className="my-4" />-- <LemonTable- loading={datasetItemsLoading}- columns={columns}- dataSource={datasetItems?.results || []}- rowKey="id"- pagination={pagination}- nouns={['dataset item', 'dataset items']}- emptyState="No items in this dataset yet."- />- </>- )-}--function DatasetMetadata({ dataset }: { dataset: Dataset }): JSX.Element {- return (- <div className="flex flex-col gap-4 max-w-160">- <div className="flex flex-col gap-2">- <h3 className="text-sm font-semibold m-0">Metadata</h3>- {isObject(dataset.metadata) ? (- <div className="bg-bg-light p-4 rounded border overflow-x-auto">- <HighlightedJSONViewer src={dataset.metadata} />+ <AccessControlAction+ resourceType={AccessControlResourceType.LlmAnalytics}+ minAccessLevel={AccessControlLevel.Editor}+ userAccessLevel={+ isDataset(dataset)+ ? toAccessControlLevel(dataset.user_access_level)+ : undefined+ }+ >+ <LemonButton+ type="primary"+ data-attr="save-dataset"+ onClick={submitDatasetForm}+ loading={isDatasetFormSubmitting}+ size="small"+ >+ {isNewDataset ? 'Create dataset' : 'Save'}+ </LemonButton>+ </AccessControlAction>+ </>+ )}+ </div></div>- ) : (- <p className="text-muted m-0">No metadata available.</p>- )}- </div>- </div>+ ) : isDataset(dataset) ? (+ <DatasetContents dataset={dataset} />+ ) : null}+ </OfflineEvaluationsScene>+ </Form>)}diff --git a/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx b/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsxindex 4252dc54025e..03815c7aea88 100644--- a/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx+++ b/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx@@ -1,5 +1,6 @@import { useActions, useValues } from 'kea'import { router } from 'kea-router'+import type { ReactNode } from 'react'import { AccessControlAction } from 'lib/components/AccessControlAction'import { LemonButton } from 'lib/lemon-ui/LemonButton'@@ -8,8 +9,6 @@ import { Link } from 'lib/lemon-ui/Link'import { ProfilePicture } from 'lib/lemon-ui/ProfilePicture'import { SceneExport } from 'scenes/sceneTypes'-import { SceneContent } from '~/layout/scenes/components/SceneContent'-import { SceneTitleSection } from '~/layout/scenes/components/SceneTitleSection'import { LemonInput } from '~/lib/lemon-ui/LemonInput'import { LemonSegmentedButton } from '~/lib/lemon-ui/LemonSegmentedButton'import { LemonTable, LemonTableColumn, LemonTableColumns } from '~/lib/lemon-ui/LemonTable'@@ -32,7 +31,7 @@ export const scene: SceneExport = {emptyState: datasetsEmptyState,}-export function AIObservabilityDatasetsScene(): JSX.Element {+export function AIObservabilityDatasetsScene({ navigation }: { navigation?: ReactNode }): JSX.Element {const { loadDatasets, setFilters, archiveDataset, restoreDataset } = useActions(aiObservabilityDatasetsLogic)const { archivingDatasetId, datasets, datasetsLoading, sorting, pagination, filters, datasetCountLabel } =useValues(aiObservabilityDatasetsLogic)@@ -44,13 +43,13 @@ export function AIObservabilityDatasetsScene(): JSX.Element {title: 'Name',dataIndex: 'name',key: 'name',- width: '25%',+ width: '21%',render: function renderName(_, dataset) {return (<Linkto={datasetUrl(dataset.id)}data-testid="dataset-link"- className="block max-w-64 @max-[50rem]/datasets:max-w-32 truncate font-semibold"+ className="block max-w-full @max-[50rem]/datasets:max-w-48 truncate font-semibold">{dataset.name}</Link>@@ -61,15 +60,20 @@ export function AIObservabilityDatasetsScene(): JSX.Element {title: 'Description',dataIndex: 'description',key: 'description',- width: '20%',+ width: '16%',className: '@max-[50rem]/datasets:hidden',render: function renderDescription(description) {- return <span className="text-muted">{String(description) || <i>–</i>}</span>+ return (+ <span className="block max-w-full truncate text-muted" title={String(description) || undefined}>+ {String(description) || <i>–</i>}+ </span>+ )},},{title: 'Created by',- className: '@max-[50rem]/datasets:hidden',+ width: '14%',+ className: '@max-[50rem]/datasets:hidden [&_.profile-package]:max-w-full [&_.profile-name]:truncate',dataIndex: 'created_by',render: function renderCreatedBy(_, item) {const { created_by } = item@@ -94,9 +98,10 @@ export function AIObservabilityDatasetsScene(): JSX.Element {{title: 'Tags',key: 'tags',- width: '20%',+ width: '16%',render: (_, dataset) => (<ResourceTags+ compactentity="datasets"resourceId={dataset.id}tags={dataset.tags}@@ -109,14 +114,16 @@ export function AIObservabilityDatasetsScene(): JSX.Element {},{...(createdAtColumn<Dataset>() as LemonTableColumn<Dataset, keyof Dataset | undefined>),- className: '@max-[60rem]/datasets:hidden',+ width: '11%',+ className: '@max-[50rem]/datasets:hidden',},{...(updatedAtColumn<Dataset>() as LemonTableColumn<Dataset, keyof Dataset | undefined>),+ width: '11%',className: '@max-[50rem]/datasets:hidden',},{- width: 0,+ width: 40,render: function renderMore(_, dataset) {return (<More@@ -163,38 +170,32 @@ export function AIObservabilityDatasetsScene(): JSX.Element {]return (- <SceneContent className="@container/datasets">- <SceneTitleSection- name="Datasets"- description="Create and manage datasets to test changes and evaluate your AI outputs. [Learn more](https://posthog.com/docs/ai-evals/datasets)"- markdown- resourceType={{ type: 'llm_datasets' }}- />- <div className="flex gap-x-4 gap-y-2 items-center flex-wrap py-4 -mt-4 mb-4 border-b justify-between">- <div className="flex gap-2 items-center flex-wrap">- <LemonSegmentedButton- value={filters.archived ? 'archived' : 'active'}- onChange={(value) => setFilters({ archived: value === 'archived' }, true, false)}- options={[- { value: 'active', label: 'Active' },- { value: 'archived', label: 'Archived' },- ]}- size="small"- data-attr="datasets-status-filter"- />- <ResourceTagFilter filters={filters} onChange={(tags) => setFilters(tags, true, false)} />- <LemonInput- type="search"- placeholder="Search datasets..."- value={filters.search}- data-attr="datasets-search-input"- onChange={(value) => setFilters({ search: value })}- className="max-w-md"- data-testid="search-datasets-input"- />- </div>- <div className="flex gap-2 items-center flex-wrap">- <div className="text-muted-alt">{datasetCountLabel}</div>+ <div className="@container/datasets space-y-4">+ <div className="flex gap-2 items-center flex-wrap" aria-label="Dataset filters">+ {navigation}+ <LemonSegmentedButton+ value={filters.archived ? 'archived' : 'active'}+ onChange={(value) => setFilters({ archived: value === 'archived' }, true, false)}+ options={[+ { value: 'active', label: 'Active' },+ { value: 'archived', label: 'Archived' },+ ]}+ size="small"+ data-attr="datasets-status-filter"+ />+ <ResourceTagFilter size="small" filters={filters} onChange={(tags) => setFilters(tags, true, false)} />+ <LemonInput+ size="small"+ type="search"+ placeholder="Search datasets..."+ value={filters.search}+ data-attr="datasets-search-input"+ onChange={(value) => setFilters({ search: value })}+ className="max-w-md"+ data-testid="search-datasets-input"+ />+ <div className="flex gap-2 items-center flex-wrap ml-auto">+ <div className="text-muted-alt text-xs">{datasetCountLabel}</div><AccessControlActionresourceType={AccessControlResourceType.LlmAnalytics}minAccessLevel={AccessControlLevel.Editor}@@ -214,7 +215,9 @@ export function AIObservabilityDatasetsScene(): JSX.Element {<LemonTableid="datasets"- className="@max-[50rem]/datasets:[&_colgroup]:hidden"+ className="@max-[50rem]/datasets:[&_colgroup]:hidden @max-[50rem]/datasets:[&_table]:table-auto"+ size="small"+ tableLayout="fixed"loading={datasetsLoading}columns={columns}dataSource={datasets.results}@@ -232,6 +235,6 @@ export function AIObservabilityDatasetsScene(): JSX.Element {loadingSkeletonRows={DATASETS_PER_PAGE}nouns={['dataset', 'datasets']}/>- </SceneContent>+ </div>)}diff --git a/products/ai_observability/frontend/datasets/DatasetContents.tsx b/products/ai_observability/frontend/datasets/DatasetContents.tsxnew file mode 100644index 000000000000..8c3ac36510f3--- /dev/null+++ b/products/ai_observability/frontend/datasets/DatasetContents.tsx@@ -0,0 +1,139 @@+import { useActions, useValues } from 'kea'+import { router } from 'kea-router'++import { LemonBanner } from '@posthog/lemon-ui'++import type { DatasetReadApi as Dataset } from '../generated/api.schemas'+import { DATASET_ITEM_VERSIONS_PER_PAGE, DatasetTab, aiObservabilityDatasetLogic } from './aiObservabilityDatasetLogic'+import { DatasetDetailsModal } from './DatasetDetailsModal'+import { DatasetItemModal } from './DatasetItemModal'+import { DatasetItems } from './DatasetItems'++export function DatasetContents({ dataset }: { dataset: Dataset }): JSX.Element {+ const {+ activeTab,+ isDatasetItemModalOpen,+ selectedDatasetItem,+ selectedDatasetItemLoadError,+ selectedDatasetItemDetailsLoading,+ selectedDatasetItemVersions,+ datasetItemVersionsLoading,+ datasetItemVersionsLoadError,+ datasetItemVersionsPage,+ restoringDatasetItemVersion,+ archivingDatasetItemId,+ canEditDataset,+ canManageDataset,+ isHistoricalRevision,+ filters,+ datasetExportLoading,+ datasetExportLoadError,+ datasetExportDisabledReason,+ } = useValues(aiObservabilityDatasetLogic)+ const {+ closeModalAndRefetchDatasetItems,+ loadDatasetItemDetails,+ restoreDatasetItem,+ restoreDatasetItemVersion,+ loadDatasetItemVersions,+ setFilters,+ setActiveTab,+ exportDataset,+ } = useActions(aiObservabilityDatasetLogic)+ const { searchParams } = useValues(router)+ const selectedItemReadOnly = !canEditDataset || !!selectedDatasetItem?.archived+ const selectedItemReadOnlyReason = isHistoricalRevision+ ? `You're viewing revision ${filters.revision}. Select Latest to edit items.`+ : dataset.archived+ ? 'This dataset is archived. Unarchive it to edit items.'+ : selectedDatasetItem?.archived+ ? 'This item is archived. Unarchive it to edit.'+ : !canManageDataset+ ? 'You have view-only access to this dataset.'+ : undefined++ return (+ <>+ {isHistoricalRevision && (+ <LemonBanner+ type="info"+ action={{+ children: 'View latest',+ onClick: () => setFilters({ revision: null, page: 1 }, false),+ }}+ >+ This is a read-only snapshot of the dataset.+ </LemonBanner>+ )}+ {datasetExportLoadError && (+ <LemonBanner+ type="error"+ action={{+ children: 'Try again',+ onClick: () => exportDataset(filters.revision ?? undefined),+ loading: datasetExportLoading,+ disabledReason: datasetExportDisabledReason,+ }}+ >+ {datasetExportLoadError.detail || "Couldn't add the dataset to exports. Try again."}+ </LemonBanner>+ )}+ <DatasetItems dataset={dataset} />+ <DatasetDetailsModal+ dataset={dataset}+ isOpen={activeTab === DatasetTab.Metadata && !isDatasetItemModalOpen}+ onClose={() => setActiveTab(DatasetTab.Items)}+ />++ <DatasetItemModal+ isOpen={isDatasetItemModalOpen}+ onClose={closeModalAndRefetchDatasetItems}+ partialDatasetItem={selectedDatasetItem}+ datasetId={dataset.id}+ displayBulkCreationButton+ readOnly={selectedItemReadOnly}+ readOnlyReason={selectedItemReadOnlyReason}+ loading={selectedDatasetItemDetailsLoading}+ loadError={selectedDatasetItemLoadError}+ versions={selectedDatasetItemVersions.results}+ versionsLoading={datasetItemVersionsLoading}+ versionsLoadError={datasetItemVersionsLoadError}+ versionsCount={selectedDatasetItemVersions.count}+ versionsPage={datasetItemVersionsPage}+ versionsPageSize={DATASET_ITEM_VERSIONS_PER_PAGE}+ canRestoreVersions={canEditDataset && !selectedDatasetItem?.archived}+ restoringVersion={restoringDatasetItemVersion}+ onRestoreVersion={restoreDatasetItemVersion}+ onVersionsPageChange={+ selectedDatasetItem+ ? (page) => loadDatasetItemVersions({ itemId: selectedDatasetItem.id, page })+ : undefined+ }+ onRetryVersions={+ selectedDatasetItem+ ? () =>+ loadDatasetItemVersions({+ itemId: selectedDatasetItem.id,+ page: datasetItemVersionsPage,+ })+ : undefined+ }+ onRetry={+ searchParams.item+ ? () =>+ loadDatasetItemDetails({+ itemId: String(searchParams.item),+ revision: filters.revision ?? undefined,+ })+ : undefined+ }+ onUnarchive={+ selectedDatasetItem?.archived && canManageDataset && !dataset.archived && !isHistoricalRevision+ ? () => restoreDatasetItem(selectedDatasetItem.id, selectedDatasetItem.version)+ : undefined+ }+ unarchiving={archivingDatasetItemId === selectedDatasetItem?.id}+ />+ </>+ )+}diff --git a/products/ai_observability/frontend/datasets/DatasetDetailsModal.tsx b/products/ai_observability/frontend/datasets/DatasetDetailsModal.tsxnew file mode 100644index 000000000000..32775e45beac--- /dev/null+++ b/products/ai_observability/frontend/datasets/DatasetDetailsModal.tsx@@ -0,0 +1,43 @@+import { LemonModal } from '@posthog/lemon-ui'++import { HighlightedJSONViewer } from 'lib/components/HighlightedJSONViewer'+import { isObject } from 'lib/utils/guards'++import type { DatasetReadApi } from '../generated/api.schemas'++export function DatasetDetailsModal({+ dataset,+ isOpen,+ onClose,+}: {+ dataset: DatasetReadApi+ isOpen: boolean+ onClose: () => void+}): JSX.Element {+ return (+ <LemonModal title="Dataset details" isOpen={isOpen} onClose={onClose} width="40rem">+ <div className="flex flex-col gap-4">+ <div>+ <h4 className="mb-1">Name</h4>+ <p className="m-0 break-words">{dataset.name}</p>+ </div>+ {dataset.description && (+ <div>+ <h4 className="mb-1">Description</h4>+ <p className="m-0 break-words">{dataset.description}</p>+ </div>+ )}+ <div>+ <h4 className="mb-1">Metadata</h4>+ {isObject(dataset.metadata) && Object.keys(dataset.metadata).length > 0 ? (+ <div className="overflow-x-auto">+ <HighlightedJSONViewer src={dataset.metadata} />+ </div>+ ) : (+ <p className="text-muted m-0">No metadata.</p>+ )}+ </div>+ </div>+ </LemonModal>+ )+}diff --git a/products/ai_observability/frontend/datasets/DatasetItems.tsx b/products/ai_observability/frontend/datasets/DatasetItems.tsxnew file mode 100644index 000000000000..e802bf47dc4f--- /dev/null+++ b/products/ai_observability/frontend/datasets/DatasetItems.tsx@@ -0,0 +1,293 @@+import { useActions, useValues } from 'kea'+import { router } from 'kea-router'++import { IconDownload, IconPlusSmall } from '@posthog/icons'+import {+ LemonBanner,+ LemonButton,+ LemonSegmentedButton,+ LemonSelect,+ LemonTag,+ Link,+ ProfilePicture,+} from '@posthog/lemon-ui'++import { More } from 'lib/lemon-ui/LemonButton/More'+import { LemonTable, LemonTableColumn, LemonTableColumns } from 'lib/lemon-ui/LemonTable'+import { createdAtColumn, updatedAtColumn } from 'lib/lemon-ui/LemonTable/columnUtils'+import { urls } from 'scenes/urls'++import { UserBasicType } from '~/types'++import type { DatasetItemReadApi as DatasetItem, DatasetReadApi as Dataset } from '../generated/api.schemas'+import { truncateValue } from '../utils'+import { aiObservabilityDatasetLogic } from './aiObservabilityDatasetLogic'+import { JSONColumn } from './JSONColumn'+import { RefreshButton } from './RefreshButton'++export function DatasetItems({ dataset }: { dataset: Dataset }): JSX.Element {+ const {+ archivingDatasetItemId,+ datasetItems,+ datasetLoading,+ datasetItemsLoading,+ datasetItemsLoadError,+ datasetRevisionsLoading,+ datasetRevisionsLoadError,+ pagination,+ filters,+ revisionOptions,+ canEditDataset,+ canManageDataset,+ isHistoricalRevision,+ isArchivingDataset,+ datasetExportLoading,+ datasetExportDisabledReason,+ } = useValues(aiObservabilityDatasetLogic)+ const {+ archiveDatasetItem,+ restoreDatasetItem,+ loadDatasetItems,+ loadDatasetRevisions,+ refreshDataset,+ setFilters,+ restoreDataset,+ exportDataset,+ triggerDatasetItemModal,+ } = useActions(aiObservabilityDatasetLogic)+ const { searchParams } = useValues(router)+ const canChangeDatasetStatus = canManageDataset && !isHistoricalRevision++ const columns: LemonTableColumns<DatasetItem> = [+ {+ title: 'ID',+ dataIndex: 'id',+ key: 'id',+ width: '10%',+ render: function renderID(_, item) {+ return (+ <Link to={urls.aiObservabilityDataset(dataset.id, { ...searchParams, item: item.id })}>+ {truncateValue(item.id)}+ </Link>+ )+ },+ },+ {+ title: 'Trace',+ dataIndex: 'source_trace_id',+ key: 'source_trace_id',+ width: '10%',+ render: function renderSourceTraceId(_, item) {+ if (!item.source_trace_id || !item.source_timestamp) {+ return <span>-</span>+ }++ return (+ <Link+ to={urls.aiObservabilityTrace(item.source_trace_id, {+ event: item.source_event_id ?? undefined,+ timestamp: item.source_timestamp,+ })}+ >+ {truncateValue(item.source_trace_id)}+ </Link>+ )+ },+ },+ {+ title: 'Input',+ dataIndex: 'input',+ key: 'input',+ width: '20%',+ render: function renderInput(_, item) {+ return <JSONColumn>{item.input}</JSONColumn>+ },+ },+ {+ title: 'Expected output',+ dataIndex: 'expected_output',+ key: 'expected_output',+ width: '20%',+ render: function renderExpectedOutput(_, item) {+ return <JSONColumn>{item.expected_output}</JSONColumn>+ },+ },+ {+ title: 'Source output',+ dataIndex: 'source_output',+ key: 'source_output',+ width: '20%',+ render: function renderSourceOutput(_, item) {+ return <JSONColumn>{item.source_output}</JSONColumn>+ },+ },+ {+ title: 'Metadata',+ dataIndex: 'metadata',+ key: 'metadata',+ width: '25%',+ render: function renderMetadata(_, item) {+ return <JSONColumn>{item.metadata}</JSONColumn>+ },+ },+ {+ title: 'Created by',+ dataIndex: 'created_by',+ render: function renderCreatedBy(_, item) {+ const { created_by } = item+ return (+ <div className="flex flex-row items-center flex-nowrap">+ {created_by && <ProfilePicture user={created_by as UserBasicType} size="md" showName />}+ </div>+ )+ },+ },+ {+ title: 'Status',+ key: 'status',+ render: function renderStatus(_, item) {+ return (+ <LemonTag size="small" type={item.archived ? 'muted' : 'success'}>+ {item.archived ? 'Archived' : 'Active'}+ </LemonTag>+ )+ },+ },+ createdAtColumn<DatasetItem>() as LemonTableColumn<DatasetItem, keyof DatasetItem | undefined>,+ updatedAtColumn<DatasetItem>() as LemonTableColumn<DatasetItem, keyof DatasetItem | undefined>,+ {+ width: 0,+ render: function renderMore(_, item) {+ return (+ <More+ overlay={+ <>+ <LemonButton+ to={urls.aiObservabilityDataset(dataset.id, { ...searchParams, item: item.id })}+ data-attr={`dataset-item-${item.id}-dropdown-edit`}+ fullWidth+ >+ {canEditDataset && !item.archived ? 'Edit' : 'View'}+ </LemonButton>++ {canManageDataset && !dataset.archived && !isHistoricalRevision && (+ <LemonButton+ status={item.archived ? undefined : 'danger'}+ onClick={() =>+ item.archived+ ? restoreDatasetItem(item.id, item.version)+ : archiveDatasetItem(item.id, item.version)+ }+ loading={archivingDatasetItemId === item.id}+ disabledReason={+ archivingDatasetItemId && archivingDatasetItemId !== item.id+ ? 'Another dataset item is being archived'+ : undefined+ }+ data-attr={`dataset-item-${item.id}-dropdown-${+ item.archived ? 'unarchive' : 'archive'+ }`}+ fullWidth+ >+ {item.archived ? 'Unarchive' : 'Archive'}+ </LemonButton>+ )}+ </>+ }+ />+ )+ },+ },+ ]++ return (+ <div className="flex flex-col gap-3">+ <div className="flex justify-between items-center gap-2 flex-wrap">+ <div className="flex items-center gap-2 flex-wrap">+ <LemonSegmentedButton+ value={filters.archived ? 'archived' : 'active'}+ onChange={(value) => setFilters({ archived: value === 'archived', page: 1 }, false)}+ options={[+ { value: 'active', label: 'Active' },+ { value: 'archived', label: 'Archived' },+ ]}+ size="small"+ data-attr="dataset-items-status-filter"+ />+ <LemonSelect+ value={filters.revision === null ? 'latest' : String(filters.revision)}+ onChange={(value) =>+ setFilters({ revision: value === 'latest' ? null : Number(value), page: 1 }, false)+ }+ options={revisionOptions}+ loading={datasetRevisionsLoading}+ size="small"+ data-attr="dataset-revision-select"+ />+ </div>+ <div className="flex items-center gap-2 flex-wrap">+ <RefreshButton+ onClick={refreshDataset}+ isRefreshing={datasetItemsLoading || datasetLoading || datasetRevisionsLoading}+ />+ <LemonButton+ type="secondary"+ onClick={() => exportDataset(filters.revision ?? undefined)}+ loading={datasetExportLoading}+ disabledReason={datasetExportDisabledReason}+ data-attr="export-dataset"+ icon={<IconDownload />}+ size="small"+ >+ Export+ </LemonButton>+ {canEditDataset && (+ <LemonButton+ type="primary"+ onClick={() => triggerDatasetItemModal(true)}+ data-attr="add-dataset-item"+ icon={<IconPlusSmall />}+ size="small"+ >+ Add item+ </LemonButton>+ )}+ {dataset.archived && canChangeDatasetStatus && (+ <LemonButton+ type="primary"+ onClick={restoreDataset}+ loading={isArchivingDataset}+ data-attr="unarchive-dataset"+ size="small"+ >+ Unarchive+ </LemonButton>+ )}+ </div>+ </div>++ {datasetItemsLoadError && (+ <LemonBanner type="error" action={{ children: 'Try again', onClick: () => loadDatasetItems(false) }}>+ {datasetItemsLoadError.detail || "Couldn't load dataset items. Try again."}+ </LemonBanner>+ )}++ {datasetRevisionsLoadError && (+ <LemonBanner type="error" action={{ children: 'Try again', onClick: loadDatasetRevisions }}>+ {datasetRevisionsLoadError.detail || "Couldn't load dataset revisions. Try again."}+ </LemonBanner>+ )}++ <LemonTable+ size="small"+ loading={datasetItemsLoading}+ columns={columns}+ dataSource={datasetItems?.results || []}+ rowKey="id"+ pagination={pagination}+ nouns={['dataset item', 'dataset items']}+ emptyState="No items in this dataset yet."+ />+ </div>+ )+}diff --git a/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.test.ts b/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.test.tsindex ea27bb97eaa6..8bd6d7e26529 100644--- a/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.test.ts+++ b/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.test.ts@@ -15,7 +15,12 @@ import type {DatasetItemReadApi as DatasetItem,DatasetReadApi as Dataset,} from '../generated/api.schemas'-import { DatasetFormValues, DatasetLogicProps, aiObservabilityDatasetLogic } from './aiObservabilityDatasetLogic'+import {+ DatasetFormValues,+ DatasetLogicProps,+ DatasetTab,+ aiObservabilityDatasetLogic,+} from './aiObservabilityDatasetLogic'import { aiObservabilityDatasetsLogic } from './aiObservabilityDatasetsLogic'import { datasetsApi } from './datasetsApi'import { EMPTY_JSON } from './utils'@@ -172,7 +177,7 @@ describe('aiObservabilityDatasetLogic', () => {})})- it('can create a new dataset with non-empty metadata', async () => {+ it('creates a dataset inside Offline evals and preserves the return filters', async () => {const formValues: DatasetFormValues = {name: 'New Dataset',description: 'New description',@@ -181,6 +186,7 @@ describe('aiObservabilityDatasetLogic', () => {mockDatasetsApi.createDataset.mockResolvedValue(mockDataset)const routerReplaceSpy = jest.spyOn(router.actions, 'replace')+ router.actions.push(urls.aiObservabilityDataset('new'), { search: 'quality', datasets_page: '2' })await expectLogic(logic, () => {logic.actions.setDatasetFormValues(formValues)@@ -193,7 +199,12 @@ describe('aiObservabilityDatasetLogic', () => {metadata: { test: 'value' },})expect(lemonToast.success).toHaveBeenCalledWith('Dataset created successfully')- expect(routerReplaceSpy).toHaveBeenCalledWith(urls.aiObservabilityDataset(mockDataset.id))+ expect(routerReplaceSpy).toHaveBeenCalledWith(+ `/ai-evals/evaluations/offline/datasets/${mockDataset.id}?search=quality&datasets_page=2`+ )+ expect(logic.values.breadcrumbs[0].path).toBe(+ '/ai-evals/evaluations/offline/experiments?tab=datasets&search=quality&datasets_page=2'+ )expect(logic.values.dataset).toEqual(mockDataset)expect(logic.values.isEditingDataset).toBe(false)})@@ -1193,15 +1204,36 @@ describe('aiObservabilityDatasetLogic', () => {})})- it('sets active tab from URL parameters via urlToAction', async () => {+ it('opens and closes dataset details through URL state without reloading items or losing filters', async () => {const datasetUrl = urls.aiObservabilityDataset(mockDataset.id)await expectLogic(logic, () => {- router.actions.push(datasetUrl, { tab: 'metadata', page: '1' })+ router.actions.push(datasetUrl, { tab: 'metadata', search: 'quality' }, { section: 'items' })}).toFinishAllListeners()- // Verify tab is set from URLexpect(logic.values.activeTab).toBe('metadata')+ mockDatasetsApi.listItems.mockClear()++ await expectLogic(logic, () => {+ logic.actions.setActiveTab(DatasetTab.Items)+ }).toFinishAllListeners()++ expect(router.values.searchParams).toEqual({ search: 'quality' })+ expect(router.values.hashParams).toEqual({ section: 'items' })+ expect(logic.values.activeTab).toBe('items')++ await expectLogic(logic, () => {+ logic.actions.setActiveTab(DatasetTab.Metadata)+ }).toFinishAllListeners()++ expect(router.values.searchParams).toEqual({ search: 'quality', tab: 'metadata' })++ await expectLogic(logic, () => {+ router.actions.push(datasetUrl, { search: 'quality' })+ }).toFinishAllListeners()++ expect(logic.values.activeTab).toBe('items')+ expect(mockDatasetsApi.listItems).not.toHaveBeenCalled()})it('closes modal and clears state when closeModalAndRefetchDatasetItems is called', async () => {diff --git a/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.ts b/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.tsindex 71faa18390a9..40df3b272edd 100644--- a/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.ts+++ b/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.ts@@ -41,7 +41,7 @@ import type {PaginatedDatasetRevisionReadListApi,} from '../generated/api.schemas'import { truncateValue } from '../utils'-import { aiObservabilityDatasetsLogic, getDatasetListUrl } from './aiObservabilityDatasetsLogic'+import { aiObservabilityDatasetsLogic, getDatasetDetailUrl, getDatasetListUrl } from './aiObservabilityDatasetsLogic'import { datasetsApi } from './datasetsApi'import { EMPTY_JSON, isStringJsonObject, parseJsonObject, prettifyJson } from './utils'@@ -866,7 +866,7 @@ export const aiObservabilityDatasetLogic = kea<aiObservabilityDatasetLogicType>(metadata: parseJsonObject(formValues.metadata),})lemonToast.success('Dataset created successfully')- router.actions.replace(urls.aiObservabilityDataset(savedDataset.id))+ router.actions.replace(getDatasetDetailUrl(savedDataset.id, router.values.searchParams))void actions.addProductIntent({product_type: ProductKey.LLM_DATASETS,@@ -1247,12 +1247,6 @@ export const aiObservabilityDatasetLogic = kea<aiObservabilityDatasetLogicType>(}},- setActiveTab: ({ tab }) => {- if (tab === DatasetTab.Items && props.datasetId !== 'new') {- actions.loadDatasetItems(true)- }- },-closeModalAndRefetchDatasetItems: async ({ refetchDatasetItems }) => {if (refetchDatasetItems) {await asyncActions.refreshAfterDatasetItemMutation()@@ -1262,12 +1256,9 @@ export const aiObservabilityDatasetLogic = kea<aiObservabilityDatasetLogicType>(urlToAction(({ actions, values }) => ({[urls.aiObservabilityDataset(':id')]: (_, searchParams) => {- if (- searchParams.tab &&- Object.values(DatasetTab).includes(searchParams.tab as DatasetTab) &&- searchParams.tab !== values.activeTab- ) {- actions.setActiveTab(searchParams.tab as DatasetTab)+ const activeTab = searchParams.tab === DatasetTab.Metadata ? DatasetTab.Metadata : DatasetTab.Items+ if (activeTab !== values.activeTab) {+ actions.setActiveTab(activeTab)}const newFilters = cleanFilters(searchParams)@@ -1291,6 +1282,14 @@ export const aiObservabilityDatasetLogic = kea<aiObservabilityDatasetLogicType>(})),trackedActionToUrl(({ values }) => ({+ setActiveTab: () => [+ router.values.location.pathname,+ {+ ...router.values.searchParams,+ tab: values.activeTab === DatasetTab.Metadata ? DatasetTab.Metadata : undefined,+ },+ router.values.hashParams,+ ],setFilters: () => {const nextSearchParams = {...router.values.searchParams,diff --git a/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.test.ts b/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.test.tsindex e320ce347539..b948a52f776a 100644--- a/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.test.ts+++ b/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.test.ts@@ -1,12 +1,15 @@import { router } from 'kea-router'import { expectLogic } from 'kea-test-utils'+import { FEATURE_FLAGS } from 'lib/constants'import { lemonToast } from 'lib/lemon-ui/LemonToast/LemonToast'+import { featureFlagLogic } from 'lib/logic/featureFlagLogic'import { urls } from 'scenes/urls'import { initKeaTests } from '~/test/init'import type { DatasetReadApi as Dataset } from '../generated/api.schemas'+import { offlineEvaluationsLogic } from '../offline-evaluations/offlineEvaluationsLogic'import {DATASETS_PER_PAGE,aiObservabilityDatasetsLogic,@@ -96,24 +99,41 @@ describe('aiObservabilityDatasetsLogic', () => {})expect(getDatasetDetailUrl(mockDataset1.id, searchParams)).toContain('datasets_page=2')expect(getDatasetListUrl(searchParams)).toContain('datasets_page=2')+ expect(getDatasetListUrl(searchParams)).toContain('tab=datasets')expect(getDatasetDetailUrl(mockDataset1.id, searchParams)).not.toContain('item=')})- it('shares tag filters in the URL and restores them when returning to datasets', async () => {+ it('shares dataset tag filters in the URL without replacing them with experiment filters', async () => {+ featureFlagLogic.mount()+ featureFlagLogic.actions.setFeatureFlags([], {+ [FEATURE_FLAGS.LLM_ANALYTICS_DATASETS]: true,+ [FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS]: true,+ })+ const tabs = offlineEvaluationsLogic()+ tabs.mount()router.actions.push(urls.aiObservabilityDatasets())const logic = aiObservabilityDatasetsLogic()logic.mount()await expectLogic(logic, () =>logic.actions.setFilters({ tags: ['quality', 'a,b'], tag_match: 'all' }, true, false)).toFinishAllListeners()- expect(router.values.searchParams).toMatchObject({ tags: ['quality', 'a,b'], tag_match: 'all' })+ expect(router.values.searchParams).toMatchObject({+ tab: 'datasets',+ tags: ['quality', 'a,b'],+ tag_match: 'all',+ })expect(mockDatasetsApi.listDatasets).toHaveBeenLastCalledWith(expect.objectContaining({ tags: '["quality","a,b"]', tag_match: 'all' }))- router.actions.push(urls.aiObservabilityScorers())- router.actions.push(urls.aiObservabilityDatasets())+ router.actions.push(urls.aiObservabilityOfflineEvaluations(), {+ tab: 'experiments',+ search: 'experiment-name',+ tags: ['experiment-only'],+ tag_match: 'any',+ })+ tabs.actions.setActiveTab('datasets')await expectLogic(logic).toFinishAllListeners()- expect(logic.values.filters.tags).toEqual(['quality', 'a,b'])+ expect(logic.values.filters).toMatchObject({ search: '', tags: ['quality', 'a,b'], tag_match: 'all' })expect(router.values.searchParams.tags).toEqual(['quality', 'a,b'])router.actions.push(urls.aiObservabilityDatasets(), { tags: [], tag_match: 'any' })await expectLogic(logic).toFinishAllListeners()diff --git a/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.ts b/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.tsindex 4ccd90becfa0..ef18baf9db37 100644--- a/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.ts+++ b/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.ts@@ -337,6 +337,7 @@ export const aiObservabilityDatasetsLogic = kea<aiObservabilityDatasetsLogicTypeconst { page, ...pagedSearchOrderParams } = cleanPagedSearchOrderParams(values.filters)const nextValues = {...pagedSearchOrderParams,+ tab: 'datasets',datasets_page: page,dataset_status: values.filters.archived ? 'archived' : undefined,tags: values.filters.tags,@@ -344,7 +345,12 @@ export const aiObservabilityDatasetsLogic = kea<aiObservabilityDatasetsLogicType}const urlValues = cleanFilters(router.values.searchParams)if (!objectsEqual(values.filters, urlValues)) {- return [urls.aiObservabilityDatasets(), nextValues, {}, { replace: true }]+ return [+ urls.aiObservabilityOfflineEvaluations(),+ nextValues,+ router.values.hashParams,+ { replace: true },+ ]}}return {@@ -353,7 +359,10 @@ export const aiObservabilityDatasetsLogic = kea<aiObservabilityDatasetsLogicType}),urlToAction(({ actions, values }) => ({- [urls.aiObservabilityDatasets()]: (_, searchParams, __, { method }) => {+ [urls.aiObservabilityOfflineEvaluations()]: (_, searchParams, __, { method }) => {+ if (searchParams.tab !== 'datasets') {+ return+ }const newFilters = cleanFilters({ ...searchParams, ...restoreTagFilters('datasets', searchParams) })if (!objectsEqual(values.filters, newFilters)) {actions.setFilters(newFilters, false)diff --git a/products/ai_observability/frontend/datasets/saveToDatasetButtonLogic.test.ts b/products/ai_observability/frontend/datasets/saveToDatasetButtonLogic.test.tsindex 83288d875389..cd712c5a2b90 100644--- a/products/ai_observability/frontend/datasets/saveToDatasetButtonLogic.test.ts+++ b/products/ai_observability/frontend/datasets/saveToDatasetButtonLogic.test.ts@@ -448,7 +448,9 @@ describe('saveToDatasetButtonLogic', () => {viewAction()- expect(router.actions.push).toHaveBeenCalledWith('/ai-evals/datasets/test-dataset-1')+ expect(router.actions.push).toHaveBeenCalledWith(+ '/ai-evals/evaluations/offline/datasets/test-dataset-1'+ )})})diff --git a/products/ai_observability/frontend/evaluations/EvaluationsTabs.tsx b/products/ai_observability/frontend/evaluations/EvaluationsTabs.tsxindex d6a9c0cfacf3..35bf3dfa7414 100644--- a/products/ai_observability/frontend/evaluations/EvaluationsTabs.tsx+++ b/products/ai_observability/frontend/evaluations/EvaluationsTabs.tsx@@ -37,10 +37,16 @@ export function EvaluationsTabs({...accessTab(Scene.AIObservabilityEvaluation, urls.aiObservabilityEvaluations()),'data-attr': 'evaluations-tab',},- !!featureFlags[FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS] && {+ (!!featureFlags[FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS] ||+ !!featureFlags[FEATURE_FLAGS.LLM_ANALYTICS_DATASETS]) && {key: 'offline-evals',label: 'Offline evals',- ...accessTab(Scene.AIObservabilityOfflineExperiments, urls.aiObservabilityOfflineEvaluations()),+ ...accessTab(+ Scene.AIObservabilityOfflineExperiments,+ featureFlags[FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS]+ ? urls.aiObservabilityOfflineEvaluations()+ : urls.aiObservabilityDatasets()+ ),'data-attr': 'offline-evals-tab',},{diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsScene.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsScene.tsxnew file mode 100644index 000000000000..64bb3d55ddde--- /dev/null+++ b/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsScene.tsx@@ -0,0 +1,19 @@+import type { ReactNode } from 'react'++import { SceneContent } from '~/layout/scenes/components/SceneContent'+import { SceneTitleSection } from '~/layout/scenes/components/SceneTitleSection'++import { EvaluationsTabs } from '../evaluations/EvaluationsTabs'++export function OfflineEvaluationsScene({ children }: { children: ReactNode }): JSX.Element {+ return (+ <SceneContent>+ <SceneTitleSection+ name="Evaluations"+ description="Manage datasets, follow score trends, and inspect offline experiment results."+ resourceType={{ type: 'llm_evaluations' }}+ />+ <EvaluationsTabs activeTab="offline-evals">{children}</EvaluationsTabs>+ </SceneContent>+ )+}diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsTabs.stories.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsTabs.stories.tsxnew file mode 100644index 000000000000..b4cd0ef68298--- /dev/null+++ b/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsTabs.stories.tsx@@ -0,0 +1,49 @@+import type { Meta, StoryObj } from '@storybook/react'+import { within } from '@testing-library/dom'++import { FEATURE_FLAGS } from 'lib/constants'+import { urls } from 'scenes/urls'++import { OfflineEvaluationsTabs } from './OfflineEvaluationsTabs'+import { OfflineExperimentsScene } from './OfflineExperimentsScene'++const meta: Meta<typeof OfflineEvaluationsTabs> = {+ title: 'Scenes-App/AI observability/Offline experiments/Navigation',+ component: OfflineEvaluationsTabs,+ args: { activeTab: 'datasets' },+ parameters: {+ pageUrl: urls.aiObservabilityDataset('example-dataset'),+ featureFlags: [FEATURE_FLAGS.LLM_ANALYTICS_DATASETS, FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS],+ },+}+export default meta++type Story = StoryObj<typeof OfflineEvaluationsTabs>++export const DatasetDetail: Story = {}+export const DatasetsOnly: Story = {+ parameters: {+ featureFlags: {+ [FEATURE_FLAGS.LLM_ANALYTICS_DATASETS]: true,+ [FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS]: false,+ },+ },+}+export const DatasetsUnavailable: Story = {+ render: () => <OfflineExperimentsScene />,+ parameters: {+ pageUrl: urls.aiObservabilityDatasets(),+ featureFlags: {+ [FEATURE_FLAGS.LLM_ANALYTICS_DATASETS]: false,+ [FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS]: true,+ },+ },+ play: async ({ canvasElement }) => {+ const canvas = within(canvasElement)+ await canvas.findByText('Datasets are coming soon.')+ const datasetsTab = canvas.getByRole('button', { name: 'Datasets' })+ if (!datasetsTab.hasAttribute('disabled') && datasetsTab.getAttribute('aria-disabled') !== 'true') {+ throw new Error('Datasets must stay disabled when the feature flag is off')+ }+ },+}diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsTabs.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsTabs.tsxnew file mode 100644index 000000000000..a1a7adadb933--- /dev/null+++ b/products/ai_observability/frontend/offline-evaluations/OfflineEvaluationsTabs.tsx@@ -0,0 +1,52 @@+import { useActions, useValues } from 'kea'++import { LemonSegmentedButton, LemonTag } from '@posthog/lemon-ui'++import { getProductAccessDisabledReason } from 'lib/utils/accessControlUtils'+import { Scene } from 'scenes/sceneTypes'++import { OfflineEvaluationsTab, offlineEvaluationsLogic } from './offlineEvaluationsLogic'++export function OfflineEvaluationsTabs({ activeTab }: { activeTab: OfflineEvaluationsTab }): JSX.Element {+ const { datasetsEnabled, offlineEvaluationsEnabled } = useValues(offlineEvaluationsLogic)+ const { setActiveTab } = useActions(offlineEvaluationsLogic)++ return (+ <LemonSegmentedButton+ size="small"+ value={activeTab}+ onChange={setActiveTab}+ options={[+ {+ value: 'experiments',+ label: (+ <span className="flex items-center gap-1">+ <span>Experiments</span>+ {!offlineEvaluationsEnabled && (+ <LemonTag type="muted" size="small">+ Coming soon+ </LemonTag>+ )}+ </span>+ ),+ disabledReason: !offlineEvaluationsEnabled+ ? 'Offline experiments are coming soon.'+ : getProductAccessDisabledReason({+ sceneKey: Scene.AIObservabilityOfflineExperiment,+ displayLabel: 'experiments',+ }),+ },+ {+ value: 'datasets',+ label: 'Datasets',+ disabledReason: !datasetsEnabled+ ? 'Datasets are coming soon.'+ : getProductAccessDisabledReason({+ sceneKey: Scene.AIObservabilityDatasets,+ displayLabel: 'datasets',+ }),+ },+ ]}+ />+ )+}diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentScene.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentScene.tsxindex a0ddd147c5d6..796643fa8ee7 100644--- a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentScene.tsx+++ b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentScene.tsx@@ -3,12 +3,11 @@ import { useValues } from 'kea'import { SceneExport } from 'scenes/sceneTypes'import { teamLogic } from 'scenes/teamLogic'-import { SceneContent } from '~/layout/scenes/components/SceneContent'-import { SceneTitleSection } from '~/layout/scenes/components/SceneTitleSection'import { ProductKey } from '~/queries/schema/schema-general'-import { EvaluationsTabs } from '../evaluations/EvaluationsTabs'import { OfflineEvaluationsGate } from './OfflineEvaluationsGate'+import { OfflineEvaluationsScene } from './OfflineEvaluationsScene'+import { OfflineEvaluationsTabs } from './OfflineEvaluationsTabs'import { OfflineExperimentContent } from './OfflineExperimentContent'export const scene: SceneExport<{ experimentId: string }> = {@@ -20,19 +19,19 @@ export const scene: SceneExport<{ experimentId: string }> = {export function OfflineExperimentScene({ experimentId }: { experimentId: string }): JSX.Element {const { currentTeamId } = useValues(teamLogic)return (- <SceneContent>- <SceneTitleSection name="Offline experiment" resourceType={{ type: 'evaluation' }} />- <EvaluationsTabs activeTab="offline-evals">- <OfflineEvaluationsGate isAvailable={!!currentTeamId}>- {currentTeamId && (- <OfflineExperimentContent- key={`${currentTeamId}:${experimentId}`}- teamId={currentTeamId}- experimentId={experimentId}- />- )}- </OfflineEvaluationsGate>- </EvaluationsTabs>- </SceneContent>+ <OfflineEvaluationsScene>+ <div>+ <OfflineEvaluationsTabs activeTab="experiments" />+ </div>+ <OfflineEvaluationsGate isAvailable={!!currentTeamId}>+ {currentTeamId && (+ <OfflineExperimentContent+ key={`${currentTeamId}:${experimentId}`}+ teamId={currentTeamId}+ experimentId={experimentId}+ />+ )}+ </OfflineEvaluationsGate>+ </OfflineEvaluationsScene>)}diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.stories.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.stories.tsxindex 612f260e7db0..1b310b1372a1 100644--- a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.stories.tsx+++ b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.stories.tsx@@ -1,5 +1,5 @@import type { Meta, StoryObj } from '@storybook/react'-import { fireEvent, waitFor } from '@testing-library/dom'+import { fireEvent, waitFor, within } from '@testing-library/dom'import { playHoverAtFraction } from '@posthog/quill-charts/story-helpers'@@ -58,7 +58,35 @@ const meta: Meta<typeof OfflineExperimentsOverview> = {export default metatype Story = StoryObj<typeof OfflineExperimentsOverview>-export const RecentExperiments: Story = {}+export const RecentExperiments: Story = {+ play: async ({ canvasElement }) => {+ const canvas = within(canvasElement)+ await canvas.findByRole('heading', { name: 'Score trends' })+ const expandButton = canvas.queryByRole('button', { name: 'Expand score trends' })+ if (expandButton) {+ fireEvent.click(expandButton)+ }+ const chart = await waitFor(() => {+ const chart = canvasElement.querySelector('[data-attr="offline-score-trend"]')+ if (!chart) {+ throw new Error('Score charts have not loaded')+ }+ return chart+ })+ fireEvent.click(canvas.getByRole('button', { name: 'Collapse score trends' }))+ await waitFor(() => {+ if (!chart.isConnected || !chart.closest('[hidden]')) {+ throw new Error('Collapsing score trends must hide the charts without unmounting them')+ }+ })+ fireEvent.click(canvas.getByRole('button', { name: 'Expand score trends' }))+ await waitFor(() => {+ if (!chart.isConnected || chart.closest('[hidden]')) {+ throw new Error('Expanding score trends must show the already mounted charts')+ }+ })+ },+}export const PartialHistory: Story = {parameters: { pageUrl: `${urls.aiObservabilityOfflineEvaluations()}?scores=${overviewScorers[0].id}` },decorators: [diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.tsxindex cfb2bbde679f..5662550e4434 100644--- a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.tsx+++ b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.tsx@@ -1,6 +1,7 @@import { useActions, useValues } from 'kea'+import type { ReactNode } from 'react'-import { IconRefresh, IconSearch } from '@posthog/icons'+import { IconChevronRight, IconRefresh, IconSearch } from '@posthog/icons'import {LemonBanner,LemonButton,@@ -32,7 +33,10 @@ import { offlineRunSourceLabel } from './offlineOverviewState'import { OfflineOverviewTrend } from './OfflineOverviewTrend'import { OfflineScoreChooser } from './OfflineScoreChooser'-export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps): JSX.Element {+export function OfflineExperimentsOverview({+ navigation,+ ...props+}: OfflineExperimentsLogicProps & { navigation?: ReactNode }): JSX.Element {const logic = offlineExperimentsLogic(props)const {experiments,@@ -48,8 +52,11 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):dateRange,cursorStack,refreshKey,+ scoreTrendsExpanded,+ scoreTrendsInitialized,} = useValues(logic)- const { setFilters, openChooser, nextPage, previousPage, refresh, loadOfflineSuggestedScorers } = useActions(logic)+ const { setFilters, openChooser, nextPage, previousPage, refresh, loadOfflineSuggestedScorers, toggleScoreTrends } =+ useActions(logic)const columns: LemonTableColumns<OfflineExperimentReadApi> = [{title: 'Experiment',@@ -60,7 +67,7 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):<Tooltip title={`${experiment.name} · ${experiment.id}`}><Linkto={urls.aiObservabilityOfflineEvaluationExperiment(experiment.id)}- className="block font-semibold truncate"+ className="block font-semibold truncate @max-[50rem]:max-w-40"data-attr="offline-experiment-open">{experiment.name}@@ -74,6 +81,7 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):width: '19%',render: (_, experiment) => (<ResourceTags+ compactentity="experiments"resourceId={experiment.id}tags={experiment.tags}@@ -86,12 +94,13 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):title: 'Author',key: 'created_by',width: '18%',- className: '[&_.profile-package]:max-w-full [&_.profile-name]:truncate',+ className:+ '[&_.profile-package]:max-w-full @max-[50rem]:[&_.profile-package]:max-w-24 [&_.profile-name]:truncate',render: (_, experiment) =>experiment.created_by ? (<ProfilePicture user={experiment.created_by as UserBasicType} size="md" showName />) : (- <span className="text-muted">No author</span>+ <span className="text-muted whitespace-nowrap">No author</span>),},{@@ -155,17 +164,28 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):]if (hasExperiments === null && !experimentsError) {- return <LemonSkeleton className="h-80" />+ return (+ <div className="space-y-4">+ {navigation}+ <LemonSkeleton className="h-80" />+ </div>+ )}if (hasExperiments === false && !experimentsLoading && !experimentsError) {- return <OfflineExperimentsEmptyState />+ return (+ <div className="space-y-4">+ {navigation}+ <OfflineExperimentsEmptyState />+ </div>+ )}return (<div className="@container space-y-6 min-w-0"><div aria-label="Experiment filters" className="flex flex-wrap items-center gap-2">+ {navigation}<DateFilter- size="medium"+ size="small"dateFrom={filters.date_from || '-30d'}dateTo={filters.date_to || null}onChange={(dateFrom, dateTo) =>@@ -175,6 +195,7 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):showCustomRelativeRange/><LemonSelect+ size="small"value={filters.run_source || ''}onChange={(run_source) => setFilters({ run_source })}data-attr="offline-experiments-source-filter"@@ -187,6 +208,7 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):]}/><LemonSelect+ size="small"value={filters.statuses || ''}onChange={(statuses) => setFilters({ statuses })}data-attr="offline-experiments-status-filter"@@ -197,11 +219,11 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):{ value: 'failed', label: 'Failed' },]}/>- <ResourceTagFilter filters={cleanTagFilters(filters)} onChange={setFilters} />+ <ResourceTagFilter size="small" filters={cleanTagFilters(filters)} onChange={setFilters} /><div data-attr="offline-experiments-author-filter"><MemberSelectvalue={filters.created_by ?? null}- size="medium"+ size="small"defaultLabel="Any author"extraOptions={[{@@ -213,7 +235,7 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):>{filters.created_by_is_null? () => (- <LemonButton type="secondary" size="medium">+ <LemonButton type="secondary" size="small">No author</LemonButton>)@@ -222,6 +244,7 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):</div><LemonButtontype="secondary"+ size="small"icon={<IconRefresh />}onClick={refresh}loading={experimentsLoading}@@ -234,52 +257,81 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):<section aria-label="Score trends" className="space-y-3"><div className="flex flex-wrap justify-between items-center gap-2"><div>- <h2 className="mb-0">Score trends</h2>- <p className="text-muted text-xs mb-0">- Scores across the selected experiments. Uploading and failed runs may have partial results.- </p>+ <div className="flex items-center gap-1">+ <h2 className="mb-0">Score trends</h2>+ <LemonButton+ icon={<IconChevronRight className={scoreTrendsExpanded ? 'rotate-90' : undefined} />}+ size="xsmall"+ className="!bg-transparent"+ onClick={toggleScoreTrends}+ aria-label={scoreTrendsExpanded ? 'Collapse score trends' : 'Expand score trends'}+ aria-expanded={scoreTrendsExpanded}+ aria-controls="offline-score-trends"+ />+ </div>+ {scoreTrendsExpanded && (+ <p className="text-muted text-xs mb-0">+ Scores across the selected experiments. Uploading and failed runs may have partial+ results.+ </p>+ )}</div>- <LemonButton type="secondary" size="small" onClick={openChooser} data-attr="offline-choose-scores">- Choose scores- </LemonButton>+ {scoreTrendsExpanded && (+ <LemonButton+ type="secondary"+ size="small"+ onClick={openChooser}+ data-attr="offline-choose-scores"+ >+ Choose scores+ </LemonButton>+ )}</div>- {!dateRange ? (- <LemonBanner type="error">Choose a valid date range.</LemonBanner>- ) : scorerIds === null || (!scorerIds.length && suggestedScorersLoading) ? (- <LemonSkeleton className="h-40" />- ) : scorerIds.length > 0 ? (- <div className="grid grid-cols-1 @min-[56rem]:grid-cols-3 gap-3">- {scorerIds.map((scorerId, index) => (- <OfflineOverviewTrend- key={scorerId}- overviewLogic={logic}- overviewId={logic.key}- xDomain={trendXDomain}- colorOffset={index}- teamId={props.teamId}- timezone={props.timezone}- scorerId={scorerId}- dateFrom={dateRange.dateFrom}- dateTo={dateRange.dateTo}- refreshKey={refreshKey}- filters={trendFilters}- />- ))}+ {scoreTrendsInitialized && (+ <div id="offline-score-trends" hidden={!scoreTrendsExpanded}>+ {!dateRange ? (+ <LemonBanner type="error">Choose a valid date range.</LemonBanner>+ ) : scorerIds === null || (!scorerIds.length && suggestedScorersLoading) ? (+ <LemonSkeleton className="h-40" />+ ) : scorerIds.length > 0 ? (+ <div className="grid grid-cols-1 @min-[56rem]:grid-cols-3 gap-3">+ {scorerIds.map((scorerId, index) => (+ <OfflineOverviewTrend+ key={scorerId}+ overviewLogic={logic}+ overviewId={logic.key}+ xDomain={trendXDomain}+ colorOffset={index}+ teamId={props.teamId}+ timezone={props.timezone}+ scorerId={scorerId}+ dateFrom={dateRange.dateFrom}+ dateTo={dateRange.dateTo}+ refreshKey={refreshKey}+ filters={trendFilters}+ />+ ))}+ </div>+ ) : suggestedScorersError ? (+ <LemonBanner+ type="error"+ action={{ children: 'Try again', onClick: loadOfflineSuggestedScorers }}+ >+ Could not load suggested scores. Try again or choose scores yourself.+ </LemonBanner>+ ) : (+ <LemonBanner type="info" action={{ children: 'Choose scores', onClick: openChooser }}>+ Choose recurring scores to follow their results over time.+ </LemonBanner>+ )}</div>- ) : suggestedScorersError ? (- <LemonBanner type="error" action={{ children: 'Try again', onClick: loadOfflineSuggestedScorers }}>- Could not load suggested scores. Try again or choose scores yourself.- </LemonBanner>- ) : (- <LemonBanner type="info" action={{ children: 'Choose scores', onClick: openChooser }}>- Choose recurring scores to follow their results over time.- </LemonBanner>)}</section><section aria-label="Recent experiments" className="space-y-3"><div className="flex flex-wrap justify-between items-center gap-2"><h2 className="mb-0">Recent experiments</h2><LemonInput+ size="small"type="search"prefix={<IconSearch />}value={filters.search || ''}diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsScene.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsScene.tsxindex c819344d3990..5cce95001c04 100644--- a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsScene.tsx+++ b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsScene.tsx@@ -1,15 +1,18 @@import { useValues } from 'kea'-import { SceneExport } from 'scenes/sceneTypes'+import { LemonBanner } from '@posthog/lemon-ui'++import { getProductAccessDisabledReason } from 'lib/utils/accessControlUtils'+import { Scene, SceneExport } from 'scenes/sceneTypes'import { teamLogic } from 'scenes/teamLogic'import { userLogic } from 'scenes/userLogic'-import { SceneContent } from '~/layout/scenes/components/SceneContent'-import { SceneTitleSection } from '~/layout/scenes/components/SceneTitleSection'import { ProductKey } from '~/queries/schema/schema-general'-import { EvaluationsTabs } from '../evaluations/EvaluationsTabs'-import { OfflineEvaluationsGate } from './OfflineEvaluationsGate'+import { AIObservabilityDatasetsScene } from '../datasets/AIObservabilityDatasetsScene'+import { offlineEvaluationsLogic } from './offlineEvaluationsLogic'+import { OfflineEvaluationsScene } from './OfflineEvaluationsScene'+import { OfflineEvaluationsTabs } from './OfflineEvaluationsTabs'import { OfflineExperimentsOverview } from './OfflineExperimentsOverview'export const scene: SceneExport = { component: OfflineExperimentsScene, productKey: ProductKey.AI_OBSERVABILITY }@@ -17,28 +20,46 @@ export const scene: SceneExport = { component: OfflineExperimentsScene, productKexport function OfflineExperimentsScene(): JSX.Element {const { currentTeam } = useValues(teamLogic)const { user } = useValues(userLogic)+ const { activeTab, datasetsEnabled, offlineEvaluationsEnabled } = useValues(offlineEvaluationsLogic)+ const experimentsDisabledReason = !offlineEvaluationsEnabled+ ? 'Offline experiments are coming soon.'+ : getProductAccessDisabledReason({+ sceneKey: Scene.AIObservabilityOfflineExperiment,+ displayLabel: 'experiments',+ })+ const datasetsDisabledReason = !datasetsEnabled+ ? 'Datasets are coming soon.'+ : getProductAccessDisabledReason({+ sceneKey: Scene.AIObservabilityDatasets,+ displayLabel: 'datasets',+ })+ const activeTabDisabledReason =+ activeTab === 'datasets'+ ? datasetsDisabledReason+ : experimentsDisabledReason ||+ (!currentTeam || !user ? 'Offline evaluations are not available in this project.' : undefined)+ const navigation = <OfflineEvaluationsTabs activeTab={activeTab} />return (- <SceneContent>- <SceneTitleSection- name="Evaluations"- description="Follow score trends and inspect offline experiment results."- resourceType={{ type: 'llm_evaluations' }}- />- <EvaluationsTabs activeTab="offline-evals">- <OfflineEvaluationsGate- isAvailable={!!currentTeam && !!user}- unavailableMessage="Offline evaluations are not available in this project."- >- {currentTeam && user && (- <OfflineExperimentsOverview- key={`${currentTeam.id}:${user.id}`}- teamId={currentTeam.id}- userId={user.id}- timezone={currentTeam.timezone}- />- )}- </OfflineEvaluationsGate>- </EvaluationsTabs>- </SceneContent>+ <OfflineEvaluationsScene>+ {activeTabDisabledReason ? (+ <>+ {navigation}+ <LemonBanner type="info">{activeTabDisabledReason}</LemonBanner>+ </>+ ) : activeTab === 'datasets' ? (+ <AIObservabilityDatasetsScene navigation={navigation} />+ ) : (+ currentTeam &&+ user && (+ <OfflineExperimentsOverview+ key={`${currentTeam.id}:${user.id}`}+ teamId={currentTeam.id}+ userId={user.id}+ timezone={currentTeam.timezone}+ navigation={navigation}+ />+ )+ )}+ </OfflineEvaluationsScene>)}diff --git a/products/ai_observability/frontend/offline-evaluations/offlineEvaluationsLogic.ts b/products/ai_observability/frontend/offline-evaluations/offlineEvaluationsLogic.tsnew file mode 100644index 000000000000..a3fa4df0f811--- /dev/null+++ b/products/ai_observability/frontend/offline-evaluations/offlineEvaluationsLogic.ts@@ -0,0 +1,128 @@+import { MakeLogicType, actions, afterMount, connect, kea, listeners, path, reducers, selectors } from 'kea'+import { actionToUrl, router, urlToAction } from 'kea-router'++import { FEATURE_FLAGS } from 'lib/constants'+import { featureFlagLogic, type FeatureFlagsSet } from 'lib/logic/featureFlagLogic'+import { removeProjectIdIfPresent } from 'lib/utils/kea-router'+import { urls } from 'scenes/urls'++import { getDatasetListUrl } from '../datasets/aiObservabilityDatasetsLogic'++export type OfflineEvaluationsTab = 'experiments' | 'datasets'++// Generated by kea-typegen. Update if you're an agent, ignore if you're human.+export interface offlineEvaluationsLogicValues {+ featureFlags: FeatureFlagsSet // featureFlagLogic+ receivedFeatureFlags: boolean // featureFlagLogic+ activeTab: OfflineEvaluationsTab+ datasetsEnabled: boolean+ datasetsOnly: boolean+ offlineEvaluationsEnabled: boolean+ selectedTab: OfflineEvaluationsTab+}++// Generated by kea-typegen. Update if you're an agent, ignore if you're human.+export interface offlineEvaluationsLogicActions {+ ensureAvailableTab: () => {+ value: true+ }+ setActiveTab: (tab: OfflineEvaluationsTab) => {+ tab: OfflineEvaluationsTab+ }+ syncTabFromUrl: (tab: OfflineEvaluationsTab) => {+ tab: OfflineEvaluationsTab+ }+}++// Generated by kea-typegen. Update if you're an agent, ignore if you're human.+export interface offlineEvaluationsLogicMeta {+ __keaTypeGenInternalSelectorTypes: {+ offlineEvaluationsEnabled: (featureFlags: FeatureFlagsSet) => boolean+ datasetsEnabled: (featureFlags: FeatureFlagsSet) => boolean+ datasetsOnly: (datasetsEnabled: boolean, offlineEvaluationsEnabled: boolean) => boolean+ activeTab: (selectedTab: OfflineEvaluationsTab, datasetsOnly: boolean) => OfflineEvaluationsTab+ }+}++export type offlineEvaluationsLogicType = MakeLogicType<+ offlineEvaluationsLogicValues,+ offlineEvaluationsLogicActions,+ Record<string, any>,+ offlineEvaluationsLogicMeta+>++export const offlineEvaluationsLogic = kea<offlineEvaluationsLogicType>([+ path(['products', 'ai_observability', 'offlineEvaluationsLogic']),+ connect({ values: [featureFlagLogic, ['featureFlags', 'receivedFeatureFlags']] }),+ actions({+ setActiveTab: (tab: OfflineEvaluationsTab) => ({ tab }),+ syncTabFromUrl: (tab: OfflineEvaluationsTab) => ({ tab }),+ ensureAvailableTab: true,+ }),+ reducers({+ selectedTab: [+ 'experiments' as OfflineEvaluationsTab,+ {+ setActiveTab: (_, { tab }) => tab,+ syncTabFromUrl: (_, { tab }) => tab,+ },+ ],+ }),+ selectors({+ offlineEvaluationsEnabled: [+ (s) => [s.featureFlags],+ (featureFlags: FeatureFlagsSet): boolean =>+ !!featureFlags[FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS],+ ],+ datasetsEnabled: [+ (s) => [s.featureFlags],+ (featureFlags: FeatureFlagsSet): boolean => !!featureFlags[FEATURE_FLAGS.LLM_ANALYTICS_DATASETS],+ ],+ datasetsOnly: [+ (s) => [s.datasetsEnabled, s.offlineEvaluationsEnabled],+ (datasetsEnabled: boolean, offlineEvaluationsEnabled: boolean): boolean =>+ datasetsEnabled && !offlineEvaluationsEnabled,+ ],+ activeTab: [+ (s) => [s.selectedTab, s.datasetsOnly],+ (selectedTab: OfflineEvaluationsTab, datasetsOnly: boolean): OfflineEvaluationsTab =>+ datasetsOnly ? 'datasets' : selectedTab,+ ],+ }),+ listeners(({ actions, values }) => ({+ ensureAvailableTab: () => {+ if (+ removeProjectIdIfPresent(router.values.location.pathname) !== urls.aiObservabilityOfflineEvaluations()+ ) {+ return+ }+ actions.syncTabFromUrl(router.values.searchParams.tab === 'datasets' ? 'datasets' : 'experiments')+ if (values.receivedFeatureFlags && values.datasetsOnly && router.values.searchParams.tab !== 'datasets') {+ router.actions.replace(+ router.values.location.pathname,+ { ...router.values.searchParams, tab: 'datasets' },+ router.values.hashParams+ )+ }+ },+ [featureFlagLogic.actionTypes.setFeatureFlags]: () => actions.ensureAvailableTab(),+ })),+ actionToUrl(({ values }) => ({+ setActiveTab: ({ tab }) => [+ values.datasetsOnly || tab === 'datasets'+ ? getDatasetListUrl(+ removeProjectIdIfPresent(router.values.location.pathname) ===+ urls.aiObservabilityOfflineEvaluations() && router.values.searchParams.tab !== 'datasets'+ ? {}+ : router.values.searchParams+ )+ : `${urls.aiObservabilityOfflineEvaluations()}?tab=experiments`,+ {},+ router.values.hashParams,+ ],+ })),+ urlToAction(({ actions }) => ({+ [urls.aiObservabilityOfflineEvaluations()]: () => actions.ensureAvailableTab(),+ })),+ afterMount(({ actions }) => actions.ensureAvailableTab()),+])diff --git a/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.test.ts b/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.test.tsindex 31848ff476e6..1fb047db36f5 100644--- a/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.test.ts+++ b/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.test.ts@@ -1,11 +1,15 @@import { combineUrl, router } from 'kea-router'+import { FEATURE_FLAGS } from 'lib/constants'+import { featureFlagLogic } from 'lib/logic/featureFlagLogic'+import { removeProjectIdIfPresent } from 'lib/utils/kea-router'import { urls } from 'scenes/urls'import { initKeaTests } from '~/test/init'import * as api from '../generated/api'import type { OfflineExperimentPageApi } from '../generated/api.schemas'+import { offlineEvaluationsLogic } from './offlineEvaluationsLogic'import { offlineExperimentsLogic } from './offlineExperimentsLogic'import { overviewExperiments, overviewScorers, overviewHistory } from './offlineOverviewFixtures'import { readOfflineScorerPreferences, saveOfflineScorerPreferences } from './offlineOverviewState'@@ -53,6 +57,113 @@ describe('offlineExperimentsLogic', () => {afterEach(() => jest.useRealTimers())+ it.each(['new', 'dataset-1'])(+ 'keeps the selected sub-tab and destination when opening dataset %s',+ async (datasetId) => {+ featureFlagLogic.mount()+ featureFlagLogic.actions.setFeatureFlags([], {+ [FEATURE_FLAGS.LLM_ANALYTICS_DATASETS]: true,+ [FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS]: true,+ })+ const tabs = offlineEvaluationsLogic()+ tabs.mount()+ expect(tabs.values.activeTab).toBe('experiments')+ const logic = offlineExperimentsLogic(props)+ logic.mount()+ tabs.actions.setActiveTab('datasets')+ await jest.advanceTimersByTimeAsync(400)+ expect(tabs.values.activeTab).toBe('datasets')+ expect(router.values.searchParams.tab).toBe('datasets')+ router.actions.push(urls.aiObservabilityDataset(datasetId), { search: 'regression' })+ expect(tabs.values.activeTab).toBe('datasets')+ logic.actions.suggestScorerIds([overviewScorers[0].id])+ expect(removeProjectIdIfPresent(router.values.location.pathname)).toBe(+ urls.aiObservabilityDataset(datasetId)+ )+ expect(router.values.searchParams).toEqual({ search: 'regression' })+ tabs.actions.setActiveTab('datasets')+ expect(router.values.searchParams).toMatchObject({ tab: 'datasets', search: 'regression' })+ router.actions.push(urls.aiObservabilityOfflineEvaluations(), { tab: 'experiments' })+ expect(tabs.values.activeTab).toBe('experiments')+ expect(router.values.searchParams.tab).toBe('experiments')+ router.actions.push(urls.aiObservabilityOfflineEvaluations(), { tab: 'unknown' })+ expect(tabs.values.activeTab).toBe('experiments')+ }+ )++ it.each([undefined, 'experiments', 'datasets'])(+ 'opens datasets and preserves filters when offline evals are disabled and tab is %s',+ (tab) => {+ featureFlagLogic.mount()+ featureFlagLogic.actions.setFeatureFlags([], {+ [FEATURE_FLAGS.LLM_ANALYTICS_DATASETS]: true,+ [FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS]: false,+ })+ const filters = { tab, search: 'regression', dataset_status: 'archived', datasets_page: 2 }+ router.actions.push(urls.aiObservabilityOfflineEvaluations(), filters, { panel: 'datasets' })+ const tabs = offlineEvaluationsLogic()+ tabs.mount()++ expect(tabs.values.activeTab).toBe('datasets')+ expect(router.values.searchParams).toEqual({ ...filters, tab: 'datasets' })+ expect(router.values.hashParams).toEqual({ panel: 'datasets' })+ expect(router.values.lastMethod).toBe(tab === 'datasets' ? 'PUSH' : 'REPLACE')++ router.actions.push(urls.aiObservabilityOfflineEvaluations(), { tab: 'experiments' })+ expect(router.values.searchParams.tab).toBe('datasets')+ tabs.actions.setActiveTab('experiments')+ expect(tabs.values.activeTab).toBe('datasets')+ expect(router.values.searchParams.tab).toBe('datasets')+ }+ )++ it('redirects after feature flags arrive without redirecting unrelated pages', () => {+ const tabs = offlineEvaluationsLogic()+ tabs.mount()+ expect(router.values.searchParams.tab).toBeUndefined()+ featureFlagLogic.actions.setFeatureFlags([], { [FEATURE_FLAGS.LLM_ANALYTICS_DATASETS]: true })+ expect(tabs.values.activeTab).toBe('datasets')+ expect(router.values.searchParams.tab).toBe('datasets')++ router.actions.push(urls.aiObservabilityScorers())+ const pathname = router.values.location.pathname+ featureFlagLogic.actions.setFeatureFlags([], { [FEATURE_FLAGS.LLM_ANALYTICS_DATASETS]: true })+ expect(router.values.location.pathname).toBe(pathname)+ expect(router.values.searchParams.tab).toBeUndefined()+ })++ it('remembers collapsed score trends and defers score requests until the first expansion', async () => {+ const logic = offlineExperimentsLogic(props)+ const unmount = logic.mount()+ expect(logic.values.scoreTrendsExpanded).toBe(true)+ logic.actions.toggleScoreTrends()+ unmount()+ router.actions.replace(urls.aiObservabilityOfflineEvaluations(), { scores: '' })+ jest.clearAllMocks()+ const remounted = offlineExperimentsLogic(props)+ remounted.mount()+ expect(remounted.values.scoreTrendsExpanded).toBe(false)+ await jest.advanceTimersByTimeAsync(400)+ expect(list).toHaveBeenCalledTimes(1)+ expect(api.aiObservabilityOfflineExperimentsScorerSummariesList).not.toHaveBeenCalled()+ expect(api.llmAnalyticsScoreDefinitionsList).not.toHaveBeenCalled()++ remounted.actions.setFilters({ run_source: 'ci' })+ remounted.actions.refresh()+ await jest.advanceTimersByTimeAsync(400)+ expect(api.aiObservabilityOfflineExperimentsScorerSummariesList).not.toHaveBeenCalled()+ expect(api.llmAnalyticsScoreDefinitionsList).not.toHaveBeenCalled()++ remounted.actions.toggleScoreTrends()+ await jest.advanceTimersByTimeAsync(400)+ expect(api.aiObservabilityOfflineExperimentsScorerSummariesList).toHaveBeenCalledTimes(1)+ expect(remounted.values.scorerIds).toEqual(overviewScorers.map(({ id }) => id))+ remounted.actions.toggleScoreTrends()+ remounted.actions.toggleScoreTrends()+ await jest.advanceTimersByTimeAsync(400)+ expect(api.aiObservabilityOfflineExperimentsScorerSummariesList).toHaveBeenCalledTimes(1)+ })+it('shares experiment tag filters with trends and restores them from the URL', async () => {const logic = offlineExperimentsLogic(props)logic.mount()diff --git a/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.ts b/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.tsindex f5941af6f903..b880fb26d040 100644--- a/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.ts+++ b/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.ts@@ -15,6 +15,7 @@ import { loaders } from 'kea-loaders'import { actionToUrl, router, urlToAction } from 'kea-router'import { dayjs } from 'lib/dayjs'+import { removeProjectIdIfPresent } from 'lib/utils/kea-router'import { urls } from 'scenes/urls'import { cleanTagFilters, restoreTagFilters, saveTagFilters } from '../components/resourceTagFilters'@@ -70,6 +71,8 @@ export interface offlineExperimentsLogicValues {} | nullrefreshKey: numberresolvedNow: string+ scoreTrendsExpanded: boolean+ scoreTrendsInitialized: booleanscorerIds: string[] | nullscorerOffset: numberscorerOptions: PaginatedScoreDefinitionListApi | null@@ -95,6 +98,9 @@ export interface offlineExperimentsLogicActions {hydrateUrl: (search: Record<string, unknown>) => {search: Record<string, unknown>}+ initializeScoreTrends: () => {+ value: true+ }loadOfflineExperiments: (_?: any) => anyloadOfflineExperimentsFailure: (error: string,@@ -186,6 +192,9 @@ export interface offlineExperimentsLogicActions {suggestScorerIds: (scorerIds: string[]) => {scorerIds: string[]}+ toggleScoreTrends: () => {+ value: true+ }}// Generated by kea-typegen. Update if you're an agent, ignore if you're human.@@ -218,6 +227,8 @@ export const offlineExperimentsLogic: LogicWrapper<offlineExperimentsLogicType>key(({ teamId, userId }) => `${teamId}:${userId}`),path((key) => ['products', 'ai_observability', 'offlineExperimentsLogic', key]),actions({+ toggleScoreTrends: true,+ initializeScoreTrends: true,setHoveredTrend: (chartId: string, timestamp: number) => ({ chartId, timestamp }),clearHoveredTrend: (chartId: string) => ({ chartId }),setFilters: (filters: Partial<OfflineExperimentFilters>) => ({ filters }),@@ -340,6 +351,8 @@ export const offlineExperimentsLogic: LogicWrapper<offlineExperimentsLogicType>],})),reducers(({ props }) => ({+ scoreTrendsExpanded: [true, { persist: true }, { toggleScoreTrends: (expanded) => !expanded }],+ scoreTrendsInitialized: [false, { initializeScoreTrends: () => true }],filters: [{ date_from: '-30d' } as OfflineExperimentFilters,{@@ -514,17 +527,32 @@ export const offlineExperimentsLogic: LogicWrapper<offlineExperimentsLogicType>],}),listeners(({ actions, values, props }) => ({+ toggleScoreTrends: () => {+ if (values.scoreTrendsExpanded && !values.scoreTrendsInitialized) {+ actions.initializeScoreTrends()+ }+ },+ initializeScoreTrends: () => {+ if (!values.scorerIds?.length && values.dateRange) {+ actions.loadOfflineSuggestedScorers()+ }+ },setFilters: () => {saveTagFilters('experiments', cleanTagFilters(values.filters), props.teamId, props.userId)actions.loadOfflineExperiments()- if (!values.scorerIds?.length && values.dateRange && values.suggestedScorers === null) {+ if (+ values.scoreTrendsInitialized &&+ !values.scorerIds?.length &&+ values.dateRange &&+ values.suggestedScorers === null+ ) {actions.loadOfflineSuggestedScorers()}},hydrateUrl: () => {saveTagFilters('experiments', cleanTagFilters(values.filters), props.teamId, props.userId)actions.loadOfflineExperiments()- if (!values.scorerIds?.length && values.dateRange) {+ if (values.scoreTrendsInitialized && !values.scorerIds?.length && values.dateRange) {if (values.suggestedScorers) {actions.suggestScorerIds(values.suggestedScorers)} else {@@ -538,7 +566,7 @@ export const offlineExperimentsLogic: LogicWrapper<offlineExperimentsLogicType>previousPage: () => actions.loadOfflineExperiments(),refresh: () => {actions.loadOfflineExperiments()- if (!values.scorerIds?.length && values.dateRange) {+ if (values.scoreTrendsInitialized && !values.scorerIds?.length && values.dateRange) {actions.loadOfflineSuggestedScorers()}},@@ -565,23 +593,35 @@ export const offlineExperimentsLogic: LogicWrapper<offlineExperimentsLogicType>},setScorerIds: ({ scorerIds }) => {saveOfflineScorerPreferences(props.userId, props.teamId, scorerIds)- if (!scorerIds.length && values.dateRange) {+ if (values.scoreTrendsInitialized && !scorerIds.length && values.dateRange) {actions.loadOfflineSuggestedScorers()}},})),actionToUrl(({ values }) => {- const location = (): [string, Record<string, unknown>, Record<string, unknown>, { replace: boolean }] => [- urls.aiObservabilityOfflineEvaluations(),- {- ...offlineFiltersToUrl(values.filters),- scores: values.scorerIds?.join(','),- cursor_stack: values.cursorStack.length ? values.cursorStack : undefined,- list_now: values.cursorStack.length ? values.resolvedNow : undefined,- },- {},- { replace: true },- ]+ const location = ():+ | [string, Record<string, unknown>, Record<string, unknown>, { replace: boolean }]+ | undefined => {+ if (+ removeProjectIdIfPresent(router.values.location.pathname) !==+ urls.aiObservabilityOfflineEvaluations() ||+ router.values.searchParams.tab === 'datasets'+ ) {+ return+ }+ return [+ urls.aiObservabilityOfflineEvaluations(),+ {+ tab: router.values.searchParams.tab === 'experiments' ? 'experiments' : undefined,+ ...offlineFiltersToUrl(values.filters),+ scores: values.scorerIds?.join(','),+ cursor_stack: values.cursorStack.length ? values.cursorStack : undefined,+ list_now: values.cursorStack.length ? values.resolvedNow : undefined,+ },+ {},+ { replace: true },+ ]+ }return {setFilters: location,hydrateUrl: location,@@ -594,10 +634,15 @@ export const offlineExperimentsLogic: LogicWrapper<offlineExperimentsLogicType>}),urlToAction(({ actions }) => ({[urls.aiObservabilityOfflineEvaluations()]: (_, search, __, { method }) => {- if (method !== 'REPLACE') {+ if (method !== 'REPLACE' && search.tab !== 'datasets') {actions.hydrateUrl(search)}},})),- afterMount(({ actions }) => actions.hydrateUrl(router.values.searchParams)),+ afterMount(({ actions, values }) => {+ actions.hydrateUrl(router.values.searchParams)+ if (values.scoreTrendsExpanded) {+ actions.initializeScoreTrends()+ }+ }),])diff --git a/products/ai_observability/manifest.tsx b/products/ai_observability/manifest.tsxindex 114916250d3f..54a261a09a5f 100644--- a/products/ai_observability/manifest.tsx+++ b/products/ai_observability/manifest.tsx@@ -193,8 +193,7 @@ export const manifest: ProductManifest = {'/ai-observability/clusters': ['AIObservabilityClusters', 'aiObservabilityClusters'],'/ai-observability/clusters/:runId': ['AIObservabilityClusters', 'aiObservabilityClusters'],[AI_OBSERVABILITY_CLUSTER_URL_PATTERN]: ['AIObservabilityCluster', 'aiObservabilityCluster'],- '/ai-evals/datasets': ['AIObservabilityDatasets', 'aiObservabilityDatasets'],- '/ai-evals/datasets/:id': ['AIObservabilityDataset', 'aiObservabilityDataset'],+ '/ai-evals/evaluations/offline/datasets/:id': ['AIObservabilityDataset', 'aiObservabilityDataset'],'/ai-evals/taggers': ['AIObservabilityTags', 'aiObservabilityTags'],'/ai-evals/taggers/:id': ['AIObservabilityTag', 'aiObservabilityTag'],'/ai-evals/evaluations': ['AIObservabilityEvaluations', 'aiObservabilityEvaluations'],@@ -238,6 +237,10 @@ export const manifest: ProductManifest = {},'/ai-evals': (_params, searchParams, hashParams) =>combineUrl(urls.aiObservabilityEvaluations(), searchParams, hashParams).url,+ '/ai-evals/datasets': (_params, searchParams, hashParams) =>+ combineUrl(urls.aiObservabilityDatasets(), { ...searchParams, tab: 'datasets' }, hashParams).url,+ '/ai-evals/datasets/:id': (params, searchParams, hashParams) =>+ combineUrl(urls.aiObservabilityDataset(params.id), searchParams, hashParams).url,'/ai-evals/evaluations/offline': (_params, searchParams, hashParams) =>combineUrl(urls.aiObservabilityOfflineEvaluations(), searchParams, hashParams).url,'/ai-evals/tags': (_params, searchParams, hashParams) =>@@ -297,7 +300,7 @@ export const manifest: ProductManifest = {'/llm-analytics/clusters/:runId/:clusterId': (params, searchParams, hashParams) =>combineUrl(urls.aiObservabilityCluster(params.runId, params.clusterId), searchParams, hashParams).url,'/llm-analytics/datasets': (_params, searchParams, hashParams) =>- combineUrl(urls.aiObservabilityDatasets(), searchParams, hashParams).url,+ combineUrl(urls.aiObservabilityDatasets(), { ...searchParams, tab: 'datasets' }, hashParams).url,'/llm-analytics/datasets/:id': (params, searchParams, hashParams) =>combineUrl(urls.aiObservabilityDataset(params.id), searchParams, hashParams).url,'/llm-analytics/tags': (_params, searchParams, hashParams) =>@@ -405,9 +408,9 @@ export const manifest: ProductManifest = {return `/ai-observability/sessions/${id}${stringifiedParams ? `?${stringifiedParams}` : ''}`},aiObservabilityPlayground: (): string => '/ai-observability/playground',- aiObservabilityDatasets: (): string => '/ai-evals/datasets',+ aiObservabilityDatasets: (): string => '/ai-evals/evaluations/offline/experiments?tab=datasets',aiObservabilityDataset: (id: string, params?: { item?: string }): string =>- combineUrl(`/ai-evals/datasets/${id}`, params).url,+ combineUrl(`/ai-evals/evaluations/offline/datasets/${id}`, params).url,aiObservabilityTags: (): string => '/ai-evals/taggers',aiObservabilityTag: (id: string): string => `/ai-evals/taggers/${id}`,aiObservabilityEvaluations: (): string => '/ai-evals/evaluations',@@ -482,34 +485,22 @@ export const manifest: ProductManifest = {searchKeywords: ['topics'],sceneKey: 'AIObservabilityClusters',},- {- path: 'Datasets',- intents: [ProductKey.LLM_DATASETS],- category: ProductItemCategory.AI_ENGINEERING,- type: 'llm_datasets',- iconType: 'llm_datasets' as FileSystemIconType,- iconColor: ['var(--color-product-llm-datasets-light)', 'var(--color-product-llm-datasets-dark)'],- href: urls.aiObservabilityDatasets(),- searchKeywords: ['test sets', 'golden set'],- flag: FEATURE_FLAGS.LLM_ANALYTICS_DATASETS,- tags: ['beta'],- sceneKey: 'AIObservabilityDatasets',- },{path: 'Evaluations',- intents: [ProductKey.LLM_EVALUATIONS],+ intents: [ProductKey.LLM_EVALUATIONS, ProductKey.LLM_DATASETS],category: ProductItemCategory.AI_ENGINEERING,type: 'llm_evaluations',iconType: 'llm_evaluations' as FileSystemIconType,iconColor: ['var(--color-product-llm-evaluations-light)', 'var(--color-product-llm-evaluations-dark)'],href: urls.aiObservabilityEvaluations(),- searchKeywords: ['llm judge', 'llm as a judge', 'grading'],+ searchKeywords: ['llm judge', 'llm as a judge', 'grading', 'datasets', 'test sets', 'golden set'],searchTabs: [{name: 'Offline evals',href: urls.aiObservabilityOfflineEvaluations(),flag: FEATURE_FLAGS.AI_OBSERVABILITY_OFFLINE_EVALUATIONS,},+ { name: 'Datasets', href: urls.aiObservabilityDatasets(), flag: FEATURE_FLAGS.LLM_ANALYTICS_DATASETS },{ name: 'Scorers', href: urls.aiObservabilityScorers() },],sceneKey: 'AIObservabilityEvaluations',diff --git a/services/mcp/src/tools/links/app-url-manifest.json b/services/mcp/src/tools/links/app-url-manifest.jsonindex adeaaac15990..c891160bdfe4 100644--- a/services/mcp/src/tools/links/app-url-manifest.json+++ b/services/mcp/src/tools/links/app-url-manifest.json@@ -70,12 +70,12 @@"scope": "project"},"aiObservabilityDataset": {- "template": "/ai-evals/datasets/{id}",+ "template": "/ai-evals/evaluations/offline/datasets/{id}","params": ["id"],"scope": "project"},"aiObservabilityDatasets": {- "template": "/ai-evals/datasets",+ "template": "/ai-evals/evaluations/offline/experiments?tab=datasets","params": [],"scope": "project"},diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/generate-app-url.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/generate-app-url.jsonindex 79845971479d..cf3d494ec3d4 100644--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/generate-app-url.json+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/generate-app-url.json@@ -13,7 +13,7 @@"type": "object"},"url": {- "description": "A path template copied verbatim from the catalog below (e.g. `/persons/{uuid}`). Its `{placeholders}` are filled from `params`. These slugs come from PostHog's canonical route table, so they are always correct — never pass a path that is not in this list.\n\n/account-connected/{kind}\n/account/credential-review\n/account/social-connected\n/activity-logs\n/activity/{tab}\n/agentic/account-mismatch\n/agentic/authorize\n/ai\n/ai-enrichment\n/ai-evals/datasets\n/ai-evals/datasets/{id}\n/ai-evals/evaluations\n/ai-evals/evaluations/offline/experiments\n/ai-evals/evaluations/offline/experiments/{experimentId}\n/ai-evals/evaluations/scorers\n/ai-evals/evaluations/scorers/{scorerId}\n/ai-evals/evaluations/scorers/{scorerId}/offline\n/ai-evals/evaluations/templates\n/ai-evals/evaluations/{id}\n/ai-evals/taggers\n/ai-evals/taggers/{id}\n/ai-observability/clusters\n/ai-observability/clusters/{runId}/{clusterId}\n/ai-observability/dashboard\n/ai-observability/errors\n/ai-observability/generations\n/ai-observability/playground\n/ai-observability/reviews\n/ai-observability/self-driving\n/ai-observability/sentiment\n/ai-observability/sessions\n/ai-observability/sessions/{id}\n/ai-observability/tools\n/ai-observability/traces\n/ai-observability/traces/{id}\n/ai-observability/users\n/ai/history\n/alerts\n/analytics\n/analytics/all\n/analytics/mine\n/analytics/product-analytics\n/analytics/spaces\n/approvals/{id}\n/autoresearch\n/autoresearch/new\n/autoresearch/{id}\n/bi\n/bi/new\n/bi/{insightShortId}\n/billing/authorization_status\n/broadcasts\n/broadcasts/new\n/broadcasts/{id}\n/business-knowledge\n/business-knowledge/playground\n/business-knowledge/settings\n/business-knowledge/{id}\n/canvas\n/canvases\n/canvases/new\n/canvases/{id}\n/cli/authorize\n/cli/live\n/code-review\n/code/canvas/{channelId}/{dashboardId}\n/code/channel/{channelId}\n/code/loop/{loopId}\n/code/task/{taskId}\n/cohorts\n/cohorts/{id}\n/cohorts/{id}/calculation-history\n/connect/vercel/link\n/coupons/{campaign}\n/create-organization\n/cross-project-dashboards\n/cross-project-dashboards/{id}\n/customer_analytics\n/customer_analytics/accounts\n/customer_analytics/accounts/by-external-id/{externalId}\n/customer_analytics/accounts/{accountId}\n/customer_analytics/announcements\n/customer_analytics/configuration\n/customer_analytics/dashboard\n/customer_analytics/feature-requests\n/customer_analytics/feed\n/customer_analytics/journeys\n/customer_analytics/journeys/new\n/customer_analytics/journeys/templates\n/customer_analytics/journeys/{id}/edit\n/customer_analytics/notes\n/customer_analytics/tasks\n/dashboard\n/dashboard/templates/{templateId}/copy-to-project\n/dashboard/{id}\n/dashboard/{id}/sharing\n/dashboard/{id}/subscriptions\n/dashboard/{id}/subscriptions/{subscriptionId}\n/dashboard/{id}/tiles/{tileId}\n/data-catalog\n/data-catalog/metrics/{name}\n/data-management/actions\n/data-management/actions/new\n/data-management/actions/new/\n/data-management/actions/{id}\n/data-management/annotations\n/data-management/annotations/{id}\n/data-management/core-events\n/data-management/database\n/data-management/destinations\n/data-management/event-filtering\n/data-management/events\n/data-management/events/{id}\n/data-management/events/{id}/edit\n/data-management/history\n/data-management/ingestion-warnings\n/data-management/ingestion-warnings-v2\n/data-management/managed-viewsets\n/data-management/materialized-columns\n/data-management/properties\n/data-management/properties/{id}\n/data-management/properties/{id}/edit\n/data-management/revenue\n/data-management/schema\n/data-management/sources\n/data-management/sources/{id}/schemas\n/data-management/sources/{sourceId}/schemas/{schemaId}\n/data-management/transformations\n/data-management/variables\n/data-management/variables/{id}\n/data-management/variables/{id}/edit\n/data-management/warehouse-destinations\n/data-management/warehouse-properties\n/data-ops\n/data-warehouse/connect\n/data-warehouse/new-source\n/debug\n/debug/hog\n/debug/precompute\n/early_access_features\n/early_access_features/{id}\n/embedded/{token}\n/endpoints\n/endpoints/{name}\n/engineering-analytics/authors\n/engineering-analytics/authors/{handle}\n/engineering-analytics/deploys\n/engineering-analytics/overview\n/engineering-analytics/pull-requests\n/engineering-analytics/repos/{repoOwner}/{repoName}/actions/runs/{runId}\n/engineering-analytics/repos/{repoOwner}/{repoName}/actions/workflows/{workflowName}\n/engineering-analytics/repos/{repoOwner}/{repoName}/pull-requests/{number}\n/engineering-analytics/repos/{repoOwner}/{repoName}/pull-requests/{number}/ci-explorer\n/engineering-analytics/teams\n/engineering-analytics/teams/{ownerTeam}\n/engineering-analytics/tests\n/engineering-analytics/workflows\n/error-tracking\n/error-tracking/alerts/new/{templateId}\n/error-tracking/alerts/{id}\n/error-tracking/fingerprint/{fingerprint}\n/error-tracking/{id}\n/etl\n/events/{id}/{timestamp}\n/experiments\n/experiments/shared-metrics\n/experiments/shared-metrics/{id}\n/experiments/staff\n/experiments/{id}\n/exports\n/feature-flags\n/feature-flags/new\n/feature-flags/staff\n/feature-flags/staff/cohorts\n/feature-flags/templates\n/feature-flags/{id}\n/files\n/functions/new/{templateId}\n/functions/{id}\n/games/368hedgehogs\n/games/flappyhog\n/games/shipit\n/groups/{groupTypeIndex}\n/groups/{groupTypeIndex}/new\n/groups/{groupTypeIndex}/{groupKey}\n/health\n/health/alerts\n/health/pipeline-status\n/health/sdk-health\n/health/{category}\n/heatmaps\n/heatmaps/new\n/heatmaps/recording\n/heatmaps/{id}\n/home\n/identity-matching\n/inbox\n/inbox/reports/triage\n/inbox/scout-trials\n/inbox/scouts/findings\n/inbox/scouts/runs\n/inbox/scouts/scratchpad\n/inbox/scouts/{skillName}\n/inbox/{tab}/{reportId}\n/insights\n/insights/new\n/insights/quick-start\n/insights/{id}\n/insights/{id}/edit\n/insights/{id}/sharing\n/insights/{id}/subscriptions\n/insights/{id}/subscriptions/{subscriptionId}\n/insights/{insightShortId}/alerts\n/instance/async_migrations\n/instance/async_migrations/future\n/instance/async_migrations/settings\n/instance/dead_letter_queue\n/instance/kafka_inspector\n/instance/metrics\n/instance/settings\n/instance/staff_users\n/instance/status\n/integrations/stripe/confirm-install\n/integrations/vercel/link-error\n/integrations/{kind}/callback\n/integrations/{slug}\n/legal\n/legal/new/{type}\n/login\n/login/2fa\n/login/2fa_setup\n/logs\n/logs/alerts/{alertId}/notifications/{hogFunctionId}\n/logs/alerts/{id}\n/logs/drop-rules/new\n/logs/drop-rules/{id}\n/logs/retention-rules/new\n/logs/retention-rules/{id}\n/managed_migrations\n/managed_migrations/new\n/marketing\n/mcp-analytics\n/mcp-analytics/activity\n/mcp-analytics/dashboard\n/mcp-analytics/intent-clustering\n/mcp-analytics/missing-capabilities\n/mcp-analytics/notifications\n/mcp-analytics/sessions\n/mcp-analytics/tool-quality\n/mcp-analytics/tool-quality/{toolName}\n/mcp-registry\n/mcp-servers\n/mcp-servers/agent/{id}\n/mcp-servers/member/{id}\n/mcp-servers/server/{id}\n/mcp-servers/{tab}\n/metrics\n/metrics/dashboard-review\n/models\n/models/{id}\n/move-to-cloud\n/my-tickets\n/notebooks\n/notebooks/widgets/{widgetId}\n/notebooks/{shortId}\n/oauth/authorize\n/onboarding\n/onboarding/simple\n/organization-deactivated\n/organization-pending-deletion\n/organization/billing\n/organization/billing/overview\n/organization/billing/real-time-usage\n/organization/confirm-creation\n/organization/create-project\n/organization/legal\n/organization/startups\n/person/{id}\n/persons\n/persons/{uuid}\n/pipeline/batch-exports/new/{service}\n/pipeline/batch-exports/{id}\n/pipeline/new/\n/pipeline/plugins/{id}\n/platform-alerts\n/platform-alerts/{id}\n/preflight\n/product_tours\n/product_tours/{id}\n/project-pending-deletion\n/prompt-management/prompts\n/prompt-management/prompts/{name}\n/pulse\n/replay-vision\n/replay-vision/new/goal\n/replay-vision/observations/{observationId}\n/replay-vision/{id}/budget\n/replay-vision/{id}/configure\n/replay-vision/{id}/details\n/replay-vision/{id}/goal\n/replay-vision/{id}/overview\n/replay-vision/{id}/triggers\n/replay/file-playback\n/replay/home\n/replay/kiosk\n/replay/playlists/{id}\n/replay/settings\n/replay/{id}\n/reset\n/reset/{userUuid}/{token}\n/reset_2fa/{userUuid}/{token}\n/resource-transfer/{resourceKind}/{resourceId}\n/sessions/{id}\n/settings/environment-approvals\n/settings/organization-authentication/{feature}/{configId}\n/settings/project\n/settings/user-feature-previews\n/shared/{token}\n/shared_dashboard/{shareToken}\n/signup\n/signup/{id}\n/site/{url}\n/skills\n/skills/community\n/skills/{categoryTab}\n/skills/{name}\n/slack-task-context\n/spaces\n/spaces/new\n/spaces/{id}\n/spaces/{id}/canvases\n/spaces/{id}/new\n/spaces/{id}/settings\n/sql\n/stamphog\n/stamphog/digests\n/stamphog/install/callback\n/stamphog/runs\n/startups\n/streamlit-apps\n/streamlit-apps/new\n/streamlit-apps/{id}\n/streamlit-apps/{id}/edit\n/subscriptions\n/subscriptions/new\n/subscriptions/{id}\n/subscriptions/{id}/edit\n/support\n/support/settings\n/support/tickets\n/support/tickets/{ticketId}\n/surveys\n/surveys/form/new\n/surveys/guided/new\n/surveys/{id}\n/tasks\n/tasks/new\n/tasks/{taskId}\n/terminal\n/themes/custom-css\n/toolbar\n/tools\n/tracing\n/tracing/retention-rules/new\n/tracing/retention-rules/{id}\n/unsubscribe\n/uptime-monitoring\n/verify_email\n/visual-review\n/visual-review/repos/{repoOwner}/{repoName}/flakiness\n/visual-review/repos/{repoOwner}/{repoName}/runs\n/visual-review/repos/{repoOwner}/{repoName}/runs/{runId}\n/visual-review/repos/{repoOwner}/{repoName}/snapshots\n/visual-review/repos/{repoOwner}/{repoName}/{runType}/snapshots/{identifier}\n/visual-review/runs/{runId}\n/visual-review/settings\n/web\n/web-scripts\n/web-scripts/new\n/web/agents\n/web/bots\n/web/content-autopilot\n/web/health\n/web/live\n/web/marketing\n/web/page-performance\n/web/page-reports\n/web/recap\n/web/session-attribution-explorer\n/web/web-vitals\n/welcome\n/wizard/runs\n/workflows\n/workflows/library/messages/{id}\n/workflows/library/templates/new\n/workflows/library/templates/{id}\n/workflows/new/workflow\n/workflows/{id}/{tab}",+ "description": "A path template copied verbatim from the catalog below (e.g. `/persons/{uuid}`). Its `{placeholders}` are filled from `params`. These slugs come from PostHog's canonical route table, so they are always correct — never pass a path that is not in this list.\n\n/account-connected/{kind}\n/account/credential-review\n/account/social-connected\n/activity-logs\n/activity/{tab}\n/agentic/account-mismatch\n/agentic/authorize\n/ai\n/ai-enrichment\n/ai-evals/evaluations\n/ai-evals/evaluations/offline/datasets/{id}\n/ai-evals/evaluations/offline/experiments\n/ai-evals/evaluations/offline/experiments/{experimentId}\n/ai-evals/evaluations/offline/experiments?tab=datasets\n/ai-evals/evaluations/scorers\n/ai-evals/evaluations/scorers/{scorerId}\n/ai-evals/evaluations/scorers/{scorerId}/offline\n/ai-evals/evaluations/templates\n/ai-evals/evaluations/{id}\n/ai-evals/taggers\n/ai-evals/taggers/{id}\n/ai-observability/clusters\n/ai-observability/clusters/{runId}/{clusterId}\n/ai-observability/dashboard\n/ai-observability/errors\n/ai-observability/generations\n/ai-observability/playground\n/ai-observability/reviews\n/ai-observability/self-driving\n/ai-observability/sentiment\n/ai-observability/sessions\n/ai-observability/sessions/{id}\n/ai-observability/tools\n/ai-observability/traces\n/ai-observability/traces/{id}\n/ai-observability/users\n/ai/history\n/alerts\n/analytics\n/analytics/all\n/analytics/mine\n/analytics/product-analytics\n/analytics/spaces\n/approvals/{id}\n/autoresearch\n/autoresearch/new\n/autoresearch/{id}\n/bi\n/bi/new\n/bi/{insightShortId}\n/billing/authorization_status\n/broadcasts\n/broadcasts/new\n/broadcasts/{id}\n/business-knowledge\n/business-knowledge/playground\n/business-knowledge/settings\n/business-knowledge/{id}\n/canvas\n/canvases\n/canvases/new\n/canvases/{id}\n/cli/authorize\n/cli/live\n/code-review\n/code/canvas/{channelId}/{dashboardId}\n/code/channel/{channelId}\n/code/loop/{loopId}\n/code/task/{taskId}\n/cohorts\n/cohorts/{id}\n/cohorts/{id}/calculation-history\n/connect/vercel/link\n/coupons/{campaign}\n/create-organization\n/cross-project-dashboards\n/cross-project-dashboards/{id}\n/customer_analytics\n/customer_analytics/accounts\n/customer_analytics/accounts/by-external-id/{externalId}\n/customer_analytics/accounts/{accountId}\n/customer_analytics/announcements\n/customer_analytics/configuration\n/customer_analytics/dashboard\n/customer_analytics/feature-requests\n/customer_analytics/feed\n/customer_analytics/journeys\n/customer_analytics/journeys/new\n/customer_analytics/journeys/templates\n/customer_analytics/journeys/{id}/edit\n/customer_analytics/notes\n/customer_analytics/tasks\n/dashboard\n/dashboard/templates/{templateId}/copy-to-project\n/dashboard/{id}\n/dashboard/{id}/sharing\n/dashboard/{id}/subscriptions\n/dashboard/{id}/subscriptions/{subscriptionId}\n/dashboard/{id}/tiles/{tileId}\n/data-catalog\n/data-catalog/metrics/{name}\n/data-management/actions\n/data-management/actions/new\n/data-management/actions/new/\n/data-management/actions/{id}\n/data-management/annotations\n/data-management/annotations/{id}\n/data-management/core-events\n/data-management/database\n/data-management/destinations\n/data-management/event-filtering\n/data-management/events\n/data-management/events/{id}\n/data-management/events/{id}/edit\n/data-management/history\n/data-management/ingestion-warnings\n/data-management/ingestion-warnings-v2\n/data-management/managed-viewsets\n/data-management/materialized-columns\n/data-management/properties\n/data-management/properties/{id}\n/data-management/properties/{id}/edit\n/data-management/revenue\n/data-management/schema\n/data-management/sources\n/data-management/sources/{id}/schemas\n/data-management/sources/{sourceId}/schemas/{schemaId}\n/data-management/transformations\n/data-management/variables\n/data-management/variables/{id}\n/data-management/variables/{id}/edit\n/data-management/warehouse-destinations\n/data-management/warehouse-properties\n/data-ops\n/data-warehouse/connect\n/data-warehouse/new-source\n/debug\n/debug/hog\n/debug/precompute\n/early_access_features\n/early_access_features/{id}\n/embedded/{token}\n/endpoints\n/endpoints/{name}\n/engineering-analytics/authors\n/engineering-analytics/authors/{handle}\n/engineering-analytics/deploys\n/engineering-analytics/overview\n/engineering-analytics/pull-requests\n/engineering-analytics/repos/{repoOwner}/{repoName}/actions/runs/{runId}\n/engineering-analytics/repos/{repoOwner}/{repoName}/actions/workflows/{workflowName}\n/engineering-analytics/repos/{repoOwner}/{repoName}/pull-requests/{number}\n/engineering-analytics/repos/{repoOwner}/{repoName}/pull-requests/{number}/ci-explorer\n/engineering-analytics/teams\n/engineering-analytics/teams/{ownerTeam}\n/engineering-analytics/tests\n/engineering-analytics/workflows\n/error-tracking\n/error-tracking/alerts/new/{templateId}\n/error-tracking/alerts/{id}\n/error-tracking/fingerprint/{fingerprint}\n/error-tracking/{id}\n/etl\n/events/{id}/{timestamp}\n/experiments\n/experiments/shared-metrics\n/experiments/shared-metrics/{id}\n/experiments/staff\n/experiments/{id}\n/exports\n/feature-flags\n/feature-flags/new\n/feature-flags/staff\n/feature-flags/staff/cohorts\n/feature-flags/templates\n/feature-flags/{id}\n/files\n/functions/new/{templateId}\n/functions/{id}\n/games/368hedgehogs\n/games/flappyhog\n/games/shipit\n/groups/{groupTypeIndex}\n/groups/{groupTypeIndex}/new\n/groups/{groupTypeIndex}/{groupKey}\n/health\n/health/alerts\n/health/pipeline-status\n/health/sdk-health\n/health/{category}\n/heatmaps\n/heatmaps/new\n/heatmaps/recording\n/heatmaps/{id}\n/home\n/identity-matching\n/inbox\n/inbox/reports/triage\n/inbox/scout-trials\n/inbox/scouts/findings\n/inbox/scouts/runs\n/inbox/scouts/scratchpad\n/inbox/scouts/{skillName}\n/inbox/{tab}/{reportId}\n/insights\n/insights/new\n/insights/quick-start\n/insights/{id}\n/insights/{id}/edit\n/insights/{id}/sharing\n/insights/{id}/subscriptions\n/insights/{id}/subscriptions/{subscriptionId}\n/insights/{insightShortId}/alerts\n/instance/async_migrations\n/instance/async_migrations/future\n/instance/async_migrations/settings\n/instance/dead_letter_queue\n/instance/kafka_inspector\n/instance/metrics\n/instance/settings\n/instance/staff_users\n/instance/status\n/integrations/stripe/confirm-install\n/integrations/vercel/link-error\n/integrations/{kind}/callback\n/integrations/{slug}\n/legal\n/legal/new/{type}\n/login\n/login/2fa\n/login/2fa_setup\n/logs\n/logs/alerts/{alertId}/notifications/{hogFunctionId}\n/logs/alerts/{id}\n/logs/drop-rules/new\n/logs/drop-rules/{id}\n/logs/retention-rules/new\n/logs/retention-rules/{id}\n/managed_migrations\n/managed_migrations/new\n/marketing\n/mcp-analytics\n/mcp-analytics/activity\n/mcp-analytics/dashboard\n/mcp-analytics/intent-clustering\n/mcp-analytics/missing-capabilities\n/mcp-analytics/notifications\n/mcp-analytics/sessions\n/mcp-analytics/tool-quality\n/mcp-analytics/tool-quality/{toolName}\n/mcp-registry\n/mcp-servers\n/mcp-servers/agent/{id}\n/mcp-servers/member/{id}\n/mcp-servers/server/{id}\n/mcp-servers/{tab}\n/metrics\n/metrics/dashboard-review\n/models\n/models/{id}\n/move-to-cloud\n/my-tickets\n/notebooks\n/notebooks/widgets/{widgetId}\n/notebooks/{shortId}\n/oauth/authorize\n/onboarding\n/onboarding/simple\n/organization-deactivated\n/organization-pending-deletion\n/organization/billing\n/organization/billing/overview\n/organization/billing/real-time-usage\n/organization/confirm-creation\n/organization/create-project\n/organization/legal\n/organization/startups\n/person/{id}\n/persons\n/persons/{uuid}\n/pipeline/batch-exports/new/{service}\n/pipeline/batch-exports/{id}\n/pipeline/new/\n/pipeline/plugins/{id}\n/platform-alerts\n/platform-alerts/{id}\n/preflight\n/product_tours\n/product_tours/{id}\n/project-pending-deletion\n/prompt-management/prompts\n/prompt-management/prompts/{name}\n/pulse\n/replay-vision\n/replay-vision/new/goal\n/replay-vision/observations/{observationId}\n/replay-vision/{id}/budget\n/replay-vision/{id}/configure\n/replay-vision/{id}/details\n/replay-vision/{id}/goal\n/replay-vision/{id}/overview\n/replay-vision/{id}/triggers\n/replay/file-playback\n/replay/home\n/replay/kiosk\n/replay/playlists/{id}\n/replay/settings\n/replay/{id}\n/reset\n/reset/{userUuid}/{token}\n/reset_2fa/{userUuid}/{token}\n/resource-transfer/{resourceKind}/{resourceId}\n/sessions/{id}\n/settings/environment-approvals\n/settings/organization-authentication/{feature}/{configId}\n/settings/project\n/settings/user-feature-previews\n/shared/{token}\n/shared_dashboard/{shareToken}\n/signup\n/signup/{id}\n/site/{url}\n/skills\n/skills/community\n/skills/{categoryTab}\n/skills/{name}\n/slack-task-context\n/spaces\n/spaces/new\n/spaces/{id}\n/spaces/{id}/canvases\n/spaces/{id}/new\n/spaces/{id}/settings\n/sql\n/stamphog\n/stamphog/digests\n/stamphog/install/callback\n/stamphog/runs\n/startups\n/streamlit-apps\n/streamlit-apps/new\n/streamlit-apps/{id}\n/streamlit-apps/{id}/edit\n/subscriptions\n/subscriptions/new\n/subscriptions/{id}\n/subscriptions/{id}/edit\n/support\n/support/settings\n/support/tickets\n/support/tickets/{ticketId}\n/surveys\n/surveys/form/new\n/surveys/guided/new\n/surveys/{id}\n/tasks\n/tasks/new\n/tasks/{taskId}\n/terminal\n/themes/custom-css\n/toolbar\n/tools\n/tracing\n/tracing/retention-rules/new\n/tracing/retention-rules/{id}\n/unsubscribe\n/uptime-monitoring\n/verify_email\n/visual-review\n/visual-review/repos/{repoOwner}/{repoName}/flakiness\n/visual-review/repos/{repoOwner}/{repoName}/runs\n/visual-review/repos/{repoOwner}/{repoName}/runs/{runId}\n/visual-review/repos/{repoOwner}/{repoName}/snapshots\n/visual-review/repos/{repoOwner}/{repoName}/{runType}/snapshots/{identifier}\n/visual-review/runs/{runId}\n/visual-review/settings\n/web\n/web-scripts\n/web-scripts/new\n/web/agents\n/web/bots\n/web/content-autopilot\n/web/health\n/web/live\n/web/marketing\n/web/page-performance\n/web/page-reports\n/web/recap\n/web/session-attribution-explorer\n/web/web-vitals\n/welcome\n/wizard/runs\n/workflows\n/workflows/library/messages/{id}\n/workflows/library/templates/new\n/workflows/library/templates/{id}\n/workflows/new/workflow\n/workflows/{id}/{tab}","type": "string"}},diff --git a/services/mcp/tests/unit/generate-app-url.test.ts b/services/mcp/tests/unit/generate-app-url.test.tsindex 59d78685e082..4edb7a849d9c 100644--- a/services/mcp/tests/unit/generate-app-url.test.ts+++ b/services/mcp/tests/unit/generate-app-url.test.ts@@ -59,6 +59,17 @@ describe('generate-app-url', () => {['/feature_flags/{id}', { id: '42' }, '/feature-flags/42'],['/feature-flags/{id}', { id: '42' }, '/feature-flags/42'],['/feature_flags', {}, '/feature-flags'],+ [+ '/ai-evals/evaluations/offline/experiments?tab=datasets',+ {},+ '/ai-evals/evaluations/offline/experiments?tab=datasets',+ ],+ ['/ai-evals/evaluations/offline/experiments', {}, '/ai-evals/evaluations/offline/experiments'],+ [+ '/ai-evals/evaluations/offline/datasets/{id}',+ { id: 'example-dataset' },+ '/ai-evals/evaluations/offline/datasets/example-dataset',+ ],])('builds the canonical app link for the template %s', async (url, params, path) => {const result = await generateAppUrlHandler(ctx, { url, params })expect(result.url).toBe(`https://us.posthog.com/project/354703${path}`)