Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(aio): add tags to datasets, experiments, and scorers (#118683)

Radu Raiceae1e36a7e10c6
diff --git a/docs/internal/ai-offline-evaluation-reporting.md b/docs/internal/ai-offline-evaluation-reporting.md
index c5c3002b98c0..4f13ce74d02c 100644
--- a/docs/internal/ai-offline-evaluation-reporting.md
+++ b/docs/internal/ai-offline-evaluation-reporting.md
@@ -278,6 +278,30 @@ Large payloads have separate storage and 30-day deadlines anchored to first acce
Retries neither extend deadlines nor restore deleted payloads.
Automatic payload deletion and usage billing are not enabled by these endpoints.
+## Resource tags
+
+Datasets, stable scorer definitions, and offline experiments use the shared project tag vocabulary.
+Tags belong to the resource, not to dataset revisions or scorer versions, and do not propagate between linked resources.
+Their list and detail responses include `tags`. Dataset and scorer creation and updates accept a `tags` array.
+Use `PATCH /api/projects/{project_id}/ai_observability/offline_experiments/{id}/` with `{"tags": ["quality", "production"]}` to replace experiment tags; it requires evaluation editor access and `evaluation:write` for personal API keys. Ingestion credentials cannot edit tags.
+Omitting `tags` preserves existing assignments, including an empty experiment PATCH body; sending `[]` clears them. Names are trimmed and lowercased, with up to 100 tags of at most 255 characters each after normalization.
+Saving unchanged scorer tags preserves the update timestamp and does not emit a scorer update event.
+Removing the last assignment of a tag removes it from the shared project tag vocabulary.
+
+All three list APIs accept `tags` as a JSON-encoded array of up to 20 names and `tag_match=any` (OR, the default) or `tag_match=all` (AND).
+For example, `?tags=%5B%22quality%22%2C%22production%22%5D&tag_match=all` finds resources with both tags, including resources with additional tags.
+Filters apply before pagination; an empty array disables tag filtering. MCP list tools expose the same parameters, and update tools can replace tags.
+Assignments use the shared generic tagged-item relation. List queries use correlated existence checks and batch tag loading, so multiple matching tags do not duplicate rows or add a query per resource.
+Each save inserts normalized tag names in sorted order to prevent deadlocks when concurrent workers create the same tags.
+
+The three tables have a Tags column before the timestamp columns, showing up to two tags with remaining tags in an overflow popover and a control to edit assignments.
+Tag filters are included in the URL and remembered separately per resource type, project, and user in browser storage. Changing tags or their match mode resets table pagination.
+Explicit URL filters take precedence over remembered filters, including `tags=[]` to clear them.
+On the offline overview, the top-level tag selector filters both the recent experiments table and score trend charts.
+Scorer history supports the same tag filters through its UI, API, and MCP tool, including comparison periods. Its tables show experiment tags without editing controls.
+Experiment views share remembered tag filters, while explicit URL filters take precedence.
+Opening scorer history from an experiment clears remembered tag filters so they cannot hide that experiment.
+
## Postgres experiment reads
Read endpoints use the same feature flag as ingestion.
diff --git a/frontend/snapshots.yml b/frontend/snapshots.yml
index 9a9d3e084c7c..f589d938ea16 100644
--- a/frontend/snapshots.yml
+++ b/frontend/snapshots.yml
@@ -113,17 +113,17 @@ snapshots:
ai-observability-offline-experiments-score-trends--numeric-passing-rule--light:
hash: v1.k794b7964.e5367e5503e0e7a14586290abedd1f8f25d9a938013d330a9b3783a9377e32cf.2SZhg6Ri2BlDEHtHrLx7RAbuPEUQFdP-D1FP3_tOPNI
ai-observability-offline-experiments-scorer-history--default--dark:
- hash: v1.k794b7964.c816b9439c7de3e2dc8dbdf53b7c0651a3336707e1c8761bb5af0033904ea804.Gs0zlg6uRuw_0gaJi4rXDpWigJ8eYcWlILeaZ6Vf1OM
+ hash: v1.k794b7964.14c39caa03b5f3200fc4a373483c9d6aec17633da886479913ee3e645f8b0faf.bPbuMAy_65PZ_O4LdmGd2UGMiGwxeMdSrTKmiUYbNxo
ai-observability-offline-experiments-scorer-history--default--light:
- hash: v1.k794b7964.4e537160d18eacf43f87d4b3b9c2ec4ef78cfbc90c34885ad9b6d1f0c1c1b917.oIGbUXPQG4hu1-CMEXy7DhYMOkNq-QEG3atRit7UEFI
+ hash: v1.k794b7964.35b64cad1671eff7ac28a2b3b882662a5419f228124694de71b6851b6010200e.LVKYx94Kh5oZ3iSPLEt4rWdNXqZ66_013u3uO7wskbk
ai-observability-offline-experiments-scorer-history--empty-history--dark:
- hash: v1.k794b7964.90a96ba426f8f452cd947b1e985db2e193ac035767c7c2fcedb55fa53e4ed631.fVY6egjOgoHGNegbNwQ6RKEXoVz3kYwP6nlkxDVgzk0
+ hash: v1.k794b7964.631e591fcd1fc7328ea05d5a593b56a2fbbe62f1867bc411f68bf41065b4a93b.WDNsJdmbscjNrENw9efsmZyEaxzTI57uMxtS7npbkZA
ai-observability-offline-experiments-scorer-history--empty-history--light:
- hash: v1.k794b7964.19bdb239c205431da7017633b52d6d2c8829f4772316cfe267836d82cd4a9901.2MqfY2PaAGOKab9dEghItbVrBQXVAbVJgSph757S144
+ hash: v1.k794b7964.c48d7913e25042e743bfed5f6600e90ecee8f3bea019bc3598d59d0c064976cd.Lr6rJSUjois9B9eB8NV6uXAW-YexL2k1akLNs-NNFeQ
ai-observability-offline-experiments-scorer-history--narrow--dark:
- hash: v1.k794b7964.98526d40673a0779b784bd7845a2446fc970b299ba2de327e321481079d5f7da.3zf-H0cmGrmEj8yHtdglR_meMHnbEtNpLVc7OaLCoGg
+ hash: v1.k794b7964.3070c141e9ae22be387cf373c5009f759190aebb0351373a087a802dbf4dc55d.p9b0tC3ZgtzpMP0PA9wK3VUrKeWwYbbqiiho2JhKG3g
ai-observability-offline-experiments-scorer-history--narrow--light:
- hash: v1.k794b7964.af8bef523d5117ff2d93ab7a27e8e93bf3a5b9fb905151e8c80f0bb1b2a226bd.w6xWigQoWCxRYBqldCxfIaom49tZxgBy0dWTLb0tWvU
+ hash: v1.k794b7964.9d4a48ec4eff9ac47c121596e3a06b49e0034137db5a9605d8f0ca4fefcbdb6b.--7F2o6VAfVCzDcKJZX49I9X3KQtFmtVVq5j4XhSJmg
ai-observability-scorer-editor--boolean-detector--dark:
hash: v1.k794b7964.841ab07e56ee68448ba1df160d2365beed4eec6db48856b984488836ad4ce85f.91iF4r5RJZ81eQDCcoFWZ5N1JTo-dznWBBKCzTSJ-mM
ai-observability-scorer-editor--boolean-detector--light:
@@ -177,21 +177,21 @@ snapshots:
ai-observability-scorer-form--offline-disabled--light:
hash: v1.k794b7964.2dc03d1322685e9016f63454a0a2966ae96614bf08802aa725ebc955af65feaf.J8cmghXsSZoVxurLs2npYwulIcvOVgODD2EKNauo1F4
ai-observability-scorers--default--dark:
- hash: v1.k794b7964.040c57a323aca5a3d73ef242bf808c27b6072bfbf504390cb3fd158cf7b0443c.a0G1txCKyiAVTJfCzuRZWBlu7zGDQDM6AXuKQHsJYh8
+ hash: v1.k794b7964.f0ba08e3e9ad197df10bcf2dce7b2853bfd60117933d18d0ffaca306ce5df629.RAqptUlrl4tbYQ0XqDSI8CCuOVcb4_1vbasWnew--wU
ai-observability-scorers--default--light:
- hash: v1.k794b7964.1c5908b6e4e82c32be6d133efd4f4747d6c9c0c78145aa49d752d6a50e709702.JBt_3emY_nSHAB0X1bt4u_2kKsWQUnovbJi1Vvl-QgQ
+ hash: v1.k794b7964.9c497d39b156b13d881c197fe14a03525b9037cde4a921c03eda4c36eb2a5e11.iw-mCPWxoJ6hSDzHLlqtBAHNctGHmxr1v5MaKifwcrA
ai-observability-scorers--empty--dark:
- hash: v1.k794b7964.2860d5c1a83d91fe88603ee17204189fccf1f48efd35d08186986b91677c16f3.wkgzjnsIXC2EZzhCNPeGz7j5MseLo8wP3XmTL0hXn3c
+ hash: v1.k794b7964.5b5af62b9fe9b1659a678b10ddfdd76f593516514eb704bb872139c6cfae3f4a.lmMiFwPSa3CHpm8QT5jDt1GeJNguXoYW0-LYpYIKriQ
ai-observability-scorers--empty--light:
- hash: v1.k794b7964.1af68c0227124111213d1e0d5b60b6eafe6bdcffc16d891599bf0cb9a7c32122.cmpkQzlbK2YMLqY1q-89sd4Qa3392L_LBESD3U5ACAc
+ hash: v1.k794b7964.92cdcc6bf06c193e17213f5040145ba82bab8f782eea57a1a697968978b5c1f8.xHPj4oGYZZZ2NaKZeSMmEcag3UJ_jef1pmnwhTv6XRY
ai-observability-scorers--narrow--dark:
- hash: v1.k794b7964.d9c30aaf6f590eba612fd176ddb79ee2c3f81b33a57ec86c0d29a66fd7297f65.GNZSe0MA11ZyqkfgTHud4dR7l-ypDeq6_onULYtlozQ
+ hash: v1.k794b7964.173a1de2de884d9bd6ead01bd2af5dfd9d0ddda02e80309cc75604d630bafb1c.HCFNIirledijJvxMlusCruykulTTYYYw2PV1fqJIM4E
ai-observability-scorers--narrow--light:
- hash: v1.k794b7964.b6b57f6dd3c5a28ed20d4571a6de7674496b045f2b6acc1436e4bba01d4e1ac7.UAe8hwHdo_tgQVjdJt-GyEgszlJSZSbMZ6lenJvbIO4
+ hash: v1.k794b7964.bb73fe202ff0046137d66db340be06b67dd7ce791912735b42bd73f237e68b96.WBBZKIWNOarewm8SehIR2kLnSnLPlVhgW9mAAIZxlFg
ai-observability-scorers--offline-disabled--dark:
- hash: v1.k794b7964.086485b762cfc1047f790eb9be1bf2d32411d88a9abb0aaa65eb9832d4460803.5sMKoH-Saf_fJOsoEBC62DjNoqB6-Pbp3yWlkrJe9wI
+ hash: v1.k794b7964.753de81720b4496f631f3947e7b1df6e5fdc75d819a5d564a1bd03b304e886f8.u9drGvsNdBGawHS3L5p2aJwOGsBOqWLHEa3itmFFwqM
ai-observability-scorers--offline-disabled--light:
- hash: v1.k794b7964.2180bcbde737821240afcedee37c293417955b45ef6f03d990c8f5aa51933010.vALi9Nmw5uF4a15geZgWDSMnFrMnrx-01cJaq6zCuJ8
+ hash: v1.k794b7964.61fbfcd2fefffbd0fce01f693e9463fc690e4f0f69bd9308eafc06de07591cfe.-isZUttu_1l77UgFFJOZVuz96QiwJ-vUlgsyPWzmk5s
charts-heatmap--cell-labels--dark:
hash: v1.k794b7964.19b11a43aa9280e0b85498c1e87397d128cb1a1c9f1451419ab32de324783441.xEjoCIC6MhETYkz5iStffBD3Oc77sUugobUIKzq3EME
charts-heatmap--cell-labels--light:
@@ -8711,13 +8711,13 @@ snapshots:
scenes-app-ai-observability-offline-experiments-experiment--request-failure--light:
hash: v1.k794b7964.a31d57c72b94117a24a7a8948a68c6959b4fd00cde6d24aa3cdd206e1ca12eb7.392D8AhKK3JzKggndN4ojaEVt_vsHWraj2vVAWxz5NI
scenes-app-ai-observability-offline-experiments-overview--all-time-synchronized-hover--dark:
- hash: v1.k794b7964.92f06a25cb40491b184d27fee0afc40fc2915d0e528bf5ba04545bd055fb9e21.zKPFHsgLXM4_6Wm2MneUKlP9VCq9SgLs4rfVKtPsmKw
+ hash: v1.k794b7964.537d45d84c3e236f478f812a16f8bf69a789c1921de4058d73e0f30691e2f8ce.eiNwt87jkPxYQ60DjZRwZWcURl1KdwsWshBFzypU3DQ
scenes-app-ai-observability-offline-experiments-overview--all-time-synchronized-hover--light:
- hash: v1.k794b7964.8757d6babacf8e430dd08287fa04166d68213a90a0eb7e4748ce592c690a71ed.5Idjj-crX1d8J5Ttk6W50nDtwJSp2DerTerhHJo8tbY
+ hash: v1.k794b7964.4620ebeb5092fbecf405a12ad9c237118e143866a52af9672bceb4a890de9363.wXZJpitG_Zu-5btxfY3hlqszSqmUh1MBdnbU3vazCU0
scenes-app-ai-observability-offline-experiments-overview--default-scorers--dark:
- hash: v1.k794b7964.ee63dbc8b0d27c8084ff93dfafec4aab4ce01024877bae9433583c6577b34724.5PqZDtVDR6Y5Gl8MTCL8hz8PTzJ9KQh9Sjm736u1Y8o
+ hash: v1.k794b7964.d2b216a282f9e9080173796c66573bc2886a78a32164c84229591f1fe2efd079.xHaO6iiDcEjfU6C7G8s8GzNPvsHs_HFVbT7aDAhiMTM
scenes-app-ai-observability-offline-experiments-overview--default-scorers--light:
- hash: v1.k794b7964.b6b01f620dec845e18cc92f29499ce8387693ad36be054815bc8c6ef4b3c97c4.jyFHo8suG9CUGwZ32OEqLDMLwdTYdNfxAx8CJOb0ZII
+ hash: v1.k794b7964.2ab5f3d19ad8a33efc493309db8bc355fe206573cf389da46f62d44857157615.CsLy6stBpE-MKi0kKep0kv6d1Dm-g0JTr1__zIFIpEo
scenes-app-ai-observability-offline-experiments-overview--empty--dark:
hash: v1.k794b7964.977ade824abb5791ed84c5b6d125bc9589d0787b69daea3efeb6d0fe91f45f70.czJW_b0HTFrXWZGbmMZfAs1UD-NkCfVNLy36XZiTH7Y
scenes-app-ai-observability-offline-experiments-overview--empty--light:
@@ -8727,53 +8727,53 @@ snapshots:
scenes-app-ai-observability-offline-experiments-overview--empty-narrow--light:
hash: v1.k794b7964.754079aa9ab7ee416590c9bc8f94fa93f59b7769e98d4e63400883bf4d23ffae.hLRMtJxEnlm36ceer7qs_5s0s0euWyZB0BAqWPZz8c0
scenes-app-ai-observability-offline-experiments-overview--filtered-trends--dark:
- hash: v1.k794b7964.9fdf5042009d4bbae22d40350dced0b1327691d6248e4819675b7a98472bff21.ZWre7Qs9GGu9PcvZQ9oM0acYc-4FRX7E_G5xD4NuYtA
+ hash: v1.k794b7964.c48c8dbfe933d798ed7656446d7b1f8ab4b4346e5cbf54da0e4a17a911f76fe8.bcjQGTQZxdtyWGx-oz_RUJL5AMT1z5zGlTt6t38LVz4
scenes-app-ai-observability-offline-experiments-overview--filtered-trends--light:
- hash: v1.k794b7964.e1fc7f7f745aa51c00658c280d976dc43a025cab66047e863420a12078ffaef5.aeD9daJ8qZwsazp4axiLYwZbot3aBeRtJG-6AVKGhlw
+ hash: v1.k794b7964.91e07330df23f6cd13c9ac3435a4ebecab6de6695249a85a4f4eed7652d8464e.L8vDCa3kiPqURuztA9LIVyJEJE_21NKGvyS92W2-Oek
scenes-app-ai-observability-offline-experiments-overview--list-error--dark:
hash: v1.k794b7964.5c3c9c05034641362b5481f98a2da78a9415a0fdc49c5dcf6a4dba369b690ff0.PaaBerh2sH4kpLWTzjnSFshCOpO-MoAZWNkKHcMpKe4
scenes-app-ai-observability-offline-experiments-overview--list-error--light:
hash: v1.k794b7964.313d0039a600be146f2f62147db1fdb4d1202ea2cec5a95ef96dca12ed58b5bb.7Tolzb0JdrUiGuzLnZcHzOb1JMM4T4AvlYSxaK6IBCs
scenes-app-ai-observability-offline-experiments-overview--multiple-versions--dark:
- hash: v1.k794b7964.707dcaf2b4c280e86c780d4d25ebe33a80dc8dd1001eb700fa872267959e1794.QloqqJZKRgf6Fapk4FzWUW0GN5pD_1kHyQq83ssvd0E
+ hash: v1.k794b7964.fb85e61c26c0ece457f33d1a3117d2e09be8de003d01a2d01309eb860479d383.TAB4EEjC3VRF7kHnOy53goKZacu2zBK_8zwvmyqdiQ8
scenes-app-ai-observability-offline-experiments-overview--multiple-versions--light:
- hash: v1.k794b7964.cfa522a15b5d22f644d50565e09e4d82608aeb5cf4c8d9b1ad9353a15d546797.AqqssDnJ4x6Yj8UrgQwQre64knWgONZ2qqn8pI6jn98
+ hash: v1.k794b7964.6587ac1425f5bc3ea004f29f79b3a444d58ba0396eda68fc58940ae8812539e9.Cmf4VY_wluLujzOsELEd9hiXSENOr24U8qqGxxNVeN4
scenes-app-ai-observability-offline-experiments-overview--multiple-versions-narrow--dark:
- hash: v1.k794b7964.80b2c606c2311f207052f68086d754183d178fa7563c7ec062da2e92d74b6ad2.WdDwFWHk_b3h1MAp5By5r_dDWg0vaABrUGRJlbZ1Qhs
+ hash: v1.k794b7964.996f76f875e6c66d3a49bfd534e9d2e550b835048f5deadfe4ec7bfd22c13b28.LCXzcB3nX7t0MZ098MBj1oe9nLmoIet9lW8poNiOKvI
scenes-app-ai-observability-offline-experiments-overview--multiple-versions-narrow--light:
- hash: v1.k794b7964.882f49bc6e7f051aa0058dbebd8054a3c9bec03aded0de0b37523c16afec8669.uLdpnV_0CSUwFU4n850od6M-_vy9-9trUPbYXsjZj0A
+ hash: v1.k794b7964.a9050bf7854de6150999342699258288cd98fc9abc18ad25b45387d6d4f0a5b0.XPQOu_DedPjxdDMV-xkdAtNkC_qa5tuOAD6y09FpxeM
scenes-app-ai-observability-offline-experiments-overview--narrow--dark:
- hash: v1.k794b7964.5b3d669178447f62d9afec6ac9b41feb5ba6a5c3dbb5fe8e4764ad5519360103.EYJ0U2DpeX5n33IHgAFmS3rRwvviWWy46sSH8_JJz-0
+ hash: v1.k794b7964.c6050b27f6f78d6f2aac415642e951fa9c9195502dff436729041be4e7e8e289.ruj_Xy6Y-uBpCZrHSSqGfZLYur_5qBWCGW6_KRLDjLU
scenes-app-ai-observability-offline-experiments-overview--narrow--light:
- hash: v1.k794b7964.50dfb823442d39234049daab3d16a32383393dbdf40aec0ebbc80e86c52e8afc.yK1xI2vx15ecjCVGMzAjzITQ_aOLjj0DBbcJk1QYL2o
+ hash: v1.k794b7964.c9ec88221ddf7507b9a7ed6ebde1240e82e7193e1672dd0865aea4c39c563712.cBgiZ6eyhkh3TXKg3ffGB5XRBmdKrho0IfdsR5zJzZM
scenes-app-ai-observability-offline-experiments-overview--no-matching-experiments--dark:
- hash: v1.k794b7964.ed9c69d2469f4935e24472b326aca74dd0587cf009ebc3f735c5c5d1f8aebc58.PBd6kunvt27jc5KNGTiCDN1j--9DMXwNAkT1leVPpRE
+ hash: v1.k794b7964.45e918a97afbe5e23e3140ba18f57bde3ae55c87c0553a4cdb4120df309733d8.fZ_g4XJOv7eTKp238ptSGkhqkTFQwh8vRglbeikiJdE
scenes-app-ai-observability-offline-experiments-overview--no-matching-experiments--light:
- hash: v1.k794b7964.43f9340ab47a3e1dd95385ea668c669b3c1d30831a29852bd1f9df53352ac204.sn0Iwtyz0Egiroy8pjaZWAjFHTk4B37xYNQsbmqU1s0
+ hash: v1.k794b7964.43ae51dc02eb5dee83bf66c48651dacc8f95a9868ec8900de21c887ba26ce307.wIeQH_SeBTNAdCnbN1JLIdCMWK1Z_6gOonWpcYsl3OI
scenes-app-ai-observability-offline-experiments-overview--partial-history--dark:
- hash: v1.k794b7964.336cc60c578ae22261f59ab862fac40f3a4cd5e98a73575eb32d0da83e28df91.8J8-T8RZ6qnFjQ7xgOm4hbhSQ3t94mH-5sPFT_nh0x0
+ hash: v1.k794b7964.de7a21cacfaa7c8f6e4c39913e7386f52e156ad9071100f7df62fa39203109ae.yMKHVcbs8fDFDAi-1vZW3XBMG9anwFBV62VcFG0kAaM
scenes-app-ai-observability-offline-experiments-overview--partial-history--light:
- hash: v1.k794b7964.b05de17a4fefb705c6462276808132444d4b4b5b3d68e9a778f040e29db6b595.29qN9vo2gGrGE4cey2Bb1SqSItdXAC6Eci1-P91MP1E
+ hash: v1.k794b7964.7e650acc2575be9815c4edece4b8399585e604d38e0934c771362ee08288aa23.meUmr6mF3RzVIByqn0hWFnrOCYpJuXBGoAFdGJfLRRM
scenes-app-ai-observability-offline-experiments-overview--partial-history-narrow--dark:
- hash: v1.k794b7964.c7b6a60ce676beaa13fb20dce5551fb5b374cb894aae7e07233d728c136b2cfd.cehwuLDOUC6wXsXXL1Alv0rBqduTfpYEuR9To6nLKd0
+ hash: v1.k794b7964.2a9a3a3311c12216bc99fd8bb08ba2a24b77f3d780ceddd4a501e4844ad89195.z5nROrbR9yDlG8llzDg18mzmIBGONgRXZRbH2nvwnxA
scenes-app-ai-observability-offline-experiments-overview--partial-history-narrow--light:
- hash: v1.k794b7964.73d8a66caeb291d286623491bf61429e46b3ffa49917b71480c4142de59fbb13.LTES5OwKaeAIGROLlqpZhWYZwMJAj1a0dYRO1LdEDIM
+ hash: v1.k794b7964.0170b29bab838a1ce5b000a9e50a5a614e336fa86d9e8806a40e7fe8511675dc.Fa48OOiFQhT8vdq6NsvdN62J9BjhT4B9FwIjbpJ3QUU
scenes-app-ai-observability-offline-experiments-overview--recent-experiments--dark:
- hash: v1.k794b7964.ee63dbc8b0d27c8084ff93dfafec4aab4ce01024877bae9433583c6577b34724.E6QgEhRXHbaZyoRJxWMMhAOGh00612-nFazNWy8MKiU
+ hash: v1.k794b7964.d2b216a282f9e9080173796c66573bc2886a78a32164c84229591f1fe2efd079.KmkKO4x2AmWgqPIgwDVBUt1GL26zNQeMCc0WS3ELdgg
scenes-app-ai-observability-offline-experiments-overview--recent-experiments--light:
- hash: v1.k794b7964.b6b01f620dec845e18cc92f29499ce8387693ad36be054815bc8c6ef4b3c97c4.iRdJ_Xc3oGA3aFBXMzDND_HhyGYAO2EHfEhlsGRn23U
+ hash: v1.k794b7964.2ab5f3d19ad8a33efc493309db8bc355fe206573cf389da46f62d44857157615.vrQKlZ3zgM8Epl2YShDovvuf8wCzHhhKelizMExl05o
scenes-app-ai-observability-offline-experiments-overview--score-error--dark:
- hash: v1.k794b7964.6a4ab392e3a9c61aa09a4d2ba700804bfc2a4ba68f0d3ed07a6aeda5a7597444.KnwGA0BS-xPs1Ygs5ms3tKdEAre7XuonWgDudR9BfSg
+ hash: v1.k794b7964.13e3cb16faa223aab3c1951c2e81b38dbe7e04b11991953b913395d5a22adef2.MOklDxh1x7jGu_YpgkjOr_qECmSkGx9pteGHPZV9U9Q
scenes-app-ai-observability-offline-experiments-overview--score-error--light:
- hash: v1.k794b7964.6e2a098cad6a7be399125ffa6bc75fde41f03d2b4580bea9b155e3c08fa4fd0f.DFDBkI_xNCpSdcaq7GKy95om_ZeTo6RPcecCQ5mMhEw
+ hash: v1.k794b7964.fd24de84c10d3bc761e22c92d795bb015276714562f62a65f81c023abe6f342c.Qld0U3x2dzcxfHKzSblh-0RWlOjh3CriMtMy2Ng5yho
scenes-app-ai-observability-offline-experiments-overview--synchronized-hover--dark:
- hash: v1.k794b7964.4e7e59cb53943db43d607476e5cf402622f4c208d2586ef11c7bec93fa8cc692.H71gAMQYhCsLXHFbRxl6T0_DjQHlDv5HzVS-I1l5ESg
+ hash: v1.k794b7964.95badff06a14f772cd75679650910d0bb7f5a6cd62684b3683c656087f9b3d4e.7-05dyZSFo3UadpiZ4X4hQ09zZqV4d_T_HiDdcdYy2w
scenes-app-ai-observability-offline-experiments-overview--synchronized-hover--light:
- hash: v1.k794b7964.e7722555def758dcfe89e5fe4faab28bcdaf2f255ca0166ead7e2a5e22621e36.3mQphMP7VB56T0FApQMk7wdOcXAmN4B8T5voa7guvo8
+ hash: v1.k794b7964.5c388b4942818fd3a11080eeb2928a171fbf97a322f441db747bbd8bde731c1f.OVp8z73MBocJrO08SdA1xrdOi9FQp85G5ZhRqmN4TwQ
scenes-app-ai-observability-offline-experiments-overview--synchronized-hover-narrow--dark:
- hash: v1.k794b7964.d86019a71240cb20abd2be302b86ab7644454cddb313d28e32f024bf16ea0a98.qOBSQo5Kpy9wuofEh1A2EzW3exYk2BwHgshxDvdyWAg
+ hash: v1.k794b7964.d65e4efc17a29e026a23fba3b352abf817a7f3eab02eef299d8d06e46b896461.JSoaS57anUs6pGLncJkgDqL85NX-YzCixG4BId2WZZk
scenes-app-ai-observability-offline-experiments-overview--synchronized-hover-narrow--light:
- hash: v1.k794b7964.d672d469c8481c6526936d0d84c58766a7e51fe6072b3e7b59953fa59eb6c6e8.ZHsxFC2MG9W76b-V6dSOzYEMhGYHT9FezLpgiYHH3VI
+ hash: v1.k794b7964.689c3e205264cc850fb0cd5adee4f67275ffd893f9f5634afa258900c3720323.1AOxrJW_lz6MsorLuPjy83lS7Lbxjm4_QaCoUaPpiXM
scenes-app-ai-observability-promptreferencetags--label-and-version-references--dark:
hash: v1.k794b7964.4cc699d7489063a64cf4ec1f930d11fc062eb0d9c9e46150f939503945bb8841.jKmH2ZmjZex98fImM00HE_xBBayalWKHtqXbVlhE_VM
scenes-app-ai-observability-promptreferencetags--label-and-version-references--light:
diff --git a/posthog/api/tagged_item.py b/posthog/api/tagged_item.py
index 8d7707c59e10..024fdebe8bc1 100644
--- a/posthog/api/tagged_item.py
+++ b/posthog/api/tagged_item.py
@@ -29,7 +29,8 @@ def set_tags_on_object(tags: list[str], obj: Any) -> list[TaggedItem]:
This is the core tag-setting logic extracted for reuse across serializers
and bulk operations.
"""
- deduped_tags = list({tagify(t) for t in tags})
+ # Concurrent transactions must insert shared tags in the same order to avoid deadlocks.
+ deduped_tags = sorted({tagify(t) for t in tags})
tagged_item_objects = []
for tag in deduped_tags:
diff --git a/posthog/api/test/test_tagged_item.py b/posthog/api/test/test_tagged_item.py
index 18c057dc8955..dbd61a42de33 100644
--- a/posthog/api/test/test_tagged_item.py
+++ b/posthog/api/test/test_tagged_item.py
@@ -2,6 +2,7 @@ from uuid import uuid4
from posthog.test.base import APIBaseTest
+from django.db.models.signals import post_save
from django.test import SimpleTestCase
from parameterized import parameterized
@@ -33,15 +34,25 @@ class TestTaggedItemSerializerMixin(APIBaseTest):
self.assertEqual(response.json()["tags"], ["random"])
self.assertEqual(Tag.objects.all().count(), 1)
- def test_create_with_tags(self):
+ def test_create_with_tags(self) -> None:
+ created_tag_names: list[str] = []
+
+ def record_tag_creation(instance: Tag, created: bool, **kwargs: object) -> None:
+ if created:
+ created_tag_names.append(instance.name)
+
+ post_save.connect(record_tag_creation, sender=Tag)
+ self.addCleanup(post_save.disconnect, record_tag_creation, sender=Tag)
+
response = self.client.post(
f"/api/projects/{self.team.id}/dashboards/",
- {"name": "Default", "pinned": "true", "tags": ["random", "hello"]},
+ {"name": "Default", "pinned": "true", "tags": [" Random ", "hello", "random"]},
)
self.assertEqual(response.status_code, status.HTTP_201_CREATED)
self.assertEqual(sorted(response.json()["tags"]), ["hello", "random"])
self.assertEqual(Tag.objects.all().count(), 2)
+ self.assertEqual(created_tag_names, ["hello", "random"])
def test_update_tags(self):
dashboard = Dashboard.objects.create(team_id=self.team.id, name="private dashboard")
diff --git a/posthog/models/tagged_item_registry.py b/posthog/models/tagged_item_registry.py
index 869a23c9b03b..6da61d5d84c8 100644
--- a/posthog/models/tagged_item_registry.py
+++ b/posthog/models/tagged_item_registry.py
@@ -49,6 +49,11 @@ class TaggableModel:
TAGGABLE_MODELS: tuple[TaggableModel, ...] = (
+ TaggableModel(model_label="ai_observability.Dataset", legacy_field="ai_dataset", object_field=OBJECT_UUID),
+ TaggableModel(model_label="ai_observability.ScoreDefinition", legacy_field="ai_scorer", object_field=OBJECT_UUID),
+ TaggableModel(
+ model_label="ai_observability.OfflineExperiment", legacy_field="offline_experiment", object_field=OBJECT_UUID
+ ),
TaggableModel(model_label="dashboards.Dashboard", legacy_field="dashboard", object_field=OBJECT_ID),
TaggableModel(model_label="product_analytics.Insight", legacy_field="insight", object_field=OBJECT_ID),
TaggableModel(
diff --git a/products/ai_observability/backend/api/datasets.py b/products/ai_observability/backend/api/datasets.py
index 0d3bda578ab3..ca76ab4c2beb 100644
--- a/products/ai_observability/backend/api/datasets.py
+++ b/products/ai_observability/backend/api/datasets.py
@@ -1,6 +1,7 @@
from typing import NoReturn, cast
from uuid import UUID
+from django.db import transaction
from django.db.models import F, OuterRef, Q, QuerySet, Subquery
from django.db.models.functions import Coalesce
from django.http import Http404
@@ -49,6 +50,11 @@ from products.ai_observability.backend.api.dataset_exports import (
)
from products.ai_observability.backend.api.dataset_serializers import StrictDatasetSerializer
from products.ai_observability.backend.api.metrics import llma_track_latency
+from products.ai_observability.backend.api.resource_tags import (
+ ResourceTagFilterSerializer,
+ ResourceTagsReadSerializer,
+ ResourceTagsWriteSerializer,
+)
from products.ai_observability.backend.dataset_queries import (
consistent_dataset_item_versions,
dataset_item_versions_at_revision,
@@ -69,6 +75,7 @@ from products.ai_observability.backend.dataset_service import (
update_dataset_item,
)
from products.ai_observability.backend.models.datasets import Dataset, DatasetItemVersion, DatasetRevision
+from products.ai_observability.backend.resource_tags import ResourceTags
from products.exports.backend.facade.api import get_export_asset_content_response
@@ -186,7 +193,7 @@ def _resolved_dataset_revision_id(dataset: Dataset) -> UUID | None:
return dataset.current_revision_id
-class DatasetReadSerializer(UserAccessControlSerializerMixin, serializers.ModelSerializer):
+class DatasetReadSerializer(ResourceTagsReadSerializer, UserAccessControlSerializerMixin, serializers.ModelSerializer):
created_by = UserBasicSerializer(read_only=True, allow_null=True)
metadata = JSONObjectField(read_only=True, help_text="JSON object with descriptive dataset metadata.")
current_revision = serializers.SerializerMethodField(
@@ -204,6 +211,7 @@ class DatasetReadSerializer(UserAccessControlSerializerMixin, serializers.ModelS
"name",
"description",
"metadata",
+ "tags",
"archived",
"current_revision",
"current_revision_id",
@@ -224,7 +232,7 @@ class DatasetReadSerializer(UserAccessControlSerializerMixin, serializers.ModelS
return _resolved_dataset_revision_id(instance)
-class DatasetCreateSerializer(DatasetMutationSerializer):
+class DatasetCreateSerializer(ResourceTagsWriteSerializer, DatasetMutationSerializer):
name = serializers.CharField(
max_length=400,
trim_whitespace=True,
@@ -244,7 +252,7 @@ class DatasetCreateSerializer(DatasetMutationSerializer):
)
-class DatasetUpdateSerializer(DatasetMutationSerializer):
+class DatasetUpdateSerializer(ResourceTagsWriteSerializer, DatasetMutationSerializer):
name = serializers.CharField(
required=False,
max_length=400,
@@ -263,7 +271,7 @@ class DatasetUpdateSerializer(DatasetMutationSerializer):
)
-class DatasetListQuerySerializer(serializers.Serializer):
+class DatasetListQuerySerializer(ResourceTagFilterSerializer):
id__in = RepeatedOrCommaSeparatedListField(
child=serializers.UUIDField(),
required=False,
@@ -597,6 +605,7 @@ def _dataset_queryset() -> QuerySet[Dataset, Dataset]:
return (
Dataset.objects.unscoped()
.select_related("created_by", "current_revision")
+ .prefetch_related(ResourceTags.prefetch())
.annotate(
resolved_current_revision=Coalesce(
F("current_revision__revision"),
@@ -694,6 +703,7 @@ class DatasetViewSet(TeamAndOrgViewSetMixin, AccessControlViewSetMixin, GenericV
queryset = queryset.filter(
Q(name__icontains=search) | Q(description__icontains=search) | Q(metadata__icontains=search)
)
+ queryset = ResourceTags.filter(queryset, self.team_id, query.get("tags", []), query["tag_match"])
queryset = queryset.order_by(query["order_by"], "id")
page = self.paginate_queryset(queryset)
@@ -729,13 +739,16 @@ class DatasetViewSet(TeamAndOrgViewSetMixin, AccessControlViewSetMixin, GenericV
serializer.is_valid(raise_exception=True)
data = serializer.validated_data
try:
- dataset = create_dataset(
- team=self.team,
- created_by=cast(User, request.user),
- name=data["name"],
- description=data["description"],
- metadata=data["metadata"],
- )
+ with transaction.atomic():
+ dataset = create_dataset(
+ team=self.team,
+ created_by=cast(User, request.user),
+ name=data["name"],
+ description=data["description"],
+ metadata=data["metadata"],
+ )
+ if "tags" in data:
+ ResourceTags.set(dataset, data["tags"])
except DatasetValidationError as error:
_validation_error_response(error)
except (DatasetMutationConflict, DatasetLimitExceeded) as error:
@@ -774,13 +787,16 @@ class DatasetViewSet(TeamAndOrgViewSetMixin, AccessControlViewSetMixin, GenericV
serializer.is_valid(raise_exception=True)
data = serializer.validated_data
try:
- dataset = update_dataset(
- team_id=self.team.id,
- dataset_id=current.id,
- name=data.get("name", UNSET),
- description=data.get("description", UNSET),
- metadata=data.get("metadata", UNSET),
- )
+ with transaction.atomic():
+ dataset = update_dataset(
+ team_id=self.team.id,
+ dataset_id=current.id,
+ name=data.get("name", UNSET),
+ description=data.get("description", UNSET),
+ metadata=data.get("metadata", UNSET),
+ )
+ if "tags" in data:
+ ResourceTags.set(dataset, data["tags"])
except DatasetValidationError as error:
_validation_error_response(error)
except DatasetMutationConflict as error:
diff --git a/products/ai_observability/backend/api/offline_experiment_read_serializers.py b/products/ai_observability/backend/api/offline_experiment_read_serializers.py
index dc8836d34792..034992a37a5c 100644
--- a/products/ai_observability/backend/api/offline_experiment_read_serializers.py
+++ b/products/ai_observability/backend/api/offline_experiment_read_serializers.py
@@ -12,6 +12,7 @@ from products.ai_observability.backend.api.offline_experiment_serializers import
ResultValueField,
)
from products.ai_observability.backend.api.query import validate_query_parameters
+from products.ai_observability.backend.api.resource_tags import ResourceTagFilterField
from products.ai_observability.backend.models.offline_evaluations import (
OfflineEvaluationResult,
OfflineExperiment,
@@ -25,6 +26,7 @@ from products.ai_observability.backend.offline_evaluation_read_types import (
OfflineResultCellQuery,
)
from products.ai_observability.backend.offline_evaluation_service import OfflineEvaluationValidationError
+from products.ai_observability.backend.resource_tags import TagMatch
from products.ai_observability.backend.score_definition_configs import ScoreDefinitionConfigField
if TYPE_CHECKING:
@@ -108,6 +110,14 @@ class OfflineResultCellQuerySerializer(_OfflineQuerySerializer[OfflineResultCell
class OfflineHistoryQuerySerializer(OfflineResultQuerySerializer):
+ tags = ResourceTagFilterField()
+ tag_match = serializers.ChoiceField(
+ choices=TagMatch.choices,
+ required=False,
+ default=TagMatch.ANY,
+ help_text="Match any selected tag (OR) or all selected tags (AND).",
+ )
+
date_from = serializers.DateTimeField(
required=False, help_text="Inclusive execution start time, in ISO 8601 format."
)
@@ -146,6 +156,8 @@ class OfflineHistoryQuerySerializer(OfflineResultQuerySerializer):
class Meta(OfflineResultQuerySerializer.Meta):
fields = [
*OfflineResultQuerySerializer.Meta.fields,
+ "tags",
+ "tag_match",
"date_from",
"date_to",
"search",
@@ -160,6 +172,9 @@ class OfflineHistoryQuerySerializer(OfflineResultQuerySerializer):
"prompt_version",
]
+ def validate_tags(self, value: list[str]) -> tuple[str, ...]:
+ return tuple(value)
+
def validate_statuses(self, value: str) -> tuple[str, ...]:
values = tuple(value.split(","))
if len(set(values)) != len(values) or any(state not in OfflineExperiment.Status.values for state in values):
@@ -187,6 +202,9 @@ class OfflineExperimentQuerySerializer(OfflineHistoryQuerySerializer):
class OfflineExperimentReadSerializer(serializers.Serializer):
+ tags = serializers.ListField(
+ child=serializers.CharField(), help_text="Project tag names assigned to this experiment."
+ )
id = serializers.UUIDField(help_text="Stable experiment UUID.")
name = serializers.CharField(help_text="Experiment name.")
run_source = serializers.ChoiceField(
diff --git a/products/ai_observability/backend/api/offline_experiments.py b/products/ai_observability/backend/api/offline_experiments.py
index 202f2c97244d..f5bb618724e7 100644
--- a/products/ai_observability/backend/api/offline_experiments.py
+++ b/products/ai_observability/backend/api/offline_experiments.py
@@ -2,6 +2,8 @@ from dataclasses import asdict
from typing import TYPE_CHECKING, cast
from uuid import UUID
+from django.shortcuts import get_object_or_404
+
from drf_spectacular.utils import OpenApiParameter
from prometheus_client import Counter
from rest_framework import serializers, viewsets
@@ -11,7 +13,7 @@ from rest_framework.request import Request
from rest_framework.response import Response
from rest_framework.throttling import BaseThrottle
-from posthog.api.mixins import TypedRequest, validated_request
+from posthog.api.mixins import TypedRequest, ValidatedRequest, validated_request
from posthog.api.monitoring import monitor
from posthog.permissions import is_service_auth
@@ -25,11 +27,16 @@ from products.ai_observability.backend.api.offline_experiment_access import (
)
from products.ai_observability.backend.api.offline_experiment_errors import OfflineEvaluationErrorSerializer
from products.ai_observability.backend.api.offline_experiment_parser import OfflineEvaluationJSONParser
-from products.ai_observability.backend.api.offline_experiment_reads import OfflineExperimentReadViewSet
+from products.ai_observability.backend.api.offline_experiment_read_serializers import OfflineExperimentReadSerializer
+from products.ai_observability.backend.api.offline_experiment_reads import (
+ READ_ERROR_RESPONSES,
+ OfflineExperimentReadViewSet,
+)
from products.ai_observability.backend.api.offline_experiment_serializers import (
ExperimentSubmissionSerializer,
UploadSubmissionSerializer,
)
+from products.ai_observability.backend.api.resource_tags import ResourceTagNamesField
from products.ai_observability.backend.models.offline_evaluations import OfflineExperiment
from products.ai_observability.backend.offline_evaluation_service import (
ExperimentReceipt,
@@ -39,6 +46,7 @@ from products.ai_observability.backend.offline_evaluation_service import (
OfflineExperimentService,
)
from products.ai_observability.backend.offline_evaluation_types import ExperimentSubmission, UploadSubmission
+from products.ai_observability.backend.resource_tags import ResourceTags
if TYPE_CHECKING:
from products.access_control.backend.facade.user_access_control import UserAccessControl
@@ -113,8 +121,12 @@ ERROR_RESPONSES = dict.fromkeys((400, 401, 403, 404, 409, 413, 429), OfflineEval
EXPERIMENT_ID_PARAMETER = OpenApiParameter("id", UUID, OpenApiParameter.PATH, description="Experiment UUID.")
+class OfflineExperimentMetadataSerializer(serializers.Serializer):
+ tags = ResourceTagNamesField(required=False)
+
+
class OfflineExperimentViewSet(OfflineExperimentReadViewSet):
- scope_object_write_actions = ["create", "upload", "complete", "fail"]
+ scope_object_write_actions = ["create", "upload", "complete", "fail", "partial_update"]
psak_allowed_actions = ["create", "upload", "complete", "fail"]
parser_classes = [OfflineEvaluationJSONParser]
ingestion_throttle_classes = [
@@ -124,15 +136,17 @@ class OfflineExperimentViewSet(OfflineExperimentReadViewSet):
OfflineEvaluationIngestionTeamSustainedThrottle,
]
serializer_class = ExperimentSubmissionSerializer
- http_method_names = ["get", "post", "head", "options"]
+ http_method_names = ["get", "post", "patch", "head", "options"]
def dangerously_get_required_scopes(self, request: Request, view: viewsets.ViewSetMixin) -> list[str] | None:
+ if self.action == "partial_update":
+ return ["evaluation:write"]
if self.action in self.scope_object_write_actions:
return ["offline_evaluation_ingestion:write"]
return super().dangerously_get_required_scopes(request, view)
def get_throttles(self) -> list[BaseThrottle]:
- if self.action in self.scope_object_write_actions:
+ if self.action in self.psak_allowed_actions:
return [throttle() for throttle in self.ingestion_throttle_classes]
return super().get_throttles()
@@ -141,7 +155,7 @@ class OfflineExperimentViewSet(OfflineExperimentReadViewSet):
def handle_exception(self, exc: Exception) -> Response:
action_name = getattr(self, "action", None)
- if action_name in self.scope_object_read_actions or action_name == "metadata":
+ if action_name in self.scope_object_read_actions or action_name in {"metadata", "partial_update"}:
return super().handle_exception(exc)
if isinstance(exc, OfflineEvaluationConflict):
OFFLINE_UPLOAD_ERRORS.labels(outcome="conflict").inc()
@@ -162,6 +176,20 @@ class OfflineExperimentViewSet(OfflineExperimentReadViewSet):
OFFLINE_UPLOAD_ERRORS.labels(outcome="rejected" if response.status_code < 500 else "server_error").inc()
return response
+ @validated_request(
+ request_serializer=OfflineExperimentMetadataSerializer,
+ responses={200: OfflineExperimentReadSerializer, **ERROR_RESPONSES, **READ_ERROR_RESPONSES},
+ parameters=[EXPERIMENT_ID_PARAMETER],
+ )
+ def partial_update(self, request: ValidatedRequest, **kwargs: object) -> Response:
+ experiment = get_object_or_404(OfflineExperiment.objects.for_team(self.team_id), pk=self._path_id())
+ self.check_object_permissions(request, experiment)
+ if "tags" in request.validated_data:
+ ResourceTags.set(experiment, request.validated_data["tags"])
+ return self._read_response(
+ lambda: self._read_service().get_experiment(experiment.id), OfflineExperimentReadSerializer
+ )
+
@validated_request(
ExperimentSubmissionSerializer,
responses={200: ExperimentReceiptSerializer, 201: ExperimentReceiptSerializer, **ERROR_RESPONSES},
diff --git a/products/ai_observability/backend/api/resource_tags.py b/products/ai_observability/backend/api/resource_tags.py
new file mode 100644
index 000000000000..e0db913e1897
--- /dev/null
+++ b/products/ai_observability/backend/api/resource_tags.py
@@ -0,0 +1,75 @@
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, cast
+
+from drf_spectacular.utils import extend_schema_field
+from rest_framework import serializers
+
+from posthog.models.tag import tagify
+
+from products.ai_observability.backend.resource_tags import ResourceTags, TagMatch
+
+if TYPE_CHECKING:
+ from products.ai_observability.backend.resource_tags import TaggedResource
+
+MAX_RESOURCE_TAGS = 100
+MAX_FILTER_TAGS = 20
+
+
+class ResourceTagNameField(serializers.CharField):
+ def to_internal_value(self, data: str) -> str:
+ return tagify(super().to_internal_value(data))
+
+
+class ResourceTagNamesField(serializers.ListField):
+ child = ResourceTagNameField(max_length=255, allow_blank=False)
+
+ def __init__(self, *, required: bool = True, max_length: int = MAX_RESOURCE_TAGS) -> None:
+ super().__init__(
+ required=required,
+ max_length=max_length,
+ help_text="Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.",
+ )
+
+ def to_internal_value(self, data: object) -> list[str]:
+ return sorted(set(super().to_internal_value(cast(list[str], data))))
+
+
+class ResourceTagsWriteSerializer(serializers.Serializer):
+ tags = ResourceTagNamesField(required=False)
+
+
+class ResourceTagsReadSerializer(serializers.Serializer):
+ tags = serializers.SerializerMethodField(help_text="Project tag names assigned to this resource.")
+
+ @extend_schema_field(serializers.ListField(child=serializers.CharField()))
+ def get_tags(self, resource: TaggedResource) -> list[str]:
+ return ResourceTags.names(resource)
+
+
+@extend_schema_field(serializers.CharField(max_length=26000))
+class ResourceTagFilterField(serializers.Field):
+ def __init__(self) -> None:
+ super().__init__(
+ required=False,
+ help_text='JSON array of tag names, for example ["quality","production"]. Up to 20 tags. An empty array disables tag filtering.',
+ )
+
+ def to_internal_value(self, data: object) -> list[str]:
+ value = serializers.CharField(max_length=26000).run_validation(data)
+ try:
+ parsed = json.loads(value)
+ except (TypeError, ValueError) as error:
+ raise serializers.ValidationError("Supply a JSON array of tag names.") from error
+ return ResourceTagNamesField(max_length=MAX_FILTER_TAGS).run_validation(parsed)
+
+
+class ResourceTagFilterSerializer(serializers.Serializer):
+ tags = ResourceTagFilterField()
+ tag_match = serializers.ChoiceField(
+ choices=TagMatch.choices,
+ default=TagMatch.ANY,
+ required=False,
+ help_text="Match any selected tag (OR) or all selected tags (AND). Additional tags are allowed.",
+ )
diff --git a/products/ai_observability/backend/api/score_definitions.py b/products/ai_observability/backend/api/score_definitions.py
index 8f9905a27236..6f484c51154a 100644
--- a/products/ai_observability/backend/api/score_definitions.py
+++ b/products/ai_observability/backend/api/score_definitions.py
@@ -26,16 +26,22 @@ from posthog.permissions import AccessControlPermission
from products.access_control.backend.presentation.access_control import AccessControlViewSetMixin
from products.ai_observability.backend.api.query import EmptyQuerySerializer, StrictQuerySerializer
+from products.ai_observability.backend.api.resource_tags import (
+ ResourceTagFilterSerializer,
+ ResourceTagsReadSerializer,
+ ResourceTagsWriteSerializer,
+)
from products.ai_observability.backend.models.score_definitions import (
ScoreDefinition,
ScoreDefinitionVersion,
StaleScoreDefinitionVersion,
)
from products.ai_observability.backend.read_pagination import CursorPage, cursor_page, decode_cursor, encode_cursor
+from products.ai_observability.backend.resource_tags import ResourceTags
from products.ai_observability.backend.score_definition_configs import ScoreDefinitionConfigField
-class ScoreDefinitionSerializer(serializers.ModelSerializer):
+class ScoreDefinitionSerializer(ResourceTagsReadSerializer, serializers.ModelSerializer):
created_by = UserBasicSerializer(
read_only=True,
allow_null=True,
@@ -63,6 +69,7 @@ class ScoreDefinitionSerializer(serializers.ModelSerializer):
"id",
"name",
"description",
+ "tags",
"kind",
"archived",
"current_version",
@@ -134,7 +141,7 @@ def _score_definition_version_page(
)
-class ScoreDefinitionCreateSerializer(serializers.Serializer):
+class ScoreDefinitionCreateSerializer(ResourceTagsWriteSerializer):
name = serializers.CharField(max_length=255, help_text="Human-readable scorer name.")
description = serializers.CharField(
required=False,
@@ -165,7 +172,7 @@ class ScoreDefinitionCreateSerializer(serializers.Serializer):
return attrs
-class ScoreDefinitionMetadataSerializer(serializers.Serializer):
+class ScoreDefinitionMetadataSerializer(ResourceTagsWriteSerializer):
name = serializers.CharField(max_length=255, required=False, help_text="Updated scorer name.")
description = serializers.CharField(
required=False,
@@ -281,7 +288,13 @@ class ScoreDefinitionViewSet(
elif self.action in {"versions", "retrieve_version"}:
queryset = self.user_access_control.filter_queryset_by_access_level(queryset, include_all_if_admin=True)
- return queryset
+ if self.action == "list":
+ tag_query = ResourceTagFilterSerializer(data=self.request.query_params)
+ tag_query.is_valid(raise_exception=True)
+ queryset = ResourceTags.filter(
+ queryset, self.team_id, tag_query.validated_data.get("tags", []), tag_query.validated_data["tag_match"]
+ )
+ return queryset.prefetch_related(ResourceTags.prefetch())
@staticmethod
def _event_properties(definition: ScoreDefinition) -> dict[str, str | bool | int]:
@@ -300,6 +313,7 @@ class ScoreDefinitionViewSet(
def _create_definition(self, validated_data: dict[str, Any]) -> ScoreDefinition:
definition_data = dict(validated_data)
config = definition_data.pop("config")
+ tags = definition_data.pop("tags", [])
definition_data.pop("archived", None)
definition_data["description"] = definition_data.get("description") or ""
@@ -309,11 +323,18 @@ class ScoreDefinitionViewSet(
**definition_data,
)
definition.create_new_version(config=config, created_by=cast(User, self.request.user))
+ ResourceTags.set(definition, tags)
return definition
+ @transaction.atomic
def _update_definition_metadata(self, definition: ScoreDefinition, validated_data: dict[str, Any]) -> None:
definition_data = dict(validated_data)
changed_fields: list[str] = []
+ if "tags" in definition_data:
+ tags = definition_data.pop("tags")
+ if tags != ResourceTags.names(definition):
+ ResourceTags.set(definition, tags)
+ changed_fields.append("tags")
if "description" in definition_data:
definition_data["description"] = definition_data["description"] or ""
@@ -326,7 +347,7 @@ class ScoreDefinitionViewSet(
if not changed_fields:
return
- definition.save(update_fields=[*changed_fields, "updated_at"])
+ definition.save(update_fields=[*[field for field in changed_fields if field != "tags"], "updated_at"])
event_properties: dict[str, str | bool | int | list[str]] = {
**self._event_properties(definition),
"changed_fields": changed_fields,
@@ -362,6 +383,7 @@ class ScoreDefinitionViewSet(
@extend_schema(
parameters=[
+ ResourceTagFilterSerializer,
OpenApiParameter(
"search",
OpenApiTypes.STR,
diff --git a/products/ai_observability/backend/api/test/test_offline_experiment_reads.py b/products/ai_observability/backend/api/test/test_offline_experiment_reads.py
index 960ff90fae15..24daf54483ce 100644
--- a/products/ai_observability/backend/api/test/test_offline_experiment_reads.py
+++ b/products/ai_observability/backend/api/test/test_offline_experiment_reads.py
@@ -1,3 +1,4 @@
+import json
from datetime import timedelta
from uuid import uuid4
@@ -25,6 +26,7 @@ from products.ai_observability.backend.models.offline_evaluations import (
OfflineExperimentItemPayload,
)
from products.ai_observability.backend.models.score_definitions import ScoreDefinition
+from products.ai_observability.backend.resource_tags import ResourceTags
@pytest.mark.ee
@@ -111,6 +113,56 @@ class TestOfflineExperimentReads(APIBaseTest):
self._history(),
]
+ def test_history_filters_experiment_tags_before_pagination(self) -> None:
+ ResourceTags.set(self.experiment, ["quality", "production"])
+ for index, tags in enumerate((["quality"], []), start=1):
+ timestamp = self.experiment.started_at - timedelta(hours=index)
+ experiment = OfflineExperiment.objects.for_team(self.team.id).create(
+ id=uuid4(),
+ team=self.team,
+ name=f"Earlier run {index}",
+ started_at=timestamp,
+ created_at=timestamp,
+ finished_at=timestamp,
+ status="completed",
+ submission_fingerprint="a" * 64,
+ )
+ ResourceTags.set(experiment, tags)
+ item = OfflineExperimentItem.objects.for_team(self.team.id).create(
+ id=uuid4(),
+ team=self.team,
+ experiment=experiment,
+ submission_fingerprint="b" * 64,
+ )
+ OfflineEvaluationResult.objects.for_team(self.team.id).create(
+ team=self.team,
+ item=item,
+ scorer_definition=self.definition,
+ scorer_version=self.version,
+ status="ok",
+ boolean_value=True,
+ submission_fingerprint="c" * 64,
+ )
+ for tags, mode, expected_count in (
+ (["quality", "production"], "any", 2),
+ (["quality", "production"], "all", 1),
+ (["missing"], "any", 0),
+ ([], "all", 3),
+ ):
+ with self.subTest(tags=tags, mode=mode):
+ query: dict[str, str | int] = {"tags": json.dumps(tags), "tag_match": mode, "limit": 1}
+ response = self.client.get(self._history(), query)
+ assert response.status_code == 200, response.data
+ assert response.data["count"] == expected_count
+ if expected_count:
+ assert response.data["results"][0]["experiment"]["tags"] == ["production", "quality"]
+ seen = [point["experiment"]["id"] for point in response.data["results"]]
+ while response.data["next_cursor"]:
+ response = self.client.get(self._history(), {**query, "cursor": response.data["next_cursor"]})
+ assert response.status_code == 200, response.data
+ seen.extend(point["experiment"]["id"] for point in response.data["results"])
+ assert len(seen) == len(set(seen)) == expected_count
+
def _authenticate(
self, auth_kind: str, scopes: list[str], user: User | None = None, team: Team | None = None
) -> None:
@@ -173,6 +225,31 @@ class TestOfflineExperimentReads(APIBaseTest):
response = self.client.get(path)
self.assertEqual(response.status_code, expected_status, response.data)
+ @parameterized.expand(
+ [
+ ("editor_session", "session", [], "editor", status.HTTP_200_OK),
+ ("viewer_session", "session", [], "viewer", status.HTTP_403_FORBIDDEN),
+ ("editor_key", "personal_key", ["evaluation:write"], "editor", status.HTTP_200_OK),
+ ("reader_key", "personal_key", ["evaluation:read"], "editor", status.HTTP_403_FORBIDDEN),
+ (
+ "ingestion_key",
+ "personal_key",
+ ["offline_evaluation_ingestion:write"],
+ "editor",
+ status.HTTP_403_FORBIDDEN,
+ ),
+ ("project_key", "project_key", ["evaluation:write"], "editor", status.HTTP_403_FORBIDDEN),
+ ]
+ )
+ def test_tag_updates_require_evaluation_editor_access(
+ self, _name: str, auth_kind: str, scopes: list[str], access: str, expected_status: int
+ ) -> None:
+ membership = self._member(evaluation_access=access)
+ self._authenticate(auth_kind, scopes, user=membership.user)
+ response = self.client.patch(self._endpoint(f"{self.experiment.id}/"), {"tags": ["quality"]}, format="json")
+ assert response.status_code == expected_status, response.data
+ assert self.experiment.tagged_items.count() == (1 if expected_status == 200 else 0)
+
def test_evaluation_only_key_reads_items_but_cannot_select_or_filter_scorers(self) -> None:
self._authenticate("personal_key", ["evaluation:read"])
for path in self._metadata_paths():
diff --git a/products/ai_observability/backend/api/test/test_resource_tags.py b/products/ai_observability/backend/api/test/test_resource_tags.py
new file mode 100644
index 000000000000..92b8b1304286
--- /dev/null
+++ b/products/ai_observability/backend/api/test/test_resource_tags.py
@@ -0,0 +1,212 @@
+import json
+from datetime import timedelta
+from uuid import uuid4
+
+from posthog.test.base import APIBaseTest, NonAtomicBaseTest
+from unittest.mock import Mock, patch
+
+from django.db import connection, transaction
+from django.test import SimpleTestCase
+from django.test.utils import CaptureQueriesContext
+from django.utils import timezone
+
+from parameterized import parameterized
+
+from posthog.models import Tag, Team
+
+from products.ai_observability.backend.api.resource_tags import ResourceTagFilterSerializer, ResourceTagsWriteSerializer
+from products.ai_observability.backend.dataset_service import create_dataset
+from products.ai_observability.backend.models.datasets import Dataset
+from products.ai_observability.backend.models.offline_evaluations import OfflineExperiment
+from products.ai_observability.backend.models.score_definitions import ScoreDefinition
+from products.ai_observability.backend.resource_tags import ResourceTags
+
+RESOURCES = ["datasets", "scorers", "experiments"]
+
+
+class TestResourceTags(APIBaseTest):
+ def setUp(self) -> None:
+ super().setUp()
+ self.enterContext(patch("posthog.permissions.posthog_feature_flag_enabled", return_value=True))
+
+ def _resource(
+ self, entity: str, name: str, team: Team | None = None
+ ) -> Dataset | ScoreDefinition | OfflineExperiment:
+ team = team or self.team
+ if entity == "datasets":
+ return Dataset.objects.for_team(team.id).create(team=team, name=name)
+ if entity == "scorers":
+ definition = ScoreDefinition.objects.create(team=team, name=name, kind="boolean")
+ definition.create_new_version(config={}, created_by=self.user)
+ return definition
+ return OfflineExperiment.objects.for_team(team.id).create(
+ id=uuid4(), team=team, name=name, started_at=timezone.now(), submission_fingerprint="a" * 64
+ )
+
+ def _url(self, entity: str) -> str:
+ route = {
+ "datasets": "datasets",
+ "scorers": "llm_analytics/score_definitions",
+ "experiments": "ai_observability/offline_experiments",
+ }[entity]
+ return f"/api/projects/{self.team.id}/{route}/"
+
+ @parameterized.expand(RESOURCES)
+ def test_replace_tags_and_filter_without_duplicates(self, entity: str) -> None:
+ first = self._resource(entity, "First")
+ second = self._resource(entity, "Second")
+ self._resource(entity, "Untagged")
+ url = self._url(entity)
+ detail_url = f"{url}{first.pk}/"
+ response = self.client.patch(detail_url, {"tags": [" Quality ", "production", "QUALITY", "a,b"]}, format="json")
+ assert response.status_code == 200, response.data
+ assert response.data["tags"] == ["a,b", "production", "quality"]
+ response = self.client.patch(detail_url, {}, format="json")
+ assert response.status_code == 200, response.data
+ assert response.data["tags"] == ["a,b", "production", "quality"]
+ ResourceTags.set(second, ["quality"])
+
+ for mode, expected in (("any", {str(first.pk), str(second.pk)}), ("all", {str(first.pk)})):
+ response = self.client.get(
+ url, {"tags": json.dumps(["Quality", "production", "quality"]), "tag_match": mode, "limit": "1"}
+ )
+ assert response.status_code == 200, response.data
+ assert response.data["count"] == len(expected)
+ assert len(response.data["results"]) == 1
+ assert str(response.data["results"][0]["id"]) in expected
+
+ for tags, expected_count in ((["a,b"], 1), (["missing"], 0), ([], 3)):
+ response = self.client.get(url, {"tags": json.dumps(tags), "tag_match": "all"})
+ assert response.status_code == 200, response.data
+ assert response.data["count"] == expected_count
+
+ response = self.client.patch(detail_url, {"tags": []}, format="json")
+ assert response.status_code == 200, response.data
+ assert response.data["tags"] == []
+ assert list(Tag.objects.filter(team=self.team).values_list("name", flat=True)) == ["quality"]
+ first.refresh_from_db()
+ if isinstance(first, ScoreDefinition):
+ assert first.versions.count() == 1
+ elif isinstance(first, Dataset):
+ assert first.current_revision_id is None
+ else:
+ assert first.status == "uploading"
+
+ @parameterized.expand(
+ [
+ ("empty", [], []),
+ ("normalized", ["production", "quality"], [" Quality ", "production", "QUALITY"]),
+ ]
+ )
+ @patch("products.ai_observability.backend.api.score_definitions.report_user_action")
+ def test_saving_unchanged_scorer_tags_does_not_record_an_update(
+ self, _name: str, current_tags: list[str], submitted_tags: list[str], report_user_action: Mock
+ ) -> None:
+ scorer = self._resource("scorers", "Tagged scorer")
+ assert isinstance(scorer, ScoreDefinition)
+ ResourceTags.set(scorer, current_tags)
+ updated_at = timezone.now() - timedelta(days=1)
+ ScoreDefinition.objects.filter(pk=scorer.pk).update(updated_at=updated_at)
+
+ with self.captureOnCommitCallbacks(execute=True):
+ response = self.client.patch(f"{self._url('scorers')}{scorer.pk}/", {"tags": submitted_tags}, format="json")
+
+ assert response.status_code == 200, response.data
+ assert response.data["tags"] == current_tags
+ scorer.refresh_from_db()
+ assert scorer.updated_at == updated_at
+ report_user_action.assert_not_called()
+
+ @parameterized.expand(RESOURCES)
+ def test_tags_do_not_cross_projects_or_resource_types(self, entity: str) -> None:
+ other_team = Team.objects.create(organization=self.organization)
+ foreign = self._resource(entity, "Other project", other_team)
+ ResourceTags.set(foreign, ["quality"])
+ other_entity = "scorers" if entity != "scorers" else "datasets"
+ ResourceTags.set(self._resource(other_entity, "Other resource"), ["quality"])
+ response = self.client.get(self._url(entity), {"tags": '["quality"]'})
+ assert response.status_code == 200, response.data
+ assert response.data["count"] == 0
+ response = self.client.patch(f"{self._url(entity)}{foreign.pk}/", {"tags": ["changed"]}, format="json")
+ assert response.status_code == 404, response.data
+ assert ResourceTags.names(foreign) == ["quality"]
+
+ @parameterized.expand(RESOURCES)
+ def test_listing_tags_has_constant_query_cost(self, entity: str) -> None:
+ for index in range(5):
+ ResourceTags.set(self._resource(entity, f"Resource {index}"), ["quality", "production"])
+ url = self._url(entity)
+ self.client.get(url)
+ counts = []
+ for limit in (1, 5):
+ with CaptureQueriesContext(connection) as queries:
+ response = self.client.get(url, {"limit": limit, "tags": '["quality"]'})
+ assert response.status_code == 200, response.data
+ assert len(response.data["results"]) == limit
+ assert all(row["tags"] == ["production", "quality"] for row in response.data["results"])
+ counts.append(sum('"posthog_taggeditem"' in query["sql"] for query in queries.captured_queries))
+ assert counts[0] == counts[1]
+
+ @parameterized.expand(["datasets", "scorers"])
+ def test_create_tags_and_preserve_them_when_omitted(self, entity: str) -> None:
+ payload: dict[str, object] = {"name": "Created with tags", "tags": [" Quality "]}
+ if entity == "scorers":
+ payload.update(kind="boolean", config={})
+ response = self.client.post(self._url(entity), payload, format="json")
+ assert response.status_code == 201, response.data
+ assert response.data["tags"] == ["quality"]
+ response = self.client.patch(
+ f"{self._url(entity)}{response.data['id']}/", {"description": "Updated"}, format="json"
+ )
+ assert response.status_code == 200, response.data
+ assert response.data["tags"] == ["quality"]
+
+ @parameterized.expand(RESOURCES)
+ def test_rejects_invalid_tag_filter(self, entity: str) -> None:
+ response = self.client.get(self._url(entity), {"tags": "not-json"})
+ assert response.status_code == 400, response.data
+
+ @parameterized.expand(RESOURCES)
+ def test_rejects_tag_names_that_exceed_the_normalized_length_limit(self, entity: str) -> None:
+ resource = self._resource(entity, "Tagged resource")
+ response = self.client.patch(f"{self._url(entity)}{resource.pk}/", {"tags": ["İ" * 128]}, format="json")
+ assert response.status_code == 400, response.data
+ assert ResourceTags.names(resource) == []
+
+
+class TestResourceTagTransactions(NonAtomicBaseTest):
+ def test_dataset_creation_allows_concurrent_project_foreign_key_checks(self) -> None:
+ other_connection = connection.copy()
+ try:
+ with transaction.atomic():
+ dataset = create_dataset(team=self.team, created_by=self.user, name="Tagged dataset")
+ ResourceTags.set(dataset, ["quality", "production"])
+ with other_connection.cursor() as cursor:
+ cursor.execute("SELECT id FROM posthog_team WHERE id = %s FOR KEY SHARE NOWAIT", [self.team.id])
+ assert cursor.fetchone() == (self.team.id,)
+ finally:
+ other_connection.close()
+
+
+class TestResourceTagValidation(SimpleTestCase):
+ @parameterized.expand(
+ [
+ ("blank", [" "]),
+ ("too_long", ["x" * 256]),
+ ("normalized_too_long", ["İ" * 128]),
+ ("too_many", [str(i) for i in range(101)]),
+ ("not_list", "quality"),
+ ]
+ )
+ def test_invalid_assignments(self, _name: str, tags: object) -> None:
+ serializer = ResourceTagsWriteSerializer(data={"tags": tags})
+ assert not serializer.is_valid()
+ assert "tags" in serializer.errors
+
+ @parameterized.expand(
+ [("invalid_json", "["), ("object", "{}"), ("too_many", json.dumps([str(i) for i in range(21)]))]
+ )
+ def test_invalid_filter(self, _name: str, tags: str) -> None:
+ serializer = ResourceTagFilterSerializer(data={"tags": tags})
+ assert not serializer.is_valid()
+ assert "tags" in serializer.errors
diff --git a/products/ai_observability/backend/dataset_service.py b/products/ai_observability/backend/dataset_service.py
index 7db03f7776c6..752f664dacff 100644
--- a/products/ai_observability/backend/dataset_service.py
+++ b/products/ai_observability/backend/dataset_service.py
@@ -6,7 +6,7 @@ from datetime import datetime
from typing import Literal
from uuid import UUID
-from django.db import IntegrityError, transaction
+from django.db import IntegrityError, connection, transaction
from django.db.models import Max
from posthog.models import Team, User
@@ -402,7 +402,8 @@ def create_dataset(
) -> Dataset:
normalized_metadata = metadata if metadata is not None else {}
normalized_name = _validate_dataset_fields(name=name, description=description, metadata=normalized_metadata)
- Team.objects.select_for_update().only("id").get(id=team.id)
+ with connection.cursor() as cursor:
+ cursor.execute("SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))", [f"aio-dataset-allocation:{team.id}"])
current_count = Dataset.objects.for_team(team.id, canonical=True).count()
if current_count >= MAX_DATASETS_PER_TEAM:
raise DatasetLimitExceeded(
diff --git a/products/ai_observability/backend/models/datasets.py b/products/ai_observability/backend/models/datasets.py
index 79b2db1714e5..36d31484de09 100644
--- a/products/ai_observability/backend/models/datasets.py
+++ b/products/ai_observability/backend/models/datasets.py
@@ -7,6 +7,7 @@ from django.db.models.expressions import NegatedExpression
from django.db.models.lookups import Exact
from posthog.models.scoping.manager import TeamScopedManager
+from posthog.models.tagged_items_relation import Taggable
from posthog.models.utils import CreatedMetaFields, UpdatedMetaFields, UUIDModel
@@ -21,7 +22,7 @@ def _json_type_is_not(field_name: str, json_type: str) -> NegatedExpression:
return NegatedExpression(_json_type_is(field_name, json_type))
-class Dataset(UUIDModel, CreatedMetaFields, UpdatedMetaFields):
+class Dataset(Taggable, UUIDModel, CreatedMetaFields, UpdatedMetaFields):
objects = TeamScopedManager()
team = models.ForeignKey("posthog.Team", on_delete=models.CASCADE, db_constraint=False, related_name="+")
diff --git a/products/ai_observability/backend/models/offline_evaluations.py b/products/ai_observability/backend/models/offline_evaluations.py
index 379efcf9961a..971842a2ed2e 100644
--- a/products/ai_observability/backend/models/offline_evaluations.py
+++ b/products/ai_observability/backend/models/offline_evaluations.py
@@ -5,6 +5,7 @@ from django.db.models.lookups import Exact
from django.utils import timezone
from posthog.models.scoping.manager import EnvironmentScopedManager
+from posthog.models.tagged_items_relation import Taggable
from posthog.models.utils import UUIDModel
@@ -14,7 +15,7 @@ class PayloadState(models.TextChoices):
EXPIRED = "expired", "Expired"
-class OfflineExperiment(models.Model):
+class OfflineExperiment(Taggable):
class Status(models.TextChoices):
UPLOADING = "uploading", "Uploading"
COMPLETED = "completed", "Completed"
diff --git a/products/ai_observability/backend/models/score_definitions.py b/products/ai_observability/backend/models/score_definitions.py
index 677591b8c11c..2b58c6e1e17f 100644
--- a/products/ai_observability/backend/models/score_definitions.py
+++ b/products/ai_observability/backend/models/score_definitions.py
@@ -4,6 +4,7 @@ from typing import Any
from django.db import models, transaction
+from posthog.models.tagged_items_relation import Taggable
from posthog.models.team.team import Team
from posthog.models.user import User
from posthog.models.utils import CreatedMetaFields, UpdatedMetaFields, UUIDModel
@@ -17,7 +18,7 @@ class StaleScoreDefinitionVersion(Exception):
self.current_version = current_version
-class ScoreDefinition(UUIDModel, CreatedMetaFields, UpdatedMetaFields):
+class ScoreDefinition(Taggable, UUIDModel, CreatedMetaFields, UpdatedMetaFields):
class Kind(models.TextChoices):
CATEGORICAL = "categorical", "categorical"
NUMERIC = "numeric", "numeric"
diff --git a/products/ai_observability/backend/offline_evaluation_read_service.py b/products/ai_observability/backend/offline_evaluation_read_service.py
index fd97fd6817dc..c706febec6f6 100644
--- a/products/ai_observability/backend/offline_evaluation_read_service.py
+++ b/products/ai_observability/backend/offline_evaluation_read_service.py
@@ -9,7 +9,7 @@ from typing import TYPE_CHECKING, cast
from uuid import UUID
from django.db import connections
-from django.db.models import Case, Count, Exists, F, OuterRef, Q, When
+from django.db.models import Case, Count, Exists, F, OuterRef, Q, When, prefetch_related_objects
from posthog.dataclasses import frozen
@@ -43,6 +43,7 @@ from products.ai_observability.backend.offline_evaluation_service import (
)
from products.ai_observability.backend.offline_evaluation_types import JSONValue, ResultValue
from products.ai_observability.backend.read_pagination import CursorPage, cursor_page, decode_cursor, encode_cursor
+from products.ai_observability.backend.resource_tags import ResourceTags
if TYPE_CHECKING:
from django.db.models import QuerySet
@@ -156,7 +157,7 @@ class OfflineEvaluationReadService:
return item
def _filter_experiments(self, query: OfflineReadQuery, *, history: bool = False) -> QuerySet[OfflineExperiment]:
- experiments = self._experiments()
+ experiments = ResourceTags.filter(self._experiments(), self.team_id, query.tags, query.tag_match)
if query.date_from is not None:
experiments = experiments.filter(started_at__gte=query.date_from)
if query.date_to is not None:
@@ -278,6 +279,7 @@ class OfflineEvaluationReadService:
}
def _experiment_reads(self, experiments: list[OfflineExperiment]) -> list[OfflineExperimentRead]:
+ prefetch_related_objects(experiments, ResourceTags.prefetch())
ids = [experiment.id for experiment in experiments]
visible_counts = self._experiment_counts(ids)
output = []
@@ -285,6 +287,7 @@ class OfflineEvaluationReadService:
counts = visible_counts.get(experiment.id, _ExperimentCounts())
output.append(
OfflineExperimentRead(
+ tags=tuple(ResourceTags.names(experiment)),
id=experiment.id,
name=experiment.name,
run_source=experiment.run_source,
diff --git a/products/ai_observability/backend/offline_evaluation_read_types.py b/products/ai_observability/backend/offline_evaluation_read_types.py
index bd03f4bf9a23..498b079676de 100644
--- a/products/ai_observability/backend/offline_evaluation_read_types.py
+++ b/products/ai_observability/backend/offline_evaluation_read_types.py
@@ -32,6 +32,8 @@ class OfflineReadQuery:
prompt_version: str | None = None
scorer_definition_id: UUID | None = None
scorer_version_ids: tuple[UUID, ...] = ()
+ tags: tuple[str, ...] = ()
+ tag_match: Literal["any", "all"] = "any"
def __post_init__(self) -> None:
if not 1 <= self.limit <= 100:
@@ -62,6 +64,7 @@ class OfflineResultCellQuery:
@frozen
class OfflineExperimentRead:
+ tags: tuple[str, ...] = ()
id: UUID
name: str
run_source: str | None
diff --git a/products/ai_observability/backend/resource_tags.py b/products/ai_observability/backend/resource_tags.py
new file mode 100644
index 000000000000..a4c0a0b303f6
--- /dev/null
+++ b/products/ai_observability/backend/resource_tags.py
@@ -0,0 +1,54 @@
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import TYPE_CHECKING
+
+from django.db import models, transaction
+from django.db.models import Exists, Prefetch, QuerySet
+
+from posthog.api.tagged_item import cleanup_orphan_tags, set_tags_on_object
+from posthog.models.tagged_item import TaggedItem
+
+if TYPE_CHECKING:
+ from products.ai_observability.backend.models.datasets import Dataset
+ from products.ai_observability.backend.models.offline_evaluations import OfflineExperiment
+ from products.ai_observability.backend.models.score_definitions import ScoreDefinition
+
+ TaggedResource = Dataset | OfflineExperiment | ScoreDefinition
+
+
+class TagMatch(models.TextChoices):
+ ANY = "any", "Any"
+ ALL = "all", "All"
+
+
+class ResourceTags:
+ @staticmethod
+ def prefetch() -> Prefetch:
+ return Prefetch("tagged_items", queryset=TaggedItem.objects.select_related("tag"), to_attr="prefetched_tags")
+
+ @staticmethod
+ def names(resource: TaggedResource) -> list[str]:
+ items = getattr(resource, "prefetched_tags", None)
+ if items is None:
+ items = resource.tagged_items.select_related("tag")
+ return sorted(item.tag.name for item in items)
+
+ @staticmethod
+ def filter[T: models.Model](queryset: QuerySet[T], team_id: int, tags: Sequence[str], match: str) -> QuerySet[T]:
+ if not tags:
+ return queryset
+ assignments = TaggedItem.objects.matching_outer(queryset.model).filter(team_id=team_id, tag__team_id=team_id)
+ if match == TagMatch.ALL:
+ for tag in sorted(set(tags)):
+ queryset = queryset.filter(Exists(assignments.filter(tag__name=tag)))
+ return queryset
+ return queryset.filter(Exists(assignments.filter(tag__name__in=tags)))
+
+ @staticmethod
+ def set(resource: TaggedResource, tags: list[str]) -> None:
+ with transaction.atomic():
+ # Serialize replacement on the resource so concurrent saves cannot merge two replacement sets.
+ type(resource)._base_manager.select_for_update().get(pk=resource.pk, team_id=resource.team_id)
+ resource.__dict__["prefetched_tags"] = set_tags_on_object(tags, resource)
+ cleanup_orphan_tags(resource.team_id)
diff --git a/products/ai_observability/frontend/components/ResourceTagFilter.test.tsx b/products/ai_observability/frontend/components/ResourceTagFilter.test.tsx
new file mode 100644
index 000000000000..bff328674964
--- /dev/null
+++ b/products/ai_observability/frontend/components/ResourceTagFilter.test.tsx
@@ -0,0 +1,36 @@
+import { cleanup, render, screen } from '@testing-library/react'
+import userEvent from '@testing-library/user-event'
+import { Provider } from 'kea'
+
+import { useMocks } from '~/mocks/jest'
+import { initKeaTests } from '~/test/init'
+
+import { ResourceTagFilter } from './ResourceTagFilter'
+
+describe('ResourceTagFilter', () => {
+ beforeEach(() => {
+ initKeaTests()
+ useMocks({ get: { '/api/projects/:teamId/tags': { results: ['quality', 'production'], next: null } } })
+ })
+ afterEach(() => cleanup())
+
+ it.each(['tags', 'tag_match'])('omits the current page when changing %s', async (field) => {
+ const onChange = jest.fn()
+ const filters = { tags: ['quality', 'production'], tag_match: 'any' as const, page: 3 }
+ render(
+ <Provider>
+ <ResourceTagFilter filters={filters} onChange={onChange} />
+ </Provider>
+ )
+
+ if (field === 'tags') {
+ await userEvent.click(screen.getByText('2 tags'))
+ await userEvent.click(await screen.findByLabelText('quality'))
+ expect(onChange).toHaveBeenCalledWith({ tags: ['production'], tag_match: 'any' })
+ } else {
+ await userEvent.click(screen.getByText('Match any'))
+ await userEvent.click(screen.getByText('Match all'))
+ expect(onChange).toHaveBeenCalledWith({ tags: ['quality', 'production'], tag_match: 'all' })
+ }
+ })
+})
diff --git a/products/ai_observability/frontend/components/ResourceTagFilter.tsx b/products/ai_observability/frontend/components/ResourceTagFilter.tsx
new file mode 100644
index 000000000000..835f25c954ca
--- /dev/null
+++ b/products/ai_observability/frontend/components/ResourceTagFilter.tsx
@@ -0,0 +1,43 @@
+import { LemonSelect } from '@posthog/lemon-ui'
+
+import { TagSelect } from 'lib/components/TagSelect'
+import { lemonToast } from 'lib/lemon-ui/LemonToast'
+
+import type { ResourceTagFilters } from './resourceTagFilters'
+
+export function ResourceTagFilter({
+ filters,
+ onChange,
+}: {
+ filters: ResourceTagFilters
+ onChange: (filters: ResourceTagFilters) => void
+}): JSX.Element {
+ return (
+ <div className="flex flex-wrap items-center gap-2">
+ <TagSelect
+ defaultLabel="Tags"
+ size="medium"
+ value={filters.tags}
+ onChange={(tags) => {
+ if (tags.length > 20) {
+ lemonToast.error('Choose up to 20 tags to filter by')
+ return
+ }
+ onChange({ tags, tag_match: filters.tag_match })
+ }}
+ listDataAttr="aio-resource-tags-filter"
+ />
+ {filters.tags.length > 1 && (
+ <LemonSelect
+ value={filters.tag_match}
+ onChange={(tag_match) => onChange({ tags: filters.tags, tag_match })}
+ options={[
+ { value: 'any', label: 'Match any' },
+ { value: 'all', label: 'Match all' },
+ ]}
+ data-attr="aio-resource-tags-match"
+ />
+ )}
+ </div>
+ )
+}
diff --git a/products/ai_observability/frontend/components/ResourceTags.tsx b/products/ai_observability/frontend/components/ResourceTags.tsx
new file mode 100644
index 000000000000..f204e5187f7f
--- /dev/null
+++ b/products/ai_observability/frontend/components/ResourceTags.tsx
@@ -0,0 +1,69 @@
+import { useActions, useValues } from 'kea'
+
+import { IconPencil } from '@posthog/icons'
+import { LemonButton } from '@posthog/lemon-ui'
+
+import { AccessControlAction } from 'lib/components/AccessControlAction'
+import { ObjectTags } from 'lib/components/ObjectTags/ObjectTags'
+import { teamLogic } from 'scenes/teamLogic'
+
+import { AccessControlLevel, AccessControlResourceType } from '~/types'
+
+import type { TaggedResource } from './resourceTagFilters'
+import { resourceTagsLogic } from './resourceTagsLogic'
+import { ResourceTagsModal } from './ResourceTagsModal'
+
+export function ResourceTags({
+ entity,
+ resourceId,
+ tags,
+ onSaved,
+ onTagClick,
+ userAccessLevel,
+ disabledReason,
+}: {
+ entity: TaggedResource
+ resourceId: string
+ tags: readonly string[]
+ onSaved: () => void
+ onTagClick: (tag: string) => void
+ userAccessLevel?: AccessControlLevel
+ disabledReason?: string
+}): JSX.Element {
+ const { currentTeamId } = useValues(teamLogic)
+ const logic = resourceTagsLogic({ entity, resourceId, tags, onSaved, teamId: currentTeamId ?? 0 })
+ const { isOpen } = useValues(logic)
+ const { open } = useActions(logic)
+ return (
+ <div className="flex items-start gap-1 min-w-0 max-w-64">
+ <ObjectTags
+ tags={[...tags]}
+ staticOnly
+ maxVisibleTags={2}
+ wrap
+ onTagClick={onTagClick}
+ data-attr="aio-resource-tags"
+ />
+ <AccessControlAction
+ resourceType={
+ entity === 'experiments'
+ ? AccessControlResourceType.Evaluation
+ : AccessControlResourceType.LlmAnalytics
+ }
+ minAccessLevel={AccessControlLevel.Editor}
+ userAccessLevel={userAccessLevel}
+ >
+ <LemonButton
+ icon={<IconPencil />}
+ size="xsmall"
+ disabledReason={disabledReason}
+ tooltip="Edit tags"
+ aria-label="Edit tags"
+ onClick={open}
+ data-attr="aio-resource-tags-edit"
+ />
+ </AccessControlAction>
+ {isOpen && <ResourceTagsModal logic={logic} />}
+ </div>
+ )
+}
diff --git a/products/ai_observability/frontend/components/ResourceTagsModal.tsx b/products/ai_observability/frontend/components/ResourceTagsModal.tsx
new file mode 100644
index 000000000000..a9a0949fca52
--- /dev/null
+++ b/products/ai_observability/frontend/components/ResourceTagsModal.tsx
@@ -0,0 +1,44 @@
+import { useActions, useValues, type BuiltLogic } from 'kea'
+
+import { LemonBanner, LemonButton, LemonInputSelect, LemonModal } from '@posthog/lemon-ui'
+
+import { tagsModel } from '~/models/tagsModel'
+
+import type { resourceTagsLogicType } from './resourceTagsLogic'
+
+export function ResourceTagsModal({ logic }: { logic: BuiltLogic<resourceTagsLogicType> }): JSX.Element {
+ const { isOpen, draft, saving, error } = useValues(logic)
+ const { close, setDraft, save } = useActions(logic)
+ const { tags: availableTags } = useValues(tagsModel)
+ return (
+ <LemonModal
+ isOpen={isOpen}
+ onClose={saving ? undefined : close}
+ title="Edit tags"
+ footer={
+ <>
+ <LemonButton type="secondary" onClick={close} disabledReason={saving ? 'Saving tags' : undefined}>
+ Cancel
+ </LemonButton>
+ <LemonButton type="primary" onClick={save} loading={saving} data-attr="aio-resource-tags-save">
+ Save
+ </LemonButton>
+ </>
+ }
+ >
+ <div className="space-y-3">
+ {error && <LemonBanner type="error">{error}</LemonBanner>}
+ <LemonInputSelect
+ mode="multiple"
+ value={draft}
+ onChange={setDraft}
+ options={availableTags.map((tag) => ({ key: tag, label: tag }))}
+ allowCustomValues
+ placeholder="Add tags"
+ disabled={saving}
+ fullWidth
+ />
+ </div>
+ </LemonModal>
+ )
+}
diff --git a/products/ai_observability/frontend/components/resourceTagFilters.test.ts b/products/ai_observability/frontend/components/resourceTagFilters.test.ts
new file mode 100644
index 000000000000..c9ee512e4e11
--- /dev/null
+++ b/products/ai_observability/frontend/components/resourceTagFilters.test.ts
@@ -0,0 +1,38 @@
+import { cleanTagFilters, restoreTagFilters, saveTagFilters } from './resourceTagFilters'
+
+describe('resource tag preferences', () => {
+ beforeEach(() => localStorage.clear())
+
+ it('keeps each resource, project, and user independent', () => {
+ const selection = { tags: ['quality', 'production'], tag_match: 'all' as const }
+ saveTagFilters('scorers', selection, 1, 10)
+ expect(restoreTagFilters('scorers', {}, 1, 10)).toEqual(selection)
+ expect(restoreTagFilters('datasets', {}, 1, 10).tags).toEqual([])
+ expect(restoreTagFilters('experiments', {}, 1, 10).tags).toEqual([])
+ expect(restoreTagFilters('scorers', {}, 2, 10).tags).toEqual([])
+ expect(restoreTagFilters('scorers', {}, 1, 11).tags).toEqual([])
+ })
+
+ it('gives shared URLs precedence, including an explicitly empty filter', () => {
+ saveTagFilters('datasets', { tags: ['saved'], tag_match: 'all' }, 1, 10)
+ expect(restoreTagFilters('datasets', { tags: ['shared'], tag_match: 'any' }, 1, 10)).toEqual({
+ tags: ['shared'],
+ tag_match: 'any',
+ })
+ expect(restoreTagFilters('datasets', { tags: [] }, 1, 10)).toEqual({ tags: [], tag_match: 'any' })
+ expect(restoreTagFilters('datasets', { tags: 'broken' }, 1, 10).tags).toEqual([])
+ })
+
+ it('preserves commas and normalizes names when decoding a URL', () => {
+ expect(cleanTagFilters({ tags: '[" Quality ", "QUALITY", "a,b"]', tag_match: 'all' })).toEqual({
+ tags: ['quality', 'a,b'],
+ tag_match: 'all',
+ })
+ })
+
+ it('can clear a saved selection', () => {
+ saveTagFilters('experiments', { tags: ['quality'], tag_match: 'all' }, 1, 10)
+ saveTagFilters('experiments', { tags: [], tag_match: 'any' }, 1, 10)
+ expect(restoreTagFilters('experiments', {}, 1, 10).tags).toEqual([])
+ })
+})
diff --git a/products/ai_observability/frontend/components/resourceTagFilters.ts b/products/ai_observability/frontend/components/resourceTagFilters.ts
new file mode 100644
index 000000000000..1c8219603f69
--- /dev/null
+++ b/products/ai_observability/frontend/components/resourceTagFilters.ts
@@ -0,0 +1,68 @@
+import { ApiConfig } from 'lib/api'
+import { userLogic } from 'scenes/userLogic'
+
+export type TaggedResource = 'datasets' | 'experiments' | 'scorers'
+
+export interface ResourceTagFilters {
+ tags: string[]
+ tag_match: 'any' | 'all'
+}
+
+export function cleanTagFilters(values: { tags?: unknown; tag_match?: unknown }): ResourceTagFilters {
+ let tags = values.tags
+ if (typeof tags === 'string') {
+ try {
+ tags = JSON.parse(tags)
+ } catch {
+ tags = []
+ }
+ }
+ return {
+ tags: Array.isArray(tags)
+ ? [
+ ...new Set(
+ tags
+ .filter((tag): tag is string => typeof tag === 'string')
+ .map((tag) => tag.trim().toLowerCase())
+ .filter(Boolean)
+ ),
+ ].slice(0, 20)
+ : [],
+ tag_match: values.tag_match === 'all' ? 'all' : 'any',
+ }
+}
+
+function preferenceKey(entity: TaggedResource, teamId: number | string, userId: number | undefined): string {
+ // Persisted key: keep the version stable unless the stored filter shape changes.
+ return `aio-tag-filters:v1:${userId ?? 'unknown'}:${teamId}:${entity}`
+}
+
+export function restoreTagFilters(
+ entity: TaggedResource,
+ search: Record<string, unknown>,
+ teamId: number | string = ApiConfig.getCurrentTeamId(),
+ userId: number | undefined = userLogic.findMounted()?.values.user?.id
+): ResourceTagFilters {
+ if ('tags' in search || 'tag_match' in search) {
+ return cleanTagFilters(search)
+ }
+ try {
+ const value: unknown = JSON.parse(localStorage.getItem(preferenceKey(entity, teamId, userId)) || 'null')
+ return cleanTagFilters(value && typeof value === 'object' ? value : {})
+ } catch {
+ return cleanTagFilters({})
+ }
+}
+
+export function saveTagFilters(
+ entity: TaggedResource,
+ filters: ResourceTagFilters,
+ teamId: number | string = ApiConfig.getCurrentTeamId(),
+ userId: number | undefined = userLogic.findMounted()?.values.user?.id
+): void {
+ try {
+ localStorage.setItem(preferenceKey(entity, teamId, userId), JSON.stringify(cleanTagFilters(filters)))
+ } catch {
+ // Storage may be disabled; URL-backed filters still work.
+ }
+}
diff --git a/products/ai_observability/frontend/components/resourceTagsLogic.test.ts b/products/ai_observability/frontend/components/resourceTagsLogic.test.ts
new file mode 100644
index 000000000000..9db4fdd0b6bf
--- /dev/null
+++ b/products/ai_observability/frontend/components/resourceTagsLogic.test.ts
@@ -0,0 +1,85 @@
+import { expectLogic } from 'kea-test-utils'
+
+import api from 'lib/api'
+
+import { tagsModel } from '~/models/tagsModel'
+import { initKeaTests } from '~/test/init'
+
+import * as generated from '../generated/api'
+import type { TaggedResource } from './resourceTagFilters'
+import { resourceTagsLogic } from './resourceTagsLogic'
+
+jest.mock('../generated/api', () => ({
+ datasetsPartialUpdate: jest.fn(),
+ llmAnalyticsScoreDefinitionsPartialUpdate: jest.fn(),
+ aiObservabilityOfflineExperimentsPartialUpdate: jest.fn(),
+}))
+jest.mock('lib/lemon-ui/LemonToast')
+
+const updates = {
+ datasets: jest.mocked(generated.datasetsPartialUpdate),
+ scorers: jest.mocked(generated.llmAnalyticsScoreDefinitionsPartialUpdate),
+ experiments: jest.mocked(generated.aiObservabilityOfflineExperimentsPartialUpdate),
+}
+
+describe('resourceTagsLogic', () => {
+ beforeEach(() => {
+ initKeaTests()
+ jest.clearAllMocks()
+ jest.spyOn(api.tags, 'list').mockResolvedValue([])
+ tagsModel.mount()
+ })
+
+ it.each<TaggedResource>(['datasets', 'scorers', 'experiments'])(
+ 'saves %s tags only once while a request is pending',
+ async (entity) => {
+ let finish: () => void = () => {}
+ const pending = new Promise<never>((resolve) => {
+ finish = () => resolve(undefined as never)
+ })
+ updates[entity].mockReturnValue(pending)
+ const onSaved = jest.fn()
+ const logic = resourceTagsLogic({
+ entity,
+ resourceId: 'resource-id',
+ teamId: 997,
+ tags: ['before'],
+ onSaved,
+ })
+ logic.mount()
+ logic.actions.open()
+ logic.actions.setDraft(['quality', 'production'])
+ logic.actions.save()
+ logic.actions.save()
+ expect(logic.values.saving).toBe(true)
+ expect(updates[entity]).toHaveBeenCalledTimes(1)
+ expect(updates[entity]).toHaveBeenCalledWith('997', 'resource-id', { tags: ['quality', 'production'] })
+ await expectLogic(logic, finish).toFinishAllListeners()
+ expect(logic.values.saving).toBe(false)
+ expect(logic.values.isOpen).toBe(false)
+ expect(onSaved).toHaveBeenCalledTimes(1)
+ }
+ )
+
+ it('keeps the draft after a failed save so it can be retried', async () => {
+ updates.datasets.mockRejectedValueOnce(new Error('Failed')).mockResolvedValueOnce({} as never)
+ const onSaved = jest.fn()
+ const logic = resourceTagsLogic({
+ entity: 'datasets',
+ resourceId: 'resource-id',
+ teamId: 997,
+ tags: ['before'],
+ onSaved,
+ })
+ logic.mount()
+ logic.actions.open()
+ logic.actions.setDraft([])
+ await expectLogic(logic, () => logic.actions.save()).toFinishAllListeners()
+ expect(logic.values).toMatchObject({ isOpen: true, saving: false, draft: [] })
+ expect(logic.values.error).toBeTruthy()
+ expect(onSaved).not.toHaveBeenCalled()
+ await expectLogic(logic, () => logic.actions.save()).toFinishAllListeners()
+ expect(logic.values.error).toBeNull()
+ expect(onSaved).toHaveBeenCalledTimes(1)
+ })
+})
diff --git a/products/ai_observability/frontend/components/resourceTagsLogic.ts b/products/ai_observability/frontend/components/resourceTagsLogic.ts
new file mode 100644
index 000000000000..ebe2e2c755b0
--- /dev/null
+++ b/products/ai_observability/frontend/components/resourceTagsLogic.ts
@@ -0,0 +1,117 @@
+import { MakeLogicType, actions, connect, kea, key, listeners, path, props, reducers, type LogicWrapper } from 'kea'
+
+import { lemonToast } from 'lib/lemon-ui/LemonToast'
+
+import { tagsModel } from '~/models/tagsModel'
+
+import {
+ datasetsPartialUpdate,
+ llmAnalyticsScoreDefinitionsPartialUpdate,
+ aiObservabilityOfflineExperimentsPartialUpdate,
+} from '../generated/api'
+import { getApiErrorDetail } from '../scoreDefinitions/scoreDefinitionModalUtils'
+import type { TaggedResource } from './resourceTagFilters'
+
+export interface ResourceTagsLogicProps {
+ entity: TaggedResource
+ resourceId: string
+ teamId: number
+ tags: readonly string[]
+ onSaved: () => void
+}
+
+// Generated by kea-typegen. Update if you're an agent, ignore if you're human.
+export interface resourceTagsLogicValues {
+ draft: string[]
+ error: string | null
+ isOpen: boolean
+ saving: boolean
+}
+
+// Generated by kea-typegen. Update if you're an agent, ignore if you're human.
+export interface resourceTagsLogicActions {
+ loadTagsIfNeeded: () => {
+ value: true
+ } // tagsModel
+ refreshTags: () => {
+ value: true
+ } // tagsModel
+ close: () => {
+ value: true
+ }
+ open: () => {
+ value: true
+ }
+ save: () => {
+ value: true
+ }
+ setDraft: (tags: string[]) => {
+ tags: string[]
+ }
+ setError: (error: string | null) => {
+ error: string | null
+ }
+ setSaving: (saving: boolean) => {
+ saving: boolean
+ }
+}
+
+// Generated by kea-typegen. Update if you're an agent, ignore if you're human.
+export interface resourceTagsLogicMeta {
+ key: string
+}
+
+export type resourceTagsLogicType = MakeLogicType<
+ resourceTagsLogicValues,
+ resourceTagsLogicActions,
+ ResourceTagsLogicProps,
+ resourceTagsLogicMeta
+>
+
+export const resourceTagsLogic: LogicWrapper<resourceTagsLogicType> = kea<resourceTagsLogicType>([
+ props({} as ResourceTagsLogicProps),
+ key(({ entity, resourceId, teamId }) => `${teamId}:${entity}:${resourceId}`),
+ path((key) => ['products', 'ai_observability', 'resourceTagsLogic', key]),
+ connect({ actions: [tagsModel, ['loadTagsIfNeeded', 'refreshTags']] }),
+ actions({
+ open: true,
+ close: true,
+ setDraft: (tags: string[]) => ({ tags }),
+ save: true,
+ setSaving: (saving: boolean) => ({ saving }),
+ setError: (error: string | null) => ({ error }),
+ }),
+ reducers(({ props }) => ({
+ isOpen: [false, { open: () => true, close: () => false }],
+ draft: [[] as string[], { open: () => [...props.tags], setDraft: (_, { tags }) => tags }],
+ saving: [false, { setSaving: (_, { saving }) => saving }],
+ error: [null as string | null, { open: () => null, setError: (_, { error }) => error }],
+ })),
+ listeners(({ actions, values, props }) => ({
+ open: () => actions.loadTagsIfNeeded(),
+ save: async () => {
+ if (values.saving) {
+ return
+ }
+ actions.setSaving(true)
+ actions.setError(null)
+ try {
+ const update =
+ props.entity === 'datasets'
+ ? datasetsPartialUpdate
+ : props.entity === 'scorers'
+ ? llmAnalyticsScoreDefinitionsPartialUpdate
+ : aiObservabilityOfflineExperimentsPartialUpdate
+ await update(String(props.teamId), props.resourceId, { tags: values.draft })
+ actions.refreshTags()
+ props.onSaved()
+ actions.close()
+ lemonToast.success('Tags updated')
+ } catch (error) {
+ actions.setError(getApiErrorDetail(error) || 'Could not update tags. Please try again.')
+ } finally {
+ actions.setSaving(false)
+ }
+ },
+ })),
+])
diff --git a/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx b/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx
index 70b6c9c5e0ad..4252dc54025e 100644
--- a/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx
+++ b/products/ai_observability/frontend/datasets/AIObservabilityDatasetsScene.tsx
@@ -19,6 +19,8 @@ import { toAccessControlLevel } from '~/lib/utils/accessControlUtils'
import { ProductKey } from '~/queries/schema/schema-general'
import { AccessControlLevel, AccessControlResourceType, type UserBasicType } from '~/types'
+import { ResourceTagFilter } from '../components/ResourceTagFilter'
+import { ResourceTags } from '../components/ResourceTags'
import { datasetsEmptyState } from '../emptyState/datasetsEmptyState'
import type { DatasetReadApi as Dataset } from '../generated/api.schemas'
import { DATASETS_PER_PAGE, aiObservabilityDatasetsLogic, getDatasetDetailUrl } from './aiObservabilityDatasetsLogic'
@@ -31,7 +33,7 @@ export const scene: SceneExport = {
}
export function AIObservabilityDatasetsScene(): JSX.Element {
- const { setFilters, archiveDataset, restoreDataset } = useActions(aiObservabilityDatasetsLogic)
+ const { loadDatasets, setFilters, archiveDataset, restoreDataset } = useActions(aiObservabilityDatasetsLogic)
const { archivingDatasetId, datasets, datasetsLoading, sorting, pagination, filters, datasetCountLabel } =
useValues(aiObservabilityDatasetsLogic)
const { searchParams } = useValues(router)
@@ -42,10 +44,14 @@ export function AIObservabilityDatasetsScene(): JSX.Element {
title: 'Name',
dataIndex: 'name',
key: 'name',
- width: '20%',
+ width: '25%',
render: function renderName(_, dataset) {
return (
- <Link to={datasetUrl(dataset.id)} data-testid="dataset-link">
+ <Link
+ to={datasetUrl(dataset.id)}
+ data-testid="dataset-link"
+ className="block max-w-64 @max-[50rem]/datasets:max-w-32 truncate font-semibold"
+ >
{dataset.name}
</Link>
)
@@ -55,13 +61,15 @@ export function AIObservabilityDatasetsScene(): JSX.Element {
title: 'Description',
dataIndex: 'description',
key: 'description',
- width: '50%',
+ width: '20%',
+ className: '@max-[50rem]/datasets:hidden',
render: function renderDescription(description) {
return <span className="text-muted">{String(description) || <i>–</i>}</span>
},
},
{
title: 'Created by',
+ className: '@max-[50rem]/datasets:hidden',
dataIndex: 'created_by',
render: function renderCreatedBy(_, item) {
const { created_by } = item
@@ -83,8 +91,30 @@ export function AIObservabilityDatasetsScene(): JSX.Element {
)
},
},
- createdAtColumn<Dataset>() as LemonTableColumn<Dataset, keyof Dataset | undefined>,
- updatedAtColumn<Dataset>() as LemonTableColumn<Dataset, keyof Dataset | undefined>,
+ {
+ title: 'Tags',
+ key: 'tags',
+ width: '20%',
+ render: (_, dataset) => (
+ <ResourceTags
+ entity="datasets"
+ resourceId={dataset.id}
+ tags={dataset.tags}
+ onSaved={() => loadDatasets(false)}
+ onTagClick={(tag) => setFilters({ tags: [tag] }, true, false)}
+ userAccessLevel={toAccessControlLevel(dataset.user_access_level)}
+ disabledReason={dataset.archived ? 'Restore this dataset to edit its tags' : undefined}
+ />
+ ),
+ },
+ {
+ ...(createdAtColumn<Dataset>() as LemonTableColumn<Dataset, keyof Dataset | undefined>),
+ className: '@max-[60rem]/datasets:hidden',
+ },
+ {
+ ...(updatedAtColumn<Dataset>() as LemonTableColumn<Dataset, keyof Dataset | undefined>),
+ className: '@max-[50rem]/datasets:hidden',
+ },
{
width: 0,
render: function renderMore(_, dataset) {
@@ -133,7 +163,7 @@ export function AIObservabilityDatasetsScene(): JSX.Element {
]
return (
- <SceneContent>
+ <SceneContent className="@container/datasets">
<SceneTitleSection
name="Datasets"
description="Create and manage datasets to test changes and evaluate your AI outputs. [Learn more](https://posthog.com/docs/ai-evals/datasets)"
@@ -152,6 +182,7 @@ export function AIObservabilityDatasetsScene(): JSX.Element {
size="small"
data-attr="datasets-status-filter"
/>
+ <ResourceTagFilter filters={filters} onChange={(tags) => setFilters(tags, true, false)} />
<LemonInput
type="search"
placeholder="Search datasets..."
@@ -183,6 +214,7 @@ export function AIObservabilityDatasetsScene(): JSX.Element {
<LemonTable
id="datasets"
+ className="@max-[50rem]/datasets:[&_colgroup]:hidden"
loading={datasetsLoading}
columns={columns}
dataSource={datasets.results}
diff --git a/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.test.ts b/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.test.ts
index da4d88ab32bc..ea27bb97eaa6 100644
--- a/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.test.ts
+++ b/products/ai_observability/frontend/datasets/aiObservabilityDatasetLogic.test.ts
@@ -43,6 +43,7 @@ jest.mock('lib/lemon-ui/LemonToast/LemonToast')
describe('aiObservabilityDatasetLogic', () => {
const mockDataset: Dataset = {
+ tags: [],
id: 'test-dataset-id',
name: 'Test Dataset',
description: 'Test description',
diff --git a/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.test.ts b/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.test.ts
index d8f4edbfd532..e320ce347539 100644
--- a/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.test.ts
+++ b/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.test.ts
@@ -1,6 +1,8 @@
+import { router } from 'kea-router'
import { expectLogic } from 'kea-test-utils'
import { lemonToast } from 'lib/lemon-ui/LemonToast/LemonToast'
+import { urls } from 'scenes/urls'
import { initKeaTests } from '~/test/init'
@@ -25,6 +27,7 @@ jest.mock('lib/lemon-ui/LemonToast/LemonToast')
describe('aiObservabilityDatasetsLogic', () => {
const mockDataset1: Dataset = {
+ tags: [],
id: 'test-dataset-1',
name: 'Test Dataset 1',
description: 'First test dataset',
@@ -40,6 +43,7 @@ describe('aiObservabilityDatasetsLogic', () => {
}
const mockDataset2: Dataset = {
+ tags: [],
id: 'test-dataset-2',
name: 'Test Dataset 2',
description: 'Second test dataset',
@@ -63,6 +67,7 @@ describe('aiObservabilityDatasetsLogic', () => {
beforeEach(() => {
initKeaTests()
+ localStorage.clear()
jest.resetAllMocks()
mockDatasetsApi.listDatasets.mockResolvedValue(mockDatasetsResponse)
@@ -94,6 +99,27 @@ describe('aiObservabilityDatasetsLogic', () => {
expect(getDatasetDetailUrl(mockDataset1.id, searchParams)).not.toContain('item=')
})
+ it('shares tag filters in the URL and restores them when returning to datasets', async () => {
+ router.actions.push(urls.aiObservabilityDatasets())
+ const logic = aiObservabilityDatasetsLogic()
+ logic.mount()
+ await expectLogic(logic, () =>
+ logic.actions.setFilters({ tags: ['quality', 'a,b'], tag_match: 'all' }, true, false)
+ ).toFinishAllListeners()
+ expect(router.values.searchParams).toMatchObject({ tags: ['quality', 'a,b'], tag_match: 'all' })
+ expect(mockDatasetsApi.listDatasets).toHaveBeenLastCalledWith(
+ expect.objectContaining({ tags: '["quality","a,b"]', tag_match: 'all' })
+ )
+ router.actions.push(urls.aiObservabilityScorers())
+ router.actions.push(urls.aiObservabilityDatasets())
+ await expectLogic(logic).toFinishAllListeners()
+ expect(logic.values.filters.tags).toEqual(['quality', 'a,b'])
+ expect(router.values.searchParams.tags).toEqual(['quality', 'a,b'])
+ router.actions.push(urls.aiObservabilityDatasets(), { tags: [], tag_match: 'any' })
+ await expectLogic(logic).toFinishAllListeners()
+ expect(logic.values.filters.tags).toEqual([])
+ })
+
describe('filters functionality', () => {
it('sets and processes filters correctly', () => {
const logic = aiObservabilityDatasetsLogic()
@@ -110,6 +136,8 @@ describe('aiObservabilityDatasetsLogic', () => {
)
expect(logic.values.filters).toEqual({
+ tags: [],
+ tag_match: 'any',
page: 2,
search: 'test search',
order_by: 'name',
@@ -140,6 +168,8 @@ describe('aiObservabilityDatasetsLogic', () => {
logic.actions.setFilters({}, false)
expect(logic.values.filters).toEqual({
+ tags: [],
+ tag_match: 'any',
page: 1,
search: '',
order_by: '-created_at',
diff --git a/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.ts b/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.ts
index 48a142783ed5..4ccd90becfa0 100644
--- a/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.ts
+++ b/products/ai_observability/frontend/datasets/aiObservabilityDatasetsLogic.ts
@@ -12,13 +12,26 @@ import { trackedActionToUrl } from '~/lib/logic/scenes/trackedActionToUrl'
import { sceneLogic } from '~/scenes/sceneLogic'
import { urls } from '~/scenes/urls'
+import {
+ cleanTagFilters,
+ restoreTagFilters,
+ saveTagFilters,
+ type ResourceTagFilters,
+} from '../components/resourceTagFilters'
import type { PaginatedDatasetReadListApi } from '../generated/api.schemas'
import { cleanPagedSearchOrderParams } from '../utils'
import { datasetsApi } from './datasetsApi'
export const DATASETS_PER_PAGE = 30
-const DATASET_NAVIGATION_SEARCH_PARAM_KEYS = new Set(['dataset_status', 'datasets_page', 'order_by', 'search'])
+const DATASET_NAVIGATION_SEARCH_PARAM_KEYS = new Set([
+ 'dataset_status',
+ 'datasets_page',
+ 'order_by',
+ 'search',
+ 'tags',
+ 'tag_match',
+])
export function getDatasetNavigationSearchParams(searchParams: Record<string, any>): Record<string, any> {
const navigationSearchParams = Object.fromEntries(
@@ -36,7 +49,7 @@ export function getDatasetDetailUrl(datasetId: string, searchParams: Record<stri
return combineUrl(urls.aiObservabilityDataset(datasetId), getDatasetNavigationSearchParams(searchParams)).url
}
-export interface DatasetFilters {
+export interface DatasetFilters extends ResourceTagFilters {
page: number
search: string
order_by: string
@@ -47,6 +60,7 @@ function cleanFilters(
values: Partial<DatasetFilters> & { dataset_status?: unknown; datasets_page?: unknown }
): DatasetFilters {
return {
+ ...cleanTagFilters(values),
page: parseInt(String(values.datasets_page ?? values.page)) || 1,
search: String(values.search || ''),
order_by: values.order_by || '-created_at',
@@ -185,6 +199,9 @@ export const aiObservabilityDatasetsLogic = kea<aiObservabilityDatasetsLogicType
offset: Math.max(0, (filters.page - 1) * DATASETS_PER_PAGE),
limit: DATASETS_PER_PAGE,
archived: filters.archived,
+ ...(filters.tags.length
+ ? { tags: JSON.stringify(filters.tags), tag_match: filters.tag_match }
+ : {}),
}
// Scroll to top if the page changed, except if changed via back/forward
@@ -258,6 +275,7 @@ export const aiObservabilityDatasetsLogic = kea<aiObservabilityDatasetsLogicType
setFilters: async ({ debounce }, _, __, previousState) => {
const oldFilters = selectors.filters(previousState)
const { filters } = values
+ saveTagFilters('datasets', filters)
if (!objectsEqual(oldFilters, filters)) {
await asyncActions.loadDatasets(debounce)
@@ -321,6 +339,8 @@ export const aiObservabilityDatasetsLogic = kea<aiObservabilityDatasetsLogicType
...pagedSearchOrderParams,
datasets_page: page,
dataset_status: values.filters.archived ? 'archived' : undefined,
+ tags: values.filters.tags,
+ tag_match: values.filters.tag_match,
}
const urlValues = cleanFilters(router.values.searchParams)
if (!objectsEqual(values.filters, urlValues)) {
@@ -334,16 +354,19 @@ export const aiObservabilityDatasetsLogic = kea<aiObservabilityDatasetsLogicType
urlToAction(({ actions, values }) => ({
[urls.aiObservabilityDatasets()]: (_, searchParams, __, { method }) => {
- const newFilters = cleanFilters(searchParams)
+ const newFilters = cleanFilters({ ...searchParams, ...restoreTagFilters('datasets', searchParams) })
if (!objectsEqual(values.filters, newFilters)) {
actions.setFilters(newFilters, false)
} else if (method !== 'REPLACE') {
+ actions.setFilters(newFilters, false)
actions.loadDatasets(false)
}
},
})),
- afterMount(({ actions }) => {
- actions.loadDatasets()
+ afterMount(({ actions, values }) => {
+ if (values.rawFilters === null) {
+ actions.loadDatasets()
+ }
}),
])
diff --git a/products/ai_observability/frontend/datasets/saveToDatasetButtonLogic.test.ts b/products/ai_observability/frontend/datasets/saveToDatasetButtonLogic.test.ts
index d555fbe8d624..83288d875389 100644
--- a/products/ai_observability/frontend/datasets/saveToDatasetButtonLogic.test.ts
+++ b/products/ai_observability/frontend/datasets/saveToDatasetButtonLogic.test.ts
@@ -31,6 +31,7 @@ jest.mock('lib/lemon-ui/LemonToast')
describe('saveToDatasetButtonLogic', () => {
const mockDataset1: Dataset = {
+ tags: [],
id: 'test-dataset-1',
name: 'Test Dataset 1',
description: 'First test dataset',
@@ -46,6 +47,7 @@ describe('saveToDatasetButtonLogic', () => {
}
const mockDataset2: Dataset = {
+ tags: [],
id: 'test-dataset-2',
name: 'Test Dataset 2',
description: 'Second test dataset',
diff --git a/products/ai_observability/frontend/generated/api.schemas.ts b/products/ai_observability/frontend/generated/api.schemas.ts
index 724773063c48..d21be1ebbbe5 100644
--- a/products/ai_observability/frontend/generated/api.schemas.ts
+++ b/products/ai_observability/frontend/generated/api.schemas.ts
@@ -323,6 +323,8 @@ export const OfflineExperimentStatusEnumApi = {
} as const
export interface OfflineExperimentReadApi {
+ /** Project tag names assigned to this experiment. */
+ tags: string[]
/** Stable experiment UUID. */
id: string
/** Experiment name. */
@@ -588,6 +590,15 @@ export interface ExperimentReceiptApi {
accepted_result_count: number
}
+export interface PatchedOfflineExperimentMetadataApi {
+ /**
+ * Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.
+ * @maxItems 100
+ * @items.maxLength 255
+ */
+ tags?: string[]
+}
+
/**
* * `not_provided` - Not provided
* * `available` - Available
@@ -1597,6 +1608,8 @@ export interface DatasetReadApi {
readonly description: string
/** JSON object with descriptive dataset metadata. */
readonly metadata: DatasetReadApiMetadata
+ /** Project tag names assigned to this resource. */
+ readonly tags: readonly string[]
readonly archived: boolean
/**
* Latest dataset revision, or null before the first item is added.
@@ -1636,6 +1649,12 @@ export interface PaginatedDatasetReadListApi {
export type DatasetCreateApiMetadata = { [key: string]: unknown }
export interface DatasetCreateApi {
+ /**
+ * Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.
+ * @maxItems 100
+ * @items.maxLength 255
+ */
+ tags?: string[]
/**
* Dataset name. Names are unique within a project.
* @maxLength 400
@@ -1656,6 +1675,12 @@ export interface DatasetCreateApi {
export type PatchedDatasetUpdateApiMetadata = { [key: string]: unknown }
export interface PatchedDatasetUpdateApi {
+ /**
+ * Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.
+ * @maxItems 100
+ * @items.maxLength 255
+ */
+ tags?: string[]
/**
* New dataset name. Names are unique within a project.
* @maxLength 400
@@ -3671,6 +3696,8 @@ export interface ScoreDefinitionApi {
readonly id: string
readonly name: string
readonly description: string
+ /** Project tag names assigned to this resource. */
+ readonly tags: readonly string[]
readonly kind: ScoreDefinitionKindEnumApi
readonly archived: boolean
/** Current immutable configuration version number. */
@@ -3700,6 +3727,12 @@ export interface PaginatedScoreDefinitionListApi {
}
export interface ScoreDefinitionCreateApi {
+ /**
+ * Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.
+ * @maxItems 100
+ * @items.maxLength 255
+ */
+ tags?: string[]
/**
* Human-readable scorer name.
* @maxLength 255
@@ -3723,6 +3756,12 @@ export interface ScoreDefinitionCreateApi {
}
export interface PatchedScoreDefinitionMetadataApi {
+ /**
+ * Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.
+ * @maxItems 100
+ * @items.maxLength 255
+ */
+ tags?: string[]
/**
* Updated scorer name.
* @maxLength 255
@@ -4818,8 +4857,28 @@ export type AiObservabilityOfflineExperimentsListParams = {
* @maxLength 255
*/
suite_key?: string
+ /**
+ * Match any selected tag (OR) or all selected tags (AND).
+ *
+ * * `any` - Any
+ * * `all` - All
+ * @minLength 1
+ */
+ tag_match?: AiObservabilityOfflineExperimentsListTagMatch
+ /**
+ * JSON array of tag names, for example ["quality","production"]. Up to 20 tags. An empty array disables tag filtering.
+ */
+ tags?: string
}
+export type AiObservabilityOfflineExperimentsListTagMatch =
+ (typeof AiObservabilityOfflineExperimentsListTagMatch)[keyof typeof AiObservabilityOfflineExperimentsListTagMatch]
+
+export const AiObservabilityOfflineExperimentsListTagMatch = {
+ Any: 'any',
+ All: 'all',
+} as const
+
export type AiObservabilityOfflineExperimentsItemsListParams = {
/**
* Continuation cursor returned by the previous page.
@@ -4993,8 +5052,28 @@ export type AiObservabilityOfflineScorersHistoryListParams = {
* @maxLength 255
*/
suite_key?: string
+ /**
+ * Match any selected tag (OR) or all selected tags (AND).
+ *
+ * * `any` - Any
+ * * `all` - All
+ * @minLength 1
+ */
+ tag_match?: AiObservabilityOfflineScorersHistoryListTagMatch
+ /**
+ * JSON array of tag names, for example ["quality","production"]. Up to 20 tags. An empty array disables tag filtering.
+ */
+ tags?: string
}
+export type AiObservabilityOfflineScorersHistoryListTagMatch =
+ (typeof AiObservabilityOfflineScorersHistoryListTagMatch)[keyof typeof AiObservabilityOfflineScorersHistoryListTagMatch]
+
+export const AiObservabilityOfflineScorersHistoryListTagMatch = {
+ Any: 'any',
+ All: 'all',
+} as const
+
export type AiObservabilityTracesRetrieveParams = {
/**
* When the trace happened, as carried by links into it. Lets a trace older than the AI events retention load from the shared events table.
@@ -5114,8 +5193,27 @@ export type DatasetsListParams = {
* @minLength 1
*/
search?: string
+ /**
+ * Match any selected tag (OR) or all selected tags (AND). Additional tags are allowed.
+ *
+ * * `any` - Any
+ * * `all` - All
+ * @minLength 1
+ */
+ tag_match?: DatasetsListTagMatch
+ /**
+ * JSON array of tag names, for example ["quality","production"]. Up to 20 tags. An empty array disables tag filtering.
+ */
+ tags?: string
}
+export type DatasetsListTagMatch = (typeof DatasetsListTagMatch)[keyof typeof DatasetsListTagMatch]
+
+export const DatasetsListTagMatch = {
+ Any: 'any',
+ All: 'all',
+} as const
+
export type DatasetsRevisionsListParams = {
/**
* Number of results to return per page.
@@ -5361,8 +5459,28 @@ export type LlmAnalyticsScoreDefinitionsListParams = {
* Search scorers by name or description.
*/
search?: string
+ /**
+ * Match any selected tag (OR) or all selected tags (AND). Additional tags are allowed.
+ *
+ * * `any` - Any
+ * * `all` - All
+ * @minLength 1
+ */
+ tag_match?: LlmAnalyticsScoreDefinitionsListTagMatch
+ /**
+ * JSON array of tag names, for example ["quality","production"]. Up to 20 tags. An empty array disables tag filtering.
+ */
+ tags?: string
}
+export type LlmAnalyticsScoreDefinitionsListTagMatch =
+ (typeof LlmAnalyticsScoreDefinitionsListTagMatch)[keyof typeof LlmAnalyticsScoreDefinitionsListTagMatch]
+
+export const LlmAnalyticsScoreDefinitionsListTagMatch = {
+ Any: 'any',
+ All: 'all',
+} as const
+
export type LlmAnalyticsScoreDefinitionsVersionsListParams = {
/**
* Continuation cursor from the prior page.
diff --git a/products/ai_observability/frontend/generated/api.ts b/products/ai_observability/frontend/generated/api.ts
index 2f7cb90e4f82..3d85f12610c8 100644
--- a/products/ai_observability/frontend/generated/api.ts
+++ b/products/ai_observability/frontend/generated/api.ts
@@ -113,6 +113,7 @@ import type {
PatchedEvaluationReportUpdateApi,
PatchedLLMPromptPublishApi,
PatchedLLMProviderKeyApi,
+ PatchedOfflineExperimentMetadataApi,
PatchedParserRecipeApi,
PatchedReviewQueueItemUpdateApi,
PatchedReviewQueueUpdateApi,
@@ -339,6 +340,24 @@ export const aiObservabilityOfflineExperimentsRetrieve = async (
})
}
+export const getAiObservabilityOfflineExperimentsPartialUpdateUrl = (projectId: string, id: string) => {
+ return `/api/projects/${projectId}/ai_observability/offline_experiments/${id}/`
+}
+
+export const aiObservabilityOfflineExperimentsPartialUpdate = async (
+ projectId: string,
+ id: string,
+ patchedOfflineExperimentMetadataApi?: PatchedOfflineExperimentMetadataApi,
+ options?: RequestInit
+): Promise<OfflineExperimentReadApi> => {
+ return apiMutator<OfflineExperimentReadApi>(getAiObservabilityOfflineExperimentsPartialUpdateUrl(projectId, id), {
+ ...options,
+ method: 'PATCH',
+ headers: { 'Content-Type': 'application/json', ...options?.headers },
+ body: JSON.stringify(patchedOfflineExperimentMetadataApi),
+ })
+}
+
export const getAiObservabilityOfflineExperimentsCompleteCreateUrl = (projectId: string, id: string) => {
return `/api/projects/${projectId}/ai_observability/offline_experiments/${id}/complete/`
}
diff --git a/products/ai_observability/frontend/generated/api.zod.ts b/products/ai_observability/frontend/generated/api.zod.ts
index 7de95e1b4844..279164a67381 100644
--- a/products/ai_observability/frontend/generated/api.zod.ts
+++ b/products/ai_observability/frontend/generated/api.zod.ts
@@ -129,6 +129,18 @@ export const AiObservabilityOfflineExperimentsCreateBody = /* @__PURE__ */ zod.o
.describe('Version of the prompt under evaluation.'),
})
+export const aiObservabilityOfflineExperimentsPartialUpdateBodyTagsItemMax = 255
+
+export const aiObservabilityOfflineExperimentsPartialUpdateBodyTagsMax = 100
+
+export const AiObservabilityOfflineExperimentsPartialUpdateBody = /* @__PURE__ */ zod.object({
+ tags: zod
+ .array(zod.string().max(aiObservabilityOfflineExperimentsPartialUpdateBodyTagsItemMax))
+ .max(aiObservabilityOfflineExperimentsPartialUpdateBodyTagsMax)
+ .optional()
+ .describe('Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.'),
+})
+
export const aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemCaseKeyMax = 255
export const aiObservabilityOfflineExperimentsUploadCreateBodyItemsItemTrialMax = 255
@@ -416,12 +428,21 @@ export const DatasetItemsRestoreBody = /* @__PURE__ */ zod.object({
/**
* Create an empty dataset. Its first revision is created with its first item.
*/
+export const datasetsCreateBodyTagsItemMax = 255
+
+export const datasetsCreateBodyTagsMax = 100
+
export const datasetsCreateBodyNameMax = 400
export const datasetsCreateBodyDescriptionDefault = ``
export const datasetsCreateBodyDescriptionMax = 10000
export const DatasetsCreateBody = /* @__PURE__ */ zod.object({
+ tags: zod
+ .array(zod.string().max(datasetsCreateBodyTagsItemMax))
+ .max(datasetsCreateBodyTagsMax)
+ .optional()
+ .describe('Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.'),
name: zod.string().max(datasetsCreateBodyNameMax).describe('Dataset name. Names are unique within a project.'),
description: zod
.string()
@@ -437,11 +458,20 @@ export const DatasetsCreateBody = /* @__PURE__ */ zod.object({
/**
* Update descriptive dataset fields without changing its revision.
*/
+export const datasetsPartialUpdateBodyTagsItemMax = 255
+
+export const datasetsPartialUpdateBodyTagsMax = 100
+
export const datasetsPartialUpdateBodyNameMax = 400
export const datasetsPartialUpdateBodyDescriptionMax = 10000
export const DatasetsPartialUpdateBody = /* @__PURE__ */ zod.object({
+ tags: zod
+ .array(zod.string().max(datasetsPartialUpdateBodyTagsItemMax))
+ .max(datasetsPartialUpdateBodyTagsMax)
+ .optional()
+ .describe('Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.'),
name: zod
.string()
.max(datasetsPartialUpdateBodyNameMax)
@@ -2310,6 +2340,10 @@ export const LlmAnalyticsReviewQueuesPartialUpdateBody = /* @__PURE__ */ zod.obj
.describe('Human-readable queue name.'),
})
+export const llmAnalyticsScoreDefinitionsCreateBodyTagsItemMax = 255
+
+export const llmAnalyticsScoreDefinitionsCreateBodyTagsMax = 100
+
export const llmAnalyticsScoreDefinitionsCreateBodyNameMax = 255
export const llmAnalyticsScoreDefinitionsCreateBodyArchivedDefault = false
@@ -2320,6 +2354,11 @@ export const llmAnalyticsScoreDefinitionsCreateBodyConfigOneOneOptionsItemLabelM
export const llmAnalyticsScoreDefinitionsCreateBodyConfigOneOnePassingRuleOneCategoriesItemMax = 128
export const LlmAnalyticsScoreDefinitionsCreateBody = /* @__PURE__ */ zod.object({
+ tags: zod
+ .array(zod.string().max(llmAnalyticsScoreDefinitionsCreateBodyTagsItemMax))
+ .max(llmAnalyticsScoreDefinitionsCreateBodyTagsMax)
+ .optional()
+ .describe('Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.'),
name: zod.string().max(llmAnalyticsScoreDefinitionsCreateBodyNameMax).describe('Human-readable scorer name.'),
description: zod.string().nullish().describe('Optional human-readable description.'),
kind: zod
@@ -2435,9 +2474,18 @@ export const LlmAnalyticsScoreDefinitionsCreateBody = /* @__PURE__ */ zod.object
.describe('Initial immutable scorer configuration.'),
})
+export const llmAnalyticsScoreDefinitionsPartialUpdateBodyTagsItemMax = 255
+
+export const llmAnalyticsScoreDefinitionsPartialUpdateBodyTagsMax = 100
+
export const llmAnalyticsScoreDefinitionsPartialUpdateBodyNameMax = 255
export const LlmAnalyticsScoreDefinitionsPartialUpdateBody = /* @__PURE__ */ zod.object({
+ tags: zod
+ .array(zod.string().max(llmAnalyticsScoreDefinitionsPartialUpdateBodyTagsItemMax))
+ .max(llmAnalyticsScoreDefinitionsPartialUpdateBodyTagsMax)
+ .optional()
+ .describe('Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.'),
name: zod
.string()
.max(llmAnalyticsScoreDefinitionsPartialUpdateBodyNameMax)
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.stories.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.stories.tsx
index 4045ba752d9e..227859e77737 100644
--- a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.stories.tsx
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.stories.tsx
@@ -29,6 +29,10 @@ const meta: Meta<typeof OfflineExperimentsOverview> = {
),
mswDecorator({
get: {
+ '/api/projects/:team/tags': ({ request }) => {
+ const tags = ['production', 'quality', 'regression', 'support']
+ return new URL(request.url).searchParams.has('limit') ? { results: tags, next: null } : tags
+ },
'/api/projects/:team/ai_observability/offline_experiments/': {
count: 8,
next_cursor: null,
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.tsx
index 0380016e3c27..ed5074e5f3c6 100644
--- a/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.tsx
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineExperimentsOverview.tsx
@@ -17,6 +17,9 @@ import { TZLabel } from 'lib/components/TZLabel'
import type { LemonTableColumns } from 'lib/lemon-ui/LemonTable'
import { urls } from 'scenes/urls'
+import { ResourceTagFilter } from '../components/ResourceTagFilter'
+import { cleanTagFilters } from '../components/resourceTagFilters'
+import { ResourceTags } from '../components/ResourceTags'
import type { OfflineExperimentReadApi } from '../generated/api.schemas'
import { OfflineExperimentsEmptyState } from './OfflineExperimentsEmptyState'
import { offlineExperimentsLogic, type OfflineExperimentsLogicProps } from './offlineExperimentsLogic'
@@ -47,7 +50,8 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):
{
title: 'Experiment',
key: 'name',
- width: '26%',
+ className: '@max-[50rem]:w-1/3',
+ width: '22%',
render: (_, experiment) => (
<Tooltip title={`${experiment.name} · ${experiment.id}`}>
<Link
@@ -60,10 +64,25 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):
</Tooltip>
),
},
+ {
+ title: 'Tags',
+ key: 'tags',
+ width: '25%',
+ render: (_, experiment) => (
+ <ResourceTags
+ entity="experiments"
+ resourceId={experiment.id}
+ tags={experiment.tags}
+ onSaved={refresh}
+ onTagClick={(tag) => setFilters({ tags: [tag] })}
+ />
+ ),
+ },
{
title: 'Execution',
key: 'execution',
- width: '18%',
+ className: '@max-[50rem]:hidden',
+ width: '14%',
render: (_, experiment) => (
<div className="truncate">
<TZLabel time={experiment.started_at} />
@@ -73,6 +92,7 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):
{
title: 'Source',
key: 'source',
+ className: '@max-[50rem]:hidden',
width: '12%',
render: (_, experiment) => (
<Tooltip title={offlineRunSourceLabel(experiment.run_source)}>
@@ -83,13 +103,14 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):
{
title: 'Upload',
key: 'status',
- width: '16%',
+ width: '12%',
render: (_, experiment) => <OfflineExperimentStatus status={experiment.status} />,
},
{
title: <Tooltip title="Result and scorer counts include only results you can access.">Coverage</Tooltip>,
key: 'coverage',
- width: '28%',
+ className: '@max-[40rem]:hidden',
+ width: '15%',
render: (_, experiment) => (
<Tooltip
title={[
@@ -159,6 +180,7 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):
{ value: 'failed', label: 'Failed' },
]}
/>
+ <ResourceTagFilter filters={cleanTagFilters(filters)} onChange={setFilters} />
<LemonButton
type="secondary"
icon={<IconRefresh />}
@@ -232,6 +254,7 @@ export function OfflineExperimentsOverview(props: OfflineExperimentsLogicProps):
</LemonBanner>
) : (
<LemonTable
+ className="@max-[50rem]:[&_colgroup]:hidden @max-[50rem]:[&_table]:table-auto"
dataSource={experiments?.results || []}
columns={columns}
rowKey="id"
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistory.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistory.tsx
index 74b05aaabbb3..689edd6cd43d 100644
--- a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistory.tsx
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistory.tsx
@@ -13,6 +13,7 @@ import { SceneContent } from '~/layout/scenes/components/SceneContent'
import { SceneTitleSection } from '~/layout/scenes/components/SceneTitleSection'
import { AccessControlLevel, AccessControlResourceType } from '~/types'
+import { ResourceTagFilter } from '../components/ResourceTagFilter'
import { EvaluationsTabs } from '../evaluations/EvaluationsTabs'
import { offlineExperimentUrl } from './offlineExperimentPresentation'
import { offlineScorerHistoryLogic, type OfflineScorerHistoryProps } from './offlineScorerHistoryLogic'
@@ -86,7 +87,7 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem
) : !definition ? (
<Spinner />
) : (
- <div className="space-y-4 min-w-0">
+ <div className="space-y-4 min-w-0 @container/scorer-history">
<div className="flex items-center justify-between flex-wrap gap-2">
<div className="flex items-center flex-wrap gap-2">
<LemonTag>{definition.kind}</LemonTag>
@@ -134,6 +135,10 @@ export function OfflineScorerHistory(props: OfflineScorerHistoryProps): JSX.Elem
]}
/>
</div>
+ <div>
+ <LemonLabel>Tags</LemonLabel>
+ <ResourceTagFilter filters={filters} onChange={setFilters} />
+ </div>
<div>
<LemonLabel>Version</LemonLabel>
<LemonSelect
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryScene.stories.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryScene.stories.tsx
index 5f901979e35c..9b0f40552c75 100644
--- a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryScene.stories.tsx
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryScene.stories.tsx
@@ -21,6 +21,10 @@ const meta: Meta<typeof OfflineScorerHistoryScene> = {
decorators: [
mswDecorator({
get: {
+ '/api/projects/:team/tags': ({ request }) => {
+ const tags = ['quality', 'production', 'regression', 'support']
+ return new URL(request.url).searchParams.has('limit') ? { results: tags, next: null } : tags
+ },
'/api/projects/:project/llm_analytics/score_definitions/:definition/': OFFLINE_STORY_DEFINITION,
'/api/projects/:project/llm_analytics/score_definitions/:definition/versions/': {
results: [OFFLINE_STORY_VERSION],
@@ -30,7 +34,13 @@ const meta: Meta<typeof OfflineScorerHistoryScene> = {
'/api/projects/:project/llm_analytics/score_definitions/:definition/versions/:version/':
OFFLINE_STORY_VERSION,
'/api/projects/:project/ai_observability/offline_scorers/:definition/history/': {
- results: [...OFFLINE_STORY_POINTS].reverse(),
+ results: [...OFFLINE_STORY_POINTS].reverse().map((point, index) => ({
+ ...point,
+ experiment: {
+ ...point.experiment,
+ tags: index % 2 ? ['quality'] : ['quality', 'production', 'regression', 'support'],
+ },
+ })),
next_cursor: 'older-page',
count: 127,
},
diff --git a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryTable.tsx b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryTable.tsx
index 6e428d31c67b..d1d83974700d 100644
--- a/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryTable.tsx
+++ b/products/ai_observability/frontend/offline-evaluations/OfflineScorerHistoryTable.tsx
@@ -1,6 +1,7 @@
import { IconChevronDown, IconChevronRight } from '@posthog/icons'
import { LemonButton, LemonTable, Link, Tooltip } from '@posthog/lemon-ui'
+import { ObjectTags } from 'lib/components/ObjectTags/ObjectTags'
import { TZLabel } from 'lib/components/TZLabel'
import type { OfflineHistoryPageApi, OfflineHistoryPointApi } from '../generated/api.schemas'
@@ -23,12 +24,13 @@ export function OfflineScorerHistoryTable({
}): JSX.Element {
return (
<LemonTable<OfflineHistoryPointApi>
+ className="@max-[50rem]/scorer-history:[&_colgroup]:hidden @max-[50rem]/scorer-history:[&_table]:table-auto"
rowKey={(point) => `${point.experiment.id}:${point.summary.scorer.id}`}
dataSource={page?.results || []}
loading={loading || !page}
size="small"
tableLayout="fixed"
- emptyState="No results in this period. Try another version or date range."
+ emptyState="No experiments match these filters."
columns={[
{
title: 'Experiment',
@@ -58,10 +60,21 @@ export function OfflineScorerHistoryTable({
)
},
},
+ {
+ title: 'Tags',
+ key: 'tags',
+ width: '24%',
+ render: (_, point) => (
+ <div className="min-w-0 max-w-64">
+ <ObjectTags tags={point.experiment.tags} staticOnly maxVisibleTags={2} wrap />
+ </div>
+ ),
+ },
{
title: 'Execution',
key: 'execution',
- width: '20%',
+ className: '@max-[50rem]/scorer-history:hidden',
+ width: '16%',
render: (_, point) => (
<div className="truncate text-xs">
<TZLabel time={point.experiment.started_at} />
@@ -71,13 +84,14 @@ export function OfflineScorerHistoryTable({
{
title: 'Score',
key: 'score',
- width: '16%',
+ width: '12%',
render: (_, point) => <OfflineScoreSummaryDisplay summary={point.summary} />,
},
{
title: 'Coverage',
key: 'coverage',
- width: '36%',
+ className: '@max-[40rem]/scorer-history:hidden',
+ width: '20%',
render: (_, { summary }) => (
<Tooltip
title={`${summary.status_counts.ok} / ${summary.observed_item_count} scored items · ${summary.status_counts.error} errors · ${summary.status_counts.skipped} skipped · ${summary.status_counts.not_applicable} not applicable · ${summary.missing_result_count} missing`}
diff --git a/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.test.ts b/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.test.ts
index 0e8af0c8ec36..31848ff476e6 100644
--- a/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.test.ts
+++ b/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.test.ts
@@ -53,6 +53,47 @@ describe('offlineExperimentsLogic', () => {
afterEach(() => jest.useRealTimers())
+ it('shares experiment tag filters with trends and restores them from the URL', async () => {
+ const logic = offlineExperimentsLogic(props)
+ logic.mount()
+ logic.actions.setFilters({ tags: ['quality', 'production'], tag_match: 'all' })
+ await jest.advanceTimersByTimeAsync(400)
+ expect(router.values.searchParams).toMatchObject({ tags: ['quality', 'production'], tag_match: 'all' })
+ expect(list).toHaveBeenLastCalledWith(
+ String(props.teamId),
+ expect.objectContaining({ tags: '["quality","production"]', tag_match: 'all' })
+ )
+ expect(logic.values.trendFilters).toMatchObject({ tags: '["quality","production"]', tag_match: 'all' })
+ jest.mocked(api.aiObservabilityOfflineScorersHistoryList).mockResolvedValue({
+ results: overviewHistory(overviewScorers[0]),
+ count: 6,
+ next_cursor: null,
+ })
+ const trend = offlineOverviewTrendLogic({
+ teamId: props.teamId,
+ scorerId: overviewScorers[0].id,
+ dateFrom: logic.values.dateRange!.dateFrom,
+ dateTo: logic.values.dateRange!.dateTo,
+ refreshKey: 0,
+ filters: logic.values.trendFilters,
+ })
+ trend.mount()
+ await jest.advanceTimersByTimeAsync(400)
+ expect(api.aiObservabilityOfflineScorersHistoryList).toHaveBeenLastCalledWith(
+ String(props.teamId),
+ overviewScorers[0].id,
+ expect.objectContaining({ tags: '["quality","production"]', tag_match: 'all' })
+ )
+ router.actions.push(urls.aiObservabilityScorers())
+ router.actions.push(urls.aiObservabilityOfflineEvaluations())
+ await jest.advanceTimersByTimeAsync(400)
+ expect(logic.values.filters.tags).toEqual(['quality', 'production'])
+ expect(router.values.searchParams.tags).toEqual(['quality', 'production'])
+ router.actions.push(urls.aiObservabilityOfflineEvaluations(), { tags: [] })
+ await jest.advanceTimersByTimeAsync(400)
+ expect(logic.values.filters.tags).toEqual([])
+ })
+
it('keeps the active chart hover when another chart leaves and clears it when filters change', () => {
const logic = offlineExperimentsLogic(props)
logic.mount()
@@ -278,7 +319,12 @@ describe('offlineExperimentsLogic', () => {
run_source: 'local',
statuses: 'uploading',
})
- expect(logic.values.trendFilters).toEqual({ run_source: 'local', statuses: 'uploading' })
+ expect(logic.values.trendFilters).toEqual({
+ run_source: 'local',
+ statuses: 'uploading',
+ tags: '[]',
+ tag_match: 'any',
+ })
jest.setSystemTime(new Date('2026-09-28T15:00:00Z'))
logic.actions.nextPage('cursor-2')
@@ -298,7 +344,12 @@ describe('offlineExperimentsLogic', () => {
logic.actions.setFilters({ date_from: '-7d', run_source: 'ci', statuses: 'failed' })
await jest.advanceTimersByTimeAsync(150)
expect(logic.values.cursorStack).toEqual([])
- expect(logic.values.trendFilters).toEqual({ run_source: 'ci', statuses: 'failed' })
+ expect(logic.values.trendFilters).toEqual({
+ run_source: 'ci',
+ statuses: 'failed',
+ tags: '[]',
+ tag_match: 'any',
+ })
expect(list.mock.calls.at(-1)?.[1]).toMatchObject({
date_from: logic.values.dateRange?.dateFrom,
date_to: logic.values.dateRange?.dateTo,
@@ -322,7 +373,12 @@ describe('offlineExperimentsLogic', () => {
await jest.advanceTimersByTimeAsync(150)
expect(logic.values.cursorStack).toEqual(['cursor-2'])
expect(logic.values.dateRange).toEqual(initialRange)
- expect(logic.values.trendFilters).toEqual({ run_source: 'local', statuses: 'uploading' })
+ expect(logic.values.trendFilters).toEqual({
+ run_source: 'local',
+ statuses: 'uploading',
+ tags: '[]',
+ tag_match: 'any',
+ })
expect(list.mock.calls.at(-1)?.[1]).toMatchObject({
date_from: initialRange?.dateFrom,
date_to: initialRange?.dateTo,
@@ -394,7 +450,11 @@ describe('offlineExperimentsLogic', () => {
expect(logic.values.experiments).toEqual(page)
expect(logic.values.experimentsError).toBeNull()
expect(logic.values.dateRange).not.toBeNull()
- expect(logic.values.trendFilters).toEqual({ statuses: 'completed,uploading,failed' })
+ expect(logic.values.trendFilters).toEqual({
+ statuses: 'completed,uploading,failed',
+ tags: '[]',
+ tag_match: 'any',
+ })
expect(logic.values.hasExperiments).toBe(true)
expect(logic.values.scorerIds).toEqual(overviewScorers.map(({ id }) => id))
})
diff --git a/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.ts b/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.ts
index ef8f01b6bf1c..5e83509d3697 100644
--- a/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.ts
+++ b/products/ai_observability/frontend/offline-evaluations/offlineExperimentsLogic.ts
@@ -17,6 +17,7 @@ import { actionToUrl, router, urlToAction } from 'kea-router'
import { dayjs } from 'lib/dayjs'
import { urls } from 'scenes/urls'
+import { cleanTagFilters, restoreTagFilters, saveTagFilters } from '../components/resourceTagFilters'
import * as api from '../generated/api'
import type {
OfflineExperimentPageApi,
@@ -249,6 +250,7 @@ export const offlineExperimentsLogic: LogicWrapper<offlineExperimentsLogicType>
}
const page = await api.aiObservabilityOfflineExperimentsList(String(props.teamId), {
...filters,
+ tags: filters.tags?.length ? JSON.stringify(filters.tags) : undefined,
date_from: range.dateFrom,
date_to: range.dateTo,
cursor: values.cursor || undefined,
@@ -342,7 +344,11 @@ export const offlineExperimentsLogic: LogicWrapper<offlineExperimentsLogicType>
{ date_from: '-30d' } as OfflineExperimentFilters,
{
setFilters: (state, { filters }) => cleanOfflineFilters({ ...state, ...filters }),
- hydrateUrl: (_, { search }) => ({ date_from: '-30d', ...offlineFiltersFromUrl(search) }),
+ hydrateUrl: (_, { search }) => ({
+ date_from: '-30d',
+ ...offlineFiltersFromUrl(search),
+ ...restoreTagFilters('experiments', search, props.teamId, props.userId),
+ }),
},
],
hasExperiments: [null as boolean | null, { setHasExperiments: (_, { hasExperiments }) => hasExperiments }],
@@ -442,6 +448,8 @@ export const offlineExperimentsLogic: LogicWrapper<offlineExperimentsLogicType>
(filters: OfflineExperimentFilters): OfflineOverviewTrendFilters => ({
run_source: filters.run_source,
statuses: filters.statuses || OFFLINE_ALL_UPLOAD_STATES,
+ tags: JSON.stringify(filters.tags || []),
+ tag_match: filters.tag_match || 'any',
}),
],
trendXDomain: [
@@ -486,7 +494,9 @@ export const offlineExperimentsLogic: LogicWrapper<offlineExperimentsLogicType>
trend.query.dateTo !== dateRange.dateTo ||
trend.query.refreshKey !== refreshKey ||
trend.query.filters?.run_source !== filters.run_source ||
- (trend.query.filters?.statuses || OFFLINE_ALL_UPLOAD_STATES) !== filters.statuses
+ (trend.query.filters?.statuses || OFFLINE_ALL_UPLOAD_STATES) !== filters.statuses ||
+ (trend.query.filters?.tags || '[]') !== filters.tags ||
+ (trend.query.filters?.tag_match || 'any') !== filters.tag_match
) {
return []
}
@@ -501,12 +511,14 @@ export const offlineExperimentsLogic: LogicWrapper<offlineExperimentsLogicType>
}),
listeners(({ actions, values, props }) => ({
setFilters: () => {
+ saveTagFilters('experiments', cleanTagFilters(values.filters), props.teamId, props.userId)
actions.loadOfflineExperiments()
if (!values.scorerIds?.length && values.dateRange && values.suggestedScorers === null) {
actions.loadOfflineSuggestedScorers()
}
},
hydrateUrl: () => {
+ saveTagFilters('experiments', cleanTagFilters(values.filters), props.teamId, props.userId)
actions.loadOfflineExperiments()
if (!values.scorerIds?.length && values.dateRange) {
if (values.suggestedScorers) {
@@ -568,6 +580,7 @@ export const offlineExperimentsLogic: LogicWrapper<offlineExperimentsLogicType>
]
return {
setFilters: location,
+ hydrateUrl: location,
setScorerIds: location,
suggestScorerIds: location,
nextPage: location,
diff --git a/products/ai_observability/frontend/offline-evaluations/offlineOverviewFixtures.ts b/products/ai_observability/frontend/offline-evaluations/offlineOverviewFixtures.ts
index c6f9962c2c15..b32f36a60570 100644
--- a/products/ai_observability/frontend/offline-evaluations/offlineOverviewFixtures.ts
+++ b/products/ai_observability/frontend/offline-evaluations/offlineOverviewFixtures.ts
@@ -2,6 +2,7 @@ import type { OfflineExperimentReadApi, OfflineHistoryPointApi, ScoreDefinitionA
export const overviewScorers: ScoreDefinitionApi[] = ['Answer quality', 'Groundedness', 'Response time'].map(
(name, index) => ({
+ tags: ['regression', 'production', 'quality'],
id: `11111111-1111-4111-8111-${String(index + 1).padStart(12, '0')}`,
name,
description: '',
@@ -25,6 +26,7 @@ export const overviewScorers: ScoreDefinitionApi[] = ['Answer quality', 'Grounde
)
export const overviewExperiments: OfflineExperimentReadApi[] = Array.from({ length: 8 }, (_, index) => ({
+ tags: ['regression', 'production', 'quality'],
id: `33333333-3333-4333-8333-${String(index + 1).padStart(12, '0')}`,
name: ['Support assistant prompt update', 'Tool selection regression', 'Response quality baseline'][index % 3],
run_source: index % 3 === 0 ? 'ci' : index % 3 === 1 ? 'local' : null,
diff --git a/products/ai_observability/frontend/offline-evaluations/offlineOverviewState.ts b/products/ai_observability/frontend/offline-evaluations/offlineOverviewState.ts
index c1254914ce81..968d2a83ce06 100644
--- a/products/ai_observability/frontend/offline-evaluations/offlineOverviewState.ts
+++ b/products/ai_observability/frontend/offline-evaluations/offlineOverviewState.ts
@@ -1,5 +1,6 @@
import { ConcurrencyController } from 'lib/utils/concurrencyController'
+import { cleanTagFilters, type ResourceTagFilters } from '../components/resourceTagFilters'
import type {
AiObservabilityOfflineExperimentsListParams,
AiObservabilityOfflineScorersHistoryListParams,
@@ -8,11 +9,12 @@ import type {
export type OfflineExperimentFilters = Pick<
AiObservabilityOfflineExperimentsListParams,
'search' | 'statuses' | 'run_source' | 'date_from' | 'date_to'
->
+> &
+ Partial<ResourceTagFilters>
export type OfflineOverviewTrendFilters = Pick<
AiObservabilityOfflineScorersHistoryListParams,
- 'run_source' | 'statuses'
+ 'run_source' | 'statuses' | 'tags' | 'tag_match'
>
export const OFFLINE_ALL_UPLOAD_STATES = 'completed,uploading,failed'
@@ -66,13 +68,16 @@ export function offlineFiltersFromUrl(search: Record<string, unknown>): OfflineE
date_to: search.date_to ?? search.trend_to,
run_source: search.run_source ?? search.trend_run_source,
}
- return Object.fromEntries(
- FILTER_KEYS.flatMap((key) => {
- const raw = shared[key]
- const value = key === 'search' && Array.isArray(raw) && raw.length === 1 ? raw[0] : raw
- return typeof value === 'string' && value ? [[key, value]] : []
- })
- )
+ return {
+ ...('tags' in search || 'tag_match' in search ? cleanTagFilters(search) : {}),
+ ...Object.fromEntries(
+ FILTER_KEYS.flatMap((key) => {
+ const raw = shared[key]
+ const value = key === 'search' && Array.isArray(raw) && raw.length === 1 ? raw[0] : raw
+ return typeof value === 'string' && value ? [[key, value]] : []
+ })
+ ),
+ }
}
export function offlineFiltersToUrl(filters: OfflineExperimentFilters): Record<string, unknown> {
diff --git a/products/ai_observability/frontend/offline-evaluations/offlineOverviewTrendLogic.test.ts b/products/ai_observability/frontend/offline-evaluations/offlineOverviewTrendLogic.test.ts
index 0bc11f5d2d67..ddf9b1a561e2 100644
--- a/products/ai_observability/frontend/offline-evaluations/offlineOverviewTrendLogic.test.ts
+++ b/products/ai_observability/frontend/offline-evaluations/offlineOverviewTrendLogic.test.ts
@@ -178,11 +178,11 @@ describe('offlineOverviewTrendLogic', () => {
expect(logic.values.periodSummary?.mean).toBe(0.64)
})
- it('reloads a shared source and upload state within the same frozen date range', async () => {
+ it('reloads shared filters including tag-only changes within the same frozen date range', async () => {
const logic = offlineOverviewTrendLogic({ ...props, filters: { run_source: 'local' } })
logic.mount()
await expectLogic(logic).toFinishAllListeners()
- const filters = { run_source: 'ci', statuses: 'failed' }
+ const filters = { run_source: 'ci', statuses: 'failed', tags: '["quality"]', tag_match: 'any' as const }
offlineOverviewTrendLogic({ ...props, filters })
await expectLogic(logic).toFinishAllListeners()
@@ -193,5 +193,21 @@ describe('offlineOverviewTrendLogic', () => {
date_to: props.dateTo,
limit: 100,
})
+ offlineOverviewTrendLogic({
+ ...props,
+ filters: { ...filters, tags: '["quality","production"]', tag_match: 'all' },
+ })
+ await expectLogic(logic).toFinishAllListeners()
+ expect(history).toHaveBeenCalledTimes(3)
+ expect(history).toHaveBeenLastCalledWith(
+ '997',
+ props.scorerId,
+ expect.objectContaining({
+ tags: '["quality","production"]',
+ tag_match: 'all',
+ date_from: props.dateFrom,
+ date_to: props.dateTo,
+ })
+ )
})
})
diff --git a/products/ai_observability/frontend/offline-evaluations/offlineScoreTrends.fixtures.ts b/products/ai_observability/frontend/offline-evaluations/offlineScoreTrends.fixtures.ts
index 5f0fa6ed912c..a5ce2b5d72e7 100644
--- a/products/ai_observability/frontend/offline-evaluations/offlineScoreTrends.fixtures.ts
+++ b/products/ai_observability/frontend/offline-evaluations/offlineScoreTrends.fixtures.ts
@@ -1,6 +1,7 @@
import type { OfflineHistoryPointApi, ScoreDefinitionApi, ScoreDefinitionVersionApi } from '../generated/api.schemas'
export const OFFLINE_STORY_DEFINITION: ScoreDefinitionApi = {
+ tags: [],
id: '10000000-0000-4000-8000-000000000001',
name: 'Answer relevance',
description: 'Relevance to the question, scored from 0 to 5.',
@@ -28,6 +29,7 @@ export const OFFLINE_STORY_VERSION: ScoreDefinitionVersionApi = {
export function makeOfflineHistoryPoint(index: number, mean: number | null = 4.2): OfflineHistoryPointApi {
return {
experiment: {
+ tags: [],
id: `30000000-0000-4000-8000-${String(index).padStart(12, '0')}`,
name: `Answer evaluation ${index}`,
run_source: index % 2 === 0 ? 'ci' : 'local',
diff --git a/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.test.ts b/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.test.ts
index 5496073387b6..5e45d06d83cf 100644
--- a/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.test.ts
+++ b/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.test.ts
@@ -6,6 +6,7 @@ import { urls } from 'scenes/urls'
import { initKeaTests } from '~/test/init'
+import { saveTagFilters } from '../components/resourceTagFilters'
import * as api from '../generated/api'
import { offlineScorerHistoryLogic, offlineScorerHistoryUrl } from './offlineScorerHistoryLogic'
import { OFFLINE_STORY_DEFINITION, OFFLINE_STORY_VERSION, makeOfflineHistoryPoint } from './offlineScoreTrends.fixtures'
@@ -21,6 +22,7 @@ describe('offlineScorerHistoryLogic', () => {
beforeEach(() => {
initKeaTests()
jest.resetAllMocks()
+ localStorage.clear()
jest.mocked(api.llmAnalyticsScoreDefinitionsRetrieve).mockResolvedValue(OFFLINE_STORY_DEFINITION)
jest.mocked(api.llmAnalyticsScoreDefinitionsVersionsList).mockResolvedValue({
results: [OFFLINE_STORY_VERSION],
@@ -35,6 +37,43 @@ describe('offlineScorerHistoryLogic', () => {
})
})
+ it('shares tag filters across periods, persists them, and lets an explicit URL clear them', async () => {
+ const props = { scorerId: OFFLINE_STORY_DEFINITION.id, teamId: 1 }
+ router.actions.push(urls.aiObservabilityOfflineScorerHistory(props.scorerId))
+ const logic = offlineScorerHistoryLogic(props)
+ logic.mount()
+ await expectLogic(logic).toFinishAllListeners()
+ await expectLogic(logic, () => logic.actions.setFilters({ compare: 'previous' })).toFinishAllListeners()
+ await expectLogic(logic, () => logic.actions.changePage('primary', 'older')).toFinishAllListeners()
+ await expectLogic(logic, () => logic.actions.changePage('comparison', 'older')).toFinishAllListeners()
+ jest.mocked(api.aiObservabilityOfflineScorersHistoryList).mockClear()
+ await expectLogic(logic, () =>
+ logic.actions.setFilters({ tags: ['quality', 'production'], tag_match: 'all' })
+ ).toFinishAllListeners()
+ expect(logic.values.primaryCursors).toEqual([null])
+ expect(logic.values.comparisonCursors).toEqual([null])
+ expect(router.values.searchParams).toMatchObject({ tags: ['quality', 'production'], tag_match: 'all' })
+ const calls = jest.mocked(api.aiObservabilityOfflineScorersHistoryList).mock.calls
+ expect(calls).toHaveLength(2)
+ for (const [, , query] of calls) {
+ expect(query).toMatchObject({ tags: '["quality","production"]', tag_match: 'all' })
+ expect(query?.cursor).toBeUndefined()
+ }
+ router.actions.push(urls.aiObservabilityScorers())
+ router.actions.push(urls.aiObservabilityOfflineScorerHistory(props.scorerId))
+ await expectLogic(logic).toFinishAllListeners()
+ expect(logic.values.filters.tags).toEqual(['quality', 'production'])
+ expect(router.values.searchParams).toMatchObject({ tags: ['quality', 'production'], tag_match: 'all' })
+ router.actions.push(urls.aiObservabilityOfflineScorerHistory(props.scorerId), { tags: [] })
+ await expectLogic(logic).toFinishAllListeners()
+ expect(logic.values.filters.tags).toEqual([])
+ expect(api.aiObservabilityOfflineScorersHistoryList).toHaveBeenLastCalledWith(
+ '1',
+ props.scorerId,
+ expect.objectContaining({ tags: '[]' })
+ )
+ })
+
it('keeps the periods cursors independent and freezes both resolved windows while paging', async () => {
const logic = offlineScorerHistoryLogic({ scorerId: OFFLINE_STORY_DEFINITION.id, teamId: 1 })
logic.mount()
@@ -145,6 +184,7 @@ describe('offlineScorerHistoryLogic', () => {
] as const)(
'includes %s experiments started %i days ago when opened from their scorer link',
async (status, age) => {
+ saveTagFilters('experiments', { tags: ['production'], tag_match: 'all' }, 1)
const startedAt = dayjs().subtract(age, 'day').toISOString()
router.actions.push(
offlineScorerHistoryUrl(
@@ -160,6 +200,7 @@ describe('offlineScorerHistoryLogic', () => {
expect(query?.scorer_version_ids).toBe(OFFLINE_STORY_VERSION.id)
expect(query?.statuses?.split(',')).toContain(status)
expect(dayjs(query?.date_from).isAfter(startedAt)).toBe(false)
+ expect(query?.tags).toBe('[]')
}
)
diff --git a/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.ts b/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.ts
index 67412bc08aaf..b06fc1d38828 100644
--- a/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.ts
+++ b/products/ai_observability/frontend/offline-evaluations/offlineScorerHistoryLogic.ts
@@ -21,6 +21,12 @@ import { objectsEqual } from 'lib/utils/objects'
import { teamLogic } from 'scenes/teamLogic'
import { urls } from 'scenes/urls'
+import {
+ cleanTagFilters,
+ restoreTagFilters,
+ saveTagFilters,
+ type ResourceTagFilters,
+} from '../components/resourceTagFilters'
import * as api from '../generated/api'
import type {
AiObservabilityOfflineScorersHistoryListParams,
@@ -43,7 +49,7 @@ export interface OfflineScorerHistoryProps {
scorerId: string
teamId: number
}
-export interface OfflineHistoryFilters {
+export interface OfflineHistoryFilters extends ResourceTagFilters {
version: string
date_from: string
date_to: string | null
@@ -63,6 +69,8 @@ const DEFAULT_FILTERS: OfflineHistoryFilters = {
compare_to: '-30d',
statuses: 'completed',
run_source: '',
+ tags: [],
+ tag_match: 'any',
}
function readFilters(search: Record<string, unknown>): OfflineHistoryFilters {
@@ -81,7 +89,7 @@ function readFilters(search: Record<string, unknown>): OfflineHistoryFilters {
if (filters.date_from === 'all' && filters.compare === 'previous') {
filters.compare = 'none'
}
- return filters
+ return { ...filters, ...cleanTagFilters(search) }
}
export function offlineScorerHistoryUrl(
@@ -93,6 +101,7 @@ export function offlineScorerHistoryUrl(
if (!experiment) {
return combineUrl(urls.aiObservabilityOfflineScorerHistory(scorer.definition_id), search).url
}
+ search.tags = []
if (experiment.status !== DEFAULT_FILTERS.statuses) {
search.statuses = OFFLINE_ALL_UPLOAD_STATES
}
@@ -516,6 +525,8 @@ export const offlineScorerHistoryLogic = kea<offlineScorerHistoryLogicType>([
limit: 50,
statuses: filters.statuses,
run_source: filters.run_source || undefined,
+ tags: JSON.stringify(filters.tags),
+ tag_match: filters.tag_match,
}),
],
trendPeriods: [
@@ -574,8 +585,9 @@ export const offlineScorerHistoryLogic = kea<offlineScorerHistoryLogicType>([
].map((version) => ({ value: version.id, label: `Version ${version.version}` })),
],
}),
- sharedListeners(({ actions, values }) => ({
+ sharedListeners(({ actions, values, props }) => ({
loadFilteredHistory: () => {
+ saveTagFilters('experiments', cleanTagFilters(values.filters), props.teamId)
if (values.filters.version) {
if (values.selectedVersion?.id !== values.filters.version) {
actions.loadOfflineHistoryVersion()
@@ -609,8 +621,8 @@ export const offlineScorerHistoryLogic = kea<offlineScorerHistoryLogicType>([
actions.loadOfflineHistoryVersions({})
},
})),
- actionToUrl(({ props, values }) => ({
- setFilters: () => [
+ actionToUrl(({ props, values }) => {
+ const location = (): [string, Record<string, unknown>, Record<string, never>, { replace: boolean }] => [
urls.aiObservabilityOfflineScorerHistory(props.scorerId),
Object.fromEntries(
Object.entries({ ...values.filters, compare_to: values.filters.compare_to ?? 'now' }).filter(
@@ -619,12 +631,13 @@ export const offlineScorerHistoryLogic = kea<offlineScorerHistoryLogicType>([
),
{},
{ replace: true },
- ],
- })),
+ ]
+ return { setFilters: location, replaceFilters: location }
+ }),
urlToAction(({ actions, values, props }) => ({
[urls.aiObservabilityOfflineScorerHistory(':scorerId', false)]: ({ scorerId }, search) => {
if (scorerId === props.scorerId) {
- const filters = readFilters(search)
+ const filters = readFilters({ ...search, ...restoreTagFilters('experiments', search, props.teamId) })
if (!filters.version && values.definition?.current_version_id) {
filters.version = values.definition.current_version_id
}
@@ -634,8 +647,9 @@ export const offlineScorerHistoryLogic = kea<offlineScorerHistoryLogicType>([
}
},
})),
- afterMount(({ actions }) => {
- actions.replaceFilters(readFilters(router.values.searchParams))
+ afterMount(({ actions, props }) => {
+ const search = router.values.searchParams
+ actions.replaceFilters(readFilters({ ...search, ...restoreTagFilters('experiments', search, props.teamId) }))
actions.loadOfflineHistoryDefinition()
actions.loadOfflineHistoryVersions({})
}),
diff --git a/products/ai_observability/frontend/scoreDefinitions/AIObservabilityScoreDefinitions.tsx b/products/ai_observability/frontend/scoreDefinitions/AIObservabilityScoreDefinitions.tsx
index 415ac9785816..7d05df0e80c0 100644
--- a/products/ai_observability/frontend/scoreDefinitions/AIObservabilityScoreDefinitions.tsx
+++ b/products/ai_observability/frontend/scoreDefinitions/AIObservabilityScoreDefinitions.tsx
@@ -23,6 +23,8 @@ import { urls } from 'scenes/urls'
import { updatedAtColumn } from '~/lib/lemon-ui/LemonTable/columnUtils'
import { AccessControlLevel, AccessControlResourceType } from '~/types'
+import { ResourceTagFilter } from '../components/ResourceTagFilter'
+import { ResourceTags } from '../components/ResourceTags'
import type {
ScoreDefinitionKindEnumApi as ScoreDefinitionKind,
ScoreDefinitionApi as ScoreDefinition,
@@ -91,6 +93,7 @@ export function AIObservabilityScoreDefinitions(): JSX.Element {
title: 'Kind',
dataIndex: 'kind',
key: 'kind',
+ className: '@max-[40rem]/scorers:hidden',
render: function renderKind(kind) {
return <LemonTag type="muted">{formatKindLabel(kind as ScoreDefinitionKind)}</LemonTag>
},
@@ -99,6 +102,7 @@ export function AIObservabilityScoreDefinitions(): JSX.Element {
title: 'Version',
dataIndex: 'current_version',
key: 'current_version',
+ className: '@max-[40rem]/scorers:hidden',
render: function renderVersion(version, scoreDefinition) {
return scoreDefinition.current_version_id ? (
<CopyToClipboardInline
@@ -125,6 +129,20 @@ export function AIObservabilityScoreDefinitions(): JSX.Element {
)
},
},
+ {
+ title: 'Tags',
+ key: 'tags',
+ width: '25%',
+ render: (_, scoreDefinition) => (
+ <ResourceTags
+ entity="scorers"
+ resourceId={scoreDefinition.id}
+ tags={scoreDefinition.tags}
+ onSaved={() => loadScoreDefinitions(false)}
+ onTagClick={(tag) => setFilters({ tags: [tag] }, true, false)}
+ />
+ ),
+ },
{
...(updatedAtColumn<ScoreDefinition>() as LemonTableColumn<
ScoreDefinition,
@@ -159,6 +177,7 @@ export function AIObservabilityScoreDefinitions(): JSX.Element {
className="min-w-64"
data-attr="llma-scorers-search-input"
/>
+ <ResourceTagFilter filters={filters} onChange={(tags) => setFilters(tags, true, false)} />
<LemonSelect<ScoreDefinitionKind | ''>
value={filters.kind}
onChange={(value) => setFilters({ kind: value || '' })}
@@ -201,6 +220,7 @@ export function AIObservabilityScoreDefinitions(): JSX.Element {
)}
{!scoreDefinitionsError && (
<LemonTable
+ className="@max-[50rem]/scorers:[&_colgroup]:hidden"
loading={scoreDefinitionsLoading}
columns={columns}
dataSource={scoreDefinitions.results}
diff --git a/products/ai_observability/frontend/scoreDefinitions/AIObservabilityScorerScene.stories.tsx b/products/ai_observability/frontend/scoreDefinitions/AIObservabilityScorerScene.stories.tsx
index 9fbe5ae03b2e..e65a7994ab55 100644
--- a/products/ai_observability/frontend/scoreDefinitions/AIObservabilityScorerScene.stories.tsx
+++ b/products/ai_observability/frontend/scoreDefinitions/AIObservabilityScorerScene.stories.tsx
@@ -9,6 +9,7 @@ import type { ScoreDefinitionApi } from '../generated/api.schemas'
import { AIObservabilityScorerScene } from './AIObservabilityScorerScene'
const definition: ScoreDefinitionApi = {
+ tags: [],
id: 'b4f331fe-7d3b-4bbe-9f88-52e1b0a8b000',
name: 'Answer quality',
description: 'How well the response answers the question, scored from 0 to 1.',
diff --git a/products/ai_observability/frontend/scoreDefinitions/AIObservabilityScorersScene.stories.tsx b/products/ai_observability/frontend/scoreDefinitions/AIObservabilityScorersScene.stories.tsx
index 3ecb14dedb67..6146e68b4f80 100644
--- a/products/ai_observability/frontend/scoreDefinitions/AIObservabilityScorersScene.stories.tsx
+++ b/products/ai_observability/frontend/scoreDefinitions/AIObservabilityScorersScene.stories.tsx
@@ -9,6 +9,7 @@ import type { ScoreDefinitionApi } from '../generated/api.schemas'
import { AIObservabilityScorersScene } from './AIObservabilityScorersScene'
const definition: ScoreDefinitionApi = {
+ tags: ['production', 'quality', 'regression', 'support', 'long-tag-name-for-overflow'],
id: 'b4f331fe-7d3b-4bbe-9f88-52e1b0a8b000',
name: 'Answer quality',
description: 'Quality of generated answers',
@@ -34,6 +35,10 @@ const meta: Meta<typeof AIObservabilityScorersScene> = {
decorators: [
mswDecorator({
get: {
+ '/api/projects/:team/tags': ({ request }) => {
+ const tags = ['production', 'quality', 'regression', 'support']
+ return new URL(request.url).searchParams.has('limit') ? { results: tags, next: null } : tags
+ },
'/api/projects/:team/llm_analytics/score_definitions/': {
results: [
definition,
diff --git a/products/ai_observability/frontend/scoreDefinitions/aiObservabilityScoreDefinitionsLogic.test.ts b/products/ai_observability/frontend/scoreDefinitions/aiObservabilityScoreDefinitionsLogic.test.ts
index cece289f2934..d6811dc3ae6c 100644
--- a/products/ai_observability/frontend/scoreDefinitions/aiObservabilityScoreDefinitionsLogic.test.ts
+++ b/products/ai_observability/frontend/scoreDefinitions/aiObservabilityScoreDefinitionsLogic.test.ts
@@ -3,10 +3,13 @@ import { MOCK_DEFAULT_TEAM } from '~/lib/api.mock'
import { router } from 'kea-router'
import { expectLogic } from 'kea-test-utils'
+import { ApiConfig } from 'lib/api'
+import { teamLogic } from 'scenes/teamLogic'
import { urls } from 'scenes/urls'
import { initKeaTests } from '~/test/init'
+import { saveTagFilters } from '../components/resourceTagFilters'
import {
llmAnalyticsScoreDefinitionsList as aiObservabilityScoreDefinitionsList,
llmAnalyticsScoreDefinitionsPartialUpdate as aiObservabilityScoreDefinitionsPartialUpdate,
@@ -31,6 +34,7 @@ const mockAIObservabilityScoreDefinitionsPartialUpdate =
>
const mockScoreDefinition: ScoreDefinition = {
+ tags: [],
id: 'score_def_1',
name: 'Quality',
description: 'Reusable quality scorer',
@@ -60,6 +64,7 @@ const mockScoreDefinition: ScoreDefinition = {
describe('aiObservabilityScoreDefinitionsLogic', () => {
beforeEach(() => {
initKeaTests()
+ localStorage.clear()
jest.resetAllMocks()
mockAIObservabilityScoreDefinitionsList.mockResolvedValue({
results: [mockScoreDefinition],
@@ -86,6 +91,33 @@ describe('aiObservabilityScoreDefinitionsLogic', () => {
})
})
+ it.each(['saved', 'url'])('waits for the project before restoring %s tag filters', async (source) => {
+ const selection = { tags: ['quality'], tag_match: 'all' as const }
+ saveTagFilters('scorers', selection)
+ router.actions.push(urls.aiObservabilityScorers(), source === 'url' ? { tags: ['shared'] } : {})
+ teamLogic.actions.loadCurrentTeamSuccess(null)
+ const currentTeamId = jest.spyOn(ApiConfig, 'getCurrentTeamId').mockImplementation(() => {
+ throw new Error('Team ID is not known.')
+ })
+ const logic = aiObservabilityScoreDefinitionsLogic()
+ try {
+ expect(() => logic.mount()).not.toThrow()
+ await expectLogic(logic).toFinishAllListeners()
+ expect(mockAIObservabilityScoreDefinitionsList).not.toHaveBeenCalled()
+ } finally {
+ currentTeamId.mockRestore()
+ }
+
+ await expectLogic(logic, () =>
+ teamLogic.actions.loadCurrentTeamSuccess(MOCK_DEFAULT_TEAM)
+ ).toFinishAllListeners()
+ expect(logic.values.filters.tags).toEqual(source === 'url' ? ['shared'] : ['quality'])
+ expect(mockAIObservabilityScoreDefinitionsList).toHaveBeenLastCalledWith(
+ String(MOCK_DEFAULT_TEAM.id),
+ expect.objectContaining({ tags: JSON.stringify(source === 'url' ? ['shared'] : ['quality']), offset: 0 })
+ )
+ })
+
it('updates filters and resets pagination', async () => {
const logic = aiObservabilityScoreDefinitionsLogic()
logic.mount()
@@ -96,6 +128,8 @@ describe('aiObservabilityScoreDefinitionsLogic', () => {
}).toFinishAllListeners()
expect(logic.values.filters).toEqual({
+ tags: [],
+ tag_match: 'any',
page: 1,
search: 'quality',
kind: 'categorical',
@@ -112,6 +146,28 @@ describe('aiObservabilityScoreDefinitionsLogic', () => {
})
})
+ it('restores tag filters on return and lets shared URLs override them', async () => {
+ router.actions.push(urls.aiObservabilityScorers())
+ const logic = aiObservabilityScoreDefinitionsLogic()
+ logic.mount()
+ await expectLogic(logic, () =>
+ logic.actions.setFilters({ tags: ['quality', 'production'], tag_match: 'all' }, true, false)
+ ).toFinishAllListeners()
+ expect(router.values.searchParams).toMatchObject({ tags: ['quality', 'production'], tag_match: 'all' })
+ expect(mockAIObservabilityScoreDefinitionsList).toHaveBeenLastCalledWith(
+ String(MOCK_DEFAULT_TEAM.id),
+ expect.objectContaining({ tags: '["quality","production"]', tag_match: 'all' })
+ )
+ router.actions.push(urls.aiObservabilityDatasets())
+ router.actions.push(urls.aiObservabilityScorers())
+ await expectLogic(logic).toFinishAllListeners()
+ expect(logic.values.filters.tags).toEqual(['quality', 'production'])
+ expect(router.values.searchParams.tags).toEqual(['quality', 'production'])
+ router.actions.push(urls.aiObservabilityScorers(), { tags: ['other'], tag_match: 'any' })
+ await expectLogic(logic).toFinishAllListeners()
+ expect(logic.values.filters.tags).toEqual(['other'])
+ })
+
it('does not show the empty count label while loading', async () => {
let resolveRequest:
| ((value: Awaited<ReturnType<typeof aiObservabilityScoreDefinitionsList>>) => void)
diff --git a/products/ai_observability/frontend/scoreDefinitions/aiObservabilityScoreDefinitionsLogic.ts b/products/ai_observability/frontend/scoreDefinitions/aiObservabilityScoreDefinitionsLogic.ts
index 15d94979f138..8b136a5c8ad9 100644
--- a/products/ai_observability/frontend/scoreDefinitions/aiObservabilityScoreDefinitionsLogic.ts
+++ b/products/ai_observability/frontend/scoreDefinitions/aiObservabilityScoreDefinitionsLogic.ts
@@ -14,6 +14,12 @@ import { PaginationManual } from '~/lib/lemon-ui/PaginationControl'
import { trackedActionToUrl } from '~/lib/logic/scenes/trackedActionToUrl'
import { urls } from '~/scenes/urls'
+import {
+ cleanTagFilters,
+ restoreTagFilters,
+ saveTagFilters,
+ type ResourceTagFilters,
+} from '../components/resourceTagFilters'
import {
llmAnalyticsScoreDefinitionsList as aiObservabilityScoreDefinitionsList,
llmAnalyticsScoreDefinitionsPartialUpdate as aiObservabilityScoreDefinitionsPartialUpdate,
@@ -27,7 +33,7 @@ import { getApiErrorDetail, getCurrentProjectId } from './scoreDefinitionModalUt
export const SCORE_DEFINITIONS_PER_PAGE = 30
-export interface ScoreDefinitionFilters {
+export interface ScoreDefinitionFilters extends ResourceTagFilters {
page: number
search: string
kind: ScoreDefinitionKind | ''
@@ -56,6 +62,7 @@ function cleanFilters(values: ScoreDefinitionFilterInput): ScoreDefinitionFilter
const orderBy = values.order_by && ALLOWED_ORDER_BY_VALUES.has(values.order_by) ? values.order_by : 'name'
return {
+ ...cleanTagFilters(values),
page: parseInt(String(values.page)) || 1,
search: String(values.search || ''),
kind:
@@ -76,6 +83,8 @@ function cleanFilters(values: ScoreDefinitionFilterInput): ScoreDefinitionFilter
function getUrlFilters(filters: ScoreDefinitionFilters): Record<string, unknown> {
return {
+ tags: filters.tags,
+ tag_match: filters.tag_match,
page: filters.page === 1 ? undefined : filters.page,
search: filters.search || undefined,
kind: filters.kind || undefined,
@@ -253,6 +262,9 @@ export const aiObservabilityScoreDefinitionsLogic = kea<aiObservabilityScoreDefi
const teamId = String(ApiConfig.getCurrentTeamId())
const page = await aiObservabilityScoreDefinitionsList(teamId, {
search: filters.search || undefined,
+ ...(filters.tags.length
+ ? { tags: JSON.stringify(filters.tags), tag_match: filters.tag_match }
+ : {}),
kind: filters.kind || undefined,
archived: filters.archived === '' ? undefined : filters.archived === 'true' ? true : false,
order_by: filters.order_by,
@@ -334,6 +346,9 @@ export const aiObservabilityScoreDefinitionsLogic = kea<aiObservabilityScoreDefi
listeners(({ actions, asyncActions, values, selectors }) => ({
setFilters: async ({ debounce }, _, __, previousState) => {
const oldFilters = selectors.filters(previousState)
+ if (values.currentTeamId) {
+ saveTagFilters('scorers', values.filters, values.currentTeamId)
+ }
if (!objectsEqual(oldFilters, values.filters)) {
await asyncActions.loadScoreDefinitions(debounce)
@@ -368,11 +383,18 @@ export const aiObservabilityScoreDefinitionsLogic = kea<aiObservabilityScoreDefi
urlToAction(({ actions, values }) => ({
[urls.aiObservabilityScorers()]: (_, searchParams, __, { method }) => {
- const newFilters = cleanFilters(searchParams)
+ if (!values.currentTeamId) {
+ return
+ }
+ const newFilters = cleanFilters({
+ ...searchParams,
+ ...restoreTagFilters('scorers', searchParams, values.currentTeamId),
+ })
if (!objectsEqual(values.filters, newFilters)) {
actions.setFilters(newFilters, false)
} else if (method !== 'REPLACE') {
+ actions.setFilters(newFilters, false)
actions.loadScoreDefinitions(false)
}
},
@@ -380,11 +402,19 @@ export const aiObservabilityScoreDefinitionsLogic = kea<aiObservabilityScoreDefi
// Fires on mount with the current value and again when the project resolves, so the
// initial load waits for bootstrap instead of throwing "Team ID is not known.".
- subscriptions(({ actions }) => ({
+ subscriptions(({ actions, values }) => ({
currentTeamId: (currentTeamId: number | null) => {
actions.resetForTeam()
if (currentTeamId) {
- actions.loadScoreDefinitions(false)
+ const filters = cleanFilters({
+ ...router.values.searchParams,
+ ...restoreTagFilters('scorers', router.values.searchParams, currentTeamId),
+ })
+ if (!objectsEqual(values.filters, filters)) {
+ actions.setFilters(filters, false, false)
+ } else {
+ actions.loadScoreDefinitions(false)
+ }
}
},
})),
diff --git a/products/ai_observability/frontend/scoreDefinitions/scoreDefinitionModalUtils.test.ts b/products/ai_observability/frontend/scoreDefinitions/scoreDefinitionModalUtils.test.ts
index 6616dcd06d57..e5ddf73a7dc1 100644
--- a/products/ai_observability/frontend/scoreDefinitions/scoreDefinitionModalUtils.test.ts
+++ b/products/ai_observability/frontend/scoreDefinitions/scoreDefinitionModalUtils.test.ts
@@ -2,6 +2,7 @@ import type { ScoreDefinitionApi, ScoreDefinitionConfigApi } from '../generated/
import { buildConfigFromDraft, createDraft, validateDraft } from './scoreDefinitionModalUtils'
const definition: ScoreDefinitionApi = {
+ tags: [],
id: 'scorer-example',
name: 'Answer quality',
description: '',
diff --git a/products/ai_observability/frontend/scoreDefinitions/scorerLogic.test.ts b/products/ai_observability/frontend/scoreDefinitions/scorerLogic.test.ts
index f5bc2132a3fc..e1991edc0f49 100644
--- a/products/ai_observability/frontend/scoreDefinitions/scorerLogic.test.ts
+++ b/products/ai_observability/frontend/scoreDefinitions/scorerLogic.test.ts
@@ -29,6 +29,7 @@ const create = jest.mocked(llmAnalyticsScoreDefinitionsCreate)
const createVersion = jest.mocked(llmAnalyticsScoreDefinitionsNewVersionCreate)
const update = jest.mocked(llmAnalyticsScoreDefinitionsPartialUpdate)
const definition: ScoreDefinitionApi = {
+ tags: [],
id: 'scorer-example',
name: 'Answer quality',
description: '',
diff --git a/products/ai_observability/frontend/traceReviews/traceReviewModalLogic.test.ts b/products/ai_observability/frontend/traceReviews/traceReviewModalLogic.test.ts
index 557f5bcc1378..e824fe1e3ca6 100644
--- a/products/ai_observability/frontend/traceReviews/traceReviewModalLogic.test.ts
+++ b/products/ai_observability/frontend/traceReviews/traceReviewModalLogic.test.ts
@@ -34,6 +34,7 @@ const mockTraceReviewsApi = traceReviewsApi as jest.Mocked<typeof traceReviewsAp
describe('traceReviewModalLogic', () => {
const booleanDefinition: ScoreDefinition = {
+ tags: [],
id: 'score_def_1',
name: 'Hallucination',
description: 'Whether the model hallucinated.',
@@ -52,6 +53,7 @@ describe('traceReviewModalLogic', () => {
}
const numericDefinition: ScoreDefinition = {
+ tags: [],
id: 'score_def_2',
name: 'Accuracy',
description: 'How accurate the response was.',
diff --git a/products/ai_observability/frontend/traceReviews/traceReviewModalLogic.ts b/products/ai_observability/frontend/traceReviews/traceReviewModalLogic.ts
index 2595b00fabcf..d45a618afbc8 100644
--- a/products/ai_observability/frontend/traceReviews/traceReviewModalLogic.ts
+++ b/products/ai_observability/frontend/traceReviews/traceReviewModalLogic.ts
@@ -91,6 +91,7 @@ function isCategoricalSelectionValid(
function buildScoreDefinitionFromReviewScore(score: TraceReviewScore, teamId: number): ScoreDefinitionApi {
return {
+ tags: [],
id: score.definition_id,
name: score.definition_name,
description: score.definition_archived
diff --git a/products/ai_observability/mcp/tools.yaml b/products/ai_observability/mcp/tools.yaml
index d4d14808ecac..6e0126fc9a54 100644
--- a/products/ai_observability/mcp/tools.yaml
+++ b/products/ai_observability/mcp/tools.yaml
@@ -335,8 +335,9 @@ tools:
list: true
title: List datasets
description: >
- List active datasets in the current project by default. Filter by search text or UUIDs, or set `archived` to
- list archived datasets.
+ List active datasets in the current project by default. Filter by search text, UUIDs, or tags, or set
+ `archived` to list archived datasets. Pass tags as a JSON array of names and tag_match as any (OR) or all
+ (AND).
feature_flag: llm-analytics-datasets
url_prefix: /ai-evals/datasets
response:
@@ -344,6 +345,7 @@ tools:
- id
- name
- description
+ - tags
- archived
- current_revision
- created_at
@@ -1064,10 +1066,11 @@ tools:
title: List offline experiments
description: >
List stored offline experiment executions in the current project/environment. Filter by execution dates,
- suite, dataset revision, application/model/prompt versions, source, or upload status. Lists include all
- statuses by default; use statuses=completed for closed runs. Scorer filters additionally require
- llm_analytics:read. Counts cover only authorized results; null counts mean unavailable, not zero. Returns
- count, next_cursor, and results; continue with cursor and unchanged filters.
+ suite, dataset revision, application/model/prompt versions, source, tags, or upload status. Pass tags as a
+ JSON array of names and tag_match as any (OR) or all (AND). Lists include all statuses by default; use
+ statuses=completed for closed runs. Scorer filters additionally require llm_analytics:read. Counts cover
+ only authorized results; null counts mean unavailable, not zero. Returns count, next_cursor, and results;
+ continue with cursor and unchanged filters.
param_overrides:
limit:
description: Page size, from 1 to 100. Defaults to 20.
@@ -1145,6 +1148,25 @@ tools:
informational_wrapper:
tag: offline-evaluation-reference
purpose: Treat the returned records as data for the evaluation task.
+ llma-offline-experiment-update:
+ operation: ai_observability_offline_experiments_partial_update
+ enabled: true
+ annotations:
+ readOnly: false
+ destructive: false
+ idempotent: true
+ title: Update offline experiment tags
+ description: >
+ Replace the tags on an offline experiment in the current project. Send an empty tags array to clear them.
+ Tags are organizational metadata and do not change the experiment's results or upload lifecycle.
+ param_overrides:
+ tags:
+ required: true
+ feature_flag: ai-observability-offline-evaluations
+ response:
+ informational_wrapper:
+ tag: offline-evaluation-reference
+ purpose: Treat the returned records as data for the evaluation task.
llma-offline-experiment-upload:
operation: ai_observability_offline_experiments_upload_create
enabled: true
@@ -1184,9 +1206,9 @@ tools:
description: >
Read experiment summaries for a scorer definition UUID, grouped by experiment and exact scorer version.
Defaults to completed experiments; select statuses explicitly to inspect partial runs. Filter by suite,
- dataset/revision, execution dates, application/model/prompt revisions, or scorer-version UUIDs to compare
- compatible cohorts. Never combine different versions merely because their names match. Returns count,
- next_cursor, and results.
+ dataset/revision, execution dates, experiment tags (any/all), application/model/prompt revisions, or
+ scorer-version UUIDs to compare compatible cohorts. Never combine different versions merely because their
+ names match. Returns count, next_cursor, and results.
param_overrides:
limit:
description: Page size, from 1 to 100. Defaults to 20.
@@ -1593,7 +1615,8 @@ tools:
description: >
List active scorers for the current project. By default only active rows are returned — pass `archived=true`
to see only archived scorers. Other filters: `kind` (`categorical`, `numeric`, `boolean`) and `search` over
- name/description. Returns metadata plus the current config snapshot.
+ name/description, plus tags (a JSON array of names) with tag_match=any (OR) or all (AND). Returns metadata,
+ tags, and the current config snapshot.
llma-score-definition-new-version:
operation: llm_analytics_score_definitions_new_version_create
enabled: true
diff --git a/services/mcp/schema/generated-tool-definitions.json b/services/mcp/schema/generated-tool-definitions.json
index 7e80a669068c..46e820ac59b4 100644
--- a/services/mcp/schema/generated-tool-definitions.json
+++ b/services/mcp/schema/generated-tool-definitions.json
@@ -7820,7 +7820,7 @@
"feature_flag": "llm-analytics-datasets"
},
"llma-dataset-list": {
- "description": "List active datasets in the current project by default. Filter by search text or UUIDs, or set `archived` to list archived datasets.",
+ "description": "List active datasets in the current project by default. Filter by search text, UUIDs, or tags, or set `archived` to list archived datasets. Pass tags as a JSON array of names and tag_match as any (OR) or all (AND).",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "List datasets",
@@ -8364,7 +8364,7 @@
"feature_flag": "ai-observability-offline-evaluations"
},
"llma-offline-experiment-list": {
- "description": "List stored offline experiment executions in the current project/environment. Filter by execution dates, suite, dataset revision, application/model/prompt versions, source, or upload status. Lists include all statuses by default; use statuses=completed for closed runs. Scorer filters additionally require llm_analytics:read. Counts cover only authorized results; null counts mean unavailable, not zero. Returns count, next_cursor, and results; continue with cursor and unchanged filters.",
+ "description": "List stored offline experiment executions in the current project/environment. Filter by execution dates, suite, dataset revision, application/model/prompt versions, source, tags, or upload status. Pass tags as a JSON array of names and tag_match as any (OR) or all (AND). Lists include all statuses by default; use statuses=completed for closed runs. Scorer filters additionally require llm_analytics:read. Counts cover only authorized results; null counts mean unavailable, not zero. Returns count, next_cursor, and results; continue with cursor and unchanged filters.",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "List offline experiments",
@@ -8423,6 +8423,21 @@
},
"feature_flag": "ai-observability-offline-evaluations"
},
+ "llma-offline-experiment-update": {
+ "description": "Replace the tags on an offline experiment in the current project. Send an empty tags array to clear them. Tags are organizational metadata and do not change the experiment's results or upload lifecycle.",
+ "category": "AI observability",
+ "feature": "llm_analytics",
+ "summary": "Update offline experiment tags",
+ "title": "Update offline experiment tags",
+ "required_scopes": ["evaluation:write"],
+ "annotations": {
+ "destructiveHint": false,
+ "idempotentHint": true,
+ "openWorldHint": true,
+ "readOnlyHint": false
+ },
+ "feature_flag": "ai-observability-offline-evaluations"
+ },
"llma-offline-experiment-upload": {
"description": "Persist externally computed scores and their shared items in one atomic request. Submit 1–1,000 results, keyed by item_id and exact scorer_version_id, with up to 1,000 item declarations. Reuse caller-generated item IDs and unchanged content for retries; a result is unique per item/scorer version. Existing items can be referenced without repeating their declaration or payload. This tool does not execute evaluators. Item payloads allow 1 MiB and result payloads 256 KiB; use small batches through MCP and the API for bulk uploads. Correct validation errors before retrying; never generate new IDs merely to retry an uncertain response.",
"category": "AI observability",
@@ -8439,7 +8454,7 @@
"feature_flag": "ai-observability-offline-evaluations"
},
"llma-offline-scorer-history": {
- "description": "Read experiment summaries for a scorer definition UUID, grouped by experiment and exact scorer version. Defaults to completed experiments; select statuses explicitly to inspect partial runs. Filter by suite, dataset/revision, execution dates, application/model/prompt revisions, or scorer-version UUIDs to compare compatible cohorts. Never combine different versions merely because their names match. Returns count, next_cursor, and results.",
+ "description": "Read experiment summaries for a scorer definition UUID, grouped by experiment and exact scorer version. Defaults to completed experiments; select statuses explicitly to inspect partial runs. Filter by suite, dataset/revision, execution dates, experiment tags (any/all), application/model/prompt revisions, or scorer-version UUIDs to compare compatible cohorts. Never combine different versions merely because their names match. Returns count, next_cursor, and results.",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "Read offline scorer history",
@@ -8762,7 +8777,7 @@
}
},
"llma-score-definition-list": {
- "description": "List active scorers for the current project. By default only active rows are returned — pass `archived=true` to see only archived scorers. Other filters: `kind` (`categorical`, `numeric`, `boolean`) and `search` over name/description. Returns metadata plus the current config snapshot.",
+ "description": "List active scorers for the current project. By default only active rows are returned — pass `archived=true` to see only archived scorers. Other filters: `kind` (`categorical`, `numeric`, `boolean`) and `search` over name/description, plus tags (a JSON array of names) with tag_match=any (OR) or all (AND). Returns metadata, tags, and the current config snapshot.",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "List scorers",
diff --git a/services/mcp/schema/tool-definitions-all.json b/services/mcp/schema/tool-definitions-all.json
index 1feb22c62631..29ce1cd33112 100644
--- a/services/mcp/schema/tool-definitions-all.json
+++ b/services/mcp/schema/tool-definitions-all.json
@@ -8031,7 +8031,7 @@
"feature_flag": "llm-analytics-datasets"
},
"llma-dataset-list": {
- "description": "List active datasets in the current project by default. Filter by search text or UUIDs, or set `archived` to list archived datasets.",
+ "description": "List active datasets in the current project by default. Filter by search text, UUIDs, or tags, or set `archived` to list archived datasets. Pass tags as a JSON array of names and tag_match as any (OR) or all (AND).",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "List datasets",
@@ -8575,7 +8575,7 @@
"feature_flag": "ai-observability-offline-evaluations"
},
"llma-offline-experiment-list": {
- "description": "List stored offline experiment executions in the current project/environment. Filter by execution dates, suite, dataset revision, application/model/prompt versions, source, or upload status. Lists include all statuses by default; use statuses=completed for closed runs. Scorer filters additionally require llm_analytics:read. Counts cover only authorized results; null counts mean unavailable, not zero. Returns count, next_cursor, and results; continue with cursor and unchanged filters.",
+ "description": "List stored offline experiment executions in the current project/environment. Filter by execution dates, suite, dataset revision, application/model/prompt versions, source, tags, or upload status. Pass tags as a JSON array of names and tag_match as any (OR) or all (AND). Lists include all statuses by default; use statuses=completed for closed runs. Scorer filters additionally require llm_analytics:read. Counts cover only authorized results; null counts mean unavailable, not zero. Returns count, next_cursor, and results; continue with cursor and unchanged filters.",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "List offline experiments",
@@ -8634,6 +8634,21 @@
},
"feature_flag": "ai-observability-offline-evaluations"
},
+ "llma-offline-experiment-update": {
+ "description": "Replace the tags on an offline experiment in the current project. Send an empty tags array to clear them. Tags are organizational metadata and do not change the experiment's results or upload lifecycle.",
+ "category": "AI observability",
+ "feature": "llm_analytics",
+ "summary": "Update offline experiment tags",
+ "title": "Update offline experiment tags",
+ "required_scopes": ["evaluation:write"],
+ "annotations": {
+ "destructiveHint": false,
+ "idempotentHint": true,
+ "openWorldHint": true,
+ "readOnlyHint": false
+ },
+ "feature_flag": "ai-observability-offline-evaluations"
+ },
"llma-offline-experiment-upload": {
"description": "Persist externally computed scores and their shared items in one atomic request. Submit 1–1,000 results, keyed by item_id and exact scorer_version_id, with up to 1,000 item declarations. Reuse caller-generated item IDs and unchanged content for retries; a result is unique per item/scorer version. Existing items can be referenced without repeating their declaration or payload. This tool does not execute evaluators. Item payloads allow 1 MiB and result payloads 256 KiB; use small batches through MCP and the API for bulk uploads. Correct validation errors before retrying; never generate new IDs merely to retry an uncertain response.",
"category": "AI observability",
@@ -8650,7 +8665,7 @@
"feature_flag": "ai-observability-offline-evaluations"
},
"llma-offline-scorer-history": {
- "description": "Read experiment summaries for a scorer definition UUID, grouped by experiment and exact scorer version. Defaults to completed experiments; select statuses explicitly to inspect partial runs. Filter by suite, dataset/revision, execution dates, application/model/prompt revisions, or scorer-version UUIDs to compare compatible cohorts. Never combine different versions merely because their names match. Returns count, next_cursor, and results.",
+ "description": "Read experiment summaries for a scorer definition UUID, grouped by experiment and exact scorer version. Defaults to completed experiments; select statuses explicitly to inspect partial runs. Filter by suite, dataset/revision, execution dates, experiment tags (any/all), application/model/prompt revisions, or scorer-version UUIDs to compare compatible cohorts. Never combine different versions merely because their names match. Returns count, next_cursor, and results.",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "Read offline scorer history",
@@ -9001,7 +9016,7 @@
}
},
"llma-score-definition-list": {
- "description": "List active scorers for the current project. By default only active rows are returned — pass `archived=true` to see only archived scorers. Other filters: `kind` (`categorical`, `numeric`, `boolean`) and `search` over name/description. Returns metadata plus the current config snapshot.",
+ "description": "List active scorers for the current project. By default only active rows are returned — pass `archived=true` to see only archived scorers. Other filters: `kind` (`categorical`, `numeric`, `boolean`) and `search` over name/description, plus tags (a JSON array of names) with tag_match=any (OR) or all (AND). Returns metadata, tags, and the current config snapshot.",
"category": "AI observability",
"feature": "llm_analytics",
"summary": "List scorers",
diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.ts
index 63fdcf8ce94c..2b8c3c536b41 100644
--- a/services/mcp/src/api/generated.ts
+++ b/services/mcp/src/api/generated.ts
@@ -35375,6 +35375,12 @@ export namespace Schemas {
export type DatasetCreateMetadata = { [key: string]: unknown };
export interface DatasetCreate {
+ /**
+ * Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.
+ * @maxItems 100
+ * @items.maxLength 255
+ */
+ tags?: string[];
/**
* Dataset name. Names are unique within a project.
* @maxLength 400
@@ -35574,6 +35580,8 @@ export namespace Schemas {
readonly description: string;
/** JSON object with descriptive dataset metadata. */
readonly metadata: DatasetReadMetadata;
+ /** Project tag names assigned to this resource. */
+ readonly tags: readonly string[];
readonly archived: boolean;
/**
* Latest dataset revision, or null before the first item is added.
@@ -69297,6 +69305,8 @@ export namespace Schemas {
} as const;
export interface OfflineExperimentRead {
+ /** Project tag names assigned to this experiment. */
+ tags: string[];
/** Stable experiment UUID. */
id: string;
/** Experiment name. */
@@ -73951,6 +73961,8 @@ export namespace Schemas {
readonly id: string;
readonly name: string;
readonly description: string;
+ /** Project tag names assigned to this resource. */
+ readonly tags: readonly string[];
readonly kind: ScoreDefinitionKindEnum;
readonly archived: boolean;
/** Current immutable configuration version number. */
@@ -80702,6 +80714,12 @@ export namespace Schemas {
export type PatchedDatasetUpdateMetadata = { [key: string]: unknown };
export interface PatchedDatasetUpdate {
+ /**
+ * Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.
+ * @maxItems 100
+ * @items.maxLength 255
+ */
+ tags?: string[];
/**
* New dataset name. Names are unique within a project.
* @maxLength 400
@@ -83676,6 +83694,15 @@ export namespace Schemas {
event_definition_id?: string | null;
}
+ export interface PatchedOfflineExperimentMetadata {
+ /**
+ * Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.
+ * @maxItems 100
+ * @items.maxLength 255
+ */
+ tags?: string[];
+ }
+
export type PatchedOrganizationProjectsItem = { [key: string]: unknown };
export type PatchedOrganizationMetadata = {[key: string]: string};
@@ -85808,6 +85835,12 @@ export namespace Schemas {
}
export interface PatchedScoreDefinitionMetadata {
+ /**
+ * Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.
+ * @maxItems 100
+ * @items.maxLength 255
+ */
+ tags?: string[];
/**
* Updated scorer name.
* @maxLength 255
@@ -100227,6 +100260,12 @@ export namespace Schemas {
}
export interface ScoreDefinitionCreate {
+ /**
+ * Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.
+ * @maxItems 100
+ * @items.maxLength 255
+ */
+ tags?: string[];
/**
* Human-readable scorer name.
* @maxLength 255
@@ -121117,8 +121156,28 @@ export namespace Schemas {
* @maxLength 255
*/
suite_key?: string;
+ /**
+ * Match any selected tag (OR) or all selected tags (AND).
+ *
+ * * `any` - Any
+ * * `all` - All
+ * @minLength 1
+ */
+ tag_match?: AiObservabilityOfflineExperimentsListTagMatch;
+ /**
+ * JSON array of tag names, for example ["quality","production"]. Up to 20 tags. An empty array disables tag filtering.
+ */
+ tags?: string;
};
+ export type AiObservabilityOfflineExperimentsListTagMatch = typeof AiObservabilityOfflineExperimentsListTagMatch[keyof typeof AiObservabilityOfflineExperimentsListTagMatch];
+
+
+ export const AiObservabilityOfflineExperimentsListTagMatch = {
+ Any: 'any',
+ All: 'all',
+ } as const;
+
export type AiObservabilityOfflineExperimentsItemsListParams = {
/**
* Continuation cursor returned by the previous page.
@@ -121292,8 +121351,28 @@ export namespace Schemas {
* @maxLength 255
*/
suite_key?: string;
+ /**
+ * Match any selected tag (OR) or all selected tags (AND).
+ *
+ * * `any` - Any
+ * * `all` - All
+ * @minLength 1
+ */
+ tag_match?: AiObservabilityOfflineScorersHistoryListTagMatch;
+ /**
+ * JSON array of tag names, for example ["quality","production"]. Up to 20 tags. An empty array disables tag filtering.
+ */
+ tags?: string;
};
+ export type AiObservabilityOfflineScorersHistoryListTagMatch = typeof AiObservabilityOfflineScorersHistoryListTagMatch[keyof typeof AiObservabilityOfflineScorersHistoryListTagMatch];
+
+
+ export const AiObservabilityOfflineScorersHistoryListTagMatch = {
+ Any: 'any',
+ All: 'all',
+ } as const;
+
export type AiObservabilityTracesRetrieveParams = {
/**
* When the trace happened, as carried by links into it. Lets a trace older than the AI events retention load from the shared events table.
@@ -123818,8 +123897,28 @@ export namespace Schemas {
* @minLength 1
*/
search?: string;
+ /**
+ * Match any selected tag (OR) or all selected tags (AND). Additional tags are allowed.
+ *
+ * * `any` - Any
+ * * `all` - All
+ * @minLength 1
+ */
+ tag_match?: DatasetsListTagMatch;
+ /**
+ * JSON array of tag names, for example ["quality","production"]. Up to 20 tags. An empty array disables tag filtering.
+ */
+ tags?: string;
};
+ export type DatasetsListTagMatch = typeof DatasetsListTagMatch[keyof typeof DatasetsListTagMatch];
+
+
+ export const DatasetsListTagMatch = {
+ Any: 'any',
+ All: 'all',
+ } as const;
+
export type DatasetsRevisionsListParams = {
/**
* Number of results to return per page.
@@ -128372,8 +128471,28 @@ export namespace Schemas {
* Search scorers by name or description.
*/
search?: string;
+ /**
+ * Match any selected tag (OR) or all selected tags (AND). Additional tags are allowed.
+ *
+ * * `any` - Any
+ * * `all` - All
+ * @minLength 1
+ */
+ tag_match?: LlmAnalyticsScoreDefinitionsListTagMatch;
+ /**
+ * JSON array of tag names, for example ["quality","production"]. Up to 20 tags. An empty array disables tag filtering.
+ */
+ tags?: string;
};
+ export type LlmAnalyticsScoreDefinitionsListTagMatch = typeof LlmAnalyticsScoreDefinitionsListTagMatch[keyof typeof LlmAnalyticsScoreDefinitionsListTagMatch];
+
+
+ export const LlmAnalyticsScoreDefinitionsListTagMatch = {
+ Any: 'any',
+ All: 'all',
+ } as const;
+
export type LlmAnalyticsScoreDefinitionsVersionsListParams = {
/**
* Continuation cursor from the prior page.
diff --git a/services/mcp/src/generated/ai_observability/api.ts b/services/mcp/src/generated/ai_observability/api.ts
index 4d4d929c17b3..0687c98c1f37 100644
--- a/services/mcp/src/generated/ai_observability/api.ts
+++ b/services/mcp/src/generated/ai_observability/api.ts
@@ -3,7 +3,7 @@
* MCP service uses these Zod schemas for generated tool handlers.
* To regenerate: hogli build:openapi
*
- * PostHog API - MCP 97 enabled ops
+ * PostHog API - MCP 98 enabled ops
* OpenAPI spec version: 1.0.0
*/
import * as zod from 'zod'
@@ -99,6 +99,8 @@ export const aiObservabilityOfflineExperimentsListQueryStatusesMax = 32
export const aiObservabilityOfflineExperimentsListQuerySuiteKeyMax = 255
+export const aiObservabilityOfflineExperimentsListQueryTagMatchDefault = `any`
+
export const AiObservabilityOfflineExperimentsListQueryParams = () => zod.object({
application_version: zod
.string()
@@ -189,6 +191,16 @@ export const AiObservabilityOfflineExperimentsListQueryParams = () => zod.object
.max(aiObservabilityOfflineExperimentsListQuerySuiteKeyMax)
.optional()
.describe('Exact evaluation suite identifier.'),
+ tag_match: zod
+ .enum(['any', 'all'])
+ .default(aiObservabilityOfflineExperimentsListQueryTagMatchDefault)
+ .describe('Match any selected tag (OR) or all selected tags (AND).\n\n\* `any` - Any\n\* `all` - All'),
+ tags: zod
+ .string()
+ .optional()
+ .describe(
+ 'JSON array of tag names, for example [\"quality\",\"production\"]. Up to 20 tags. An empty array disables tag filtering.'
+ ),
})
export const AiObservabilityOfflineExperimentsCreateParams = () => zod.object({
@@ -300,6 +312,27 @@ export const AiObservabilityOfflineExperimentsRetrieveParams = () => zod.object(
),
})
+export const AiObservabilityOfflineExperimentsPartialUpdateParams = () => zod.object({
+ id: zod.string().describe('Experiment UUID.'),
+ project_id: zod
+ .string()
+ .describe(
+ "Project ID of the project you're trying to access. To find the ID of the project, make a call to \/api\/projects\/."
+ ),
+})
+
+export const aiObservabilityOfflineExperimentsPartialUpdateBodyTagsItemMax = 255
+
+export const aiObservabilityOfflineExperimentsPartialUpdateBodyTagsMax = 100
+
+export const AiObservabilityOfflineExperimentsPartialUpdateBody = () => zod.object({
+ tags: zod
+ .array(zod.string().max(aiObservabilityOfflineExperimentsPartialUpdateBodyTagsItemMax))
+ .max(aiObservabilityOfflineExperimentsPartialUpdateBodyTagsMax)
+ .optional()
+ .describe('Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.'),
+})
+
export const AiObservabilityOfflineExperimentsCompleteCreateParams = () => zod.object({
id: zod.string().describe('Experiment UUID.'),
project_id: zod
@@ -695,6 +728,8 @@ export const aiObservabilityOfflineScorersHistoryListQueryStatusesMax = 32
export const aiObservabilityOfflineScorersHistoryListQuerySuiteKeyMax = 255
+export const aiObservabilityOfflineScorersHistoryListQueryTagMatchDefault = `any`
+
export const AiObservabilityOfflineScorersHistoryListQueryParams = () => zod.object({
application_version: zod
.string()
@@ -784,6 +819,16 @@ export const AiObservabilityOfflineScorersHistoryListQueryParams = () => zod.obj
.max(aiObservabilityOfflineScorersHistoryListQuerySuiteKeyMax)
.optional()
.describe('Exact evaluation suite identifier.'),
+ tag_match: zod
+ .enum(['any', 'all'])
+ .default(aiObservabilityOfflineScorersHistoryListQueryTagMatchDefault)
+ .describe('Match any selected tag (OR) or all selected tags (AND).\n\n\* `any` - Any\n\* `all` - All'),
+ tags: zod
+ .string()
+ .optional()
+ .describe(
+ 'JSON array of tag names, for example [\"quality\",\"production\"]. Up to 20 tags. An empty array disables tag filtering.'
+ ),
})
/**
@@ -1013,6 +1058,8 @@ export const datasetsListQueryIdInMax = 100
export const datasetsListQueryOrderByDefault = `-created_at`
+export const datasetsListQueryTagMatchDefault = `any`
+
export const DatasetsListQueryParams = () => zod.object({
archived: zod
.boolean()
@@ -1034,6 +1081,18 @@ export const DatasetsListQueryParams = () => zod.object({
'Field and direction used to order results.\n\n\* `created_at` - created_at\n\* `-created_at` - -created_at\n\* `updated_at` - updated_at\n\* `-updated_at` - -updated_at'
),
search: zod.string().min(1).optional().describe('Search dataset names, descriptions, and metadata.'),
+ tag_match: zod
+ .enum(['any', 'all'])
+ .default(datasetsListQueryTagMatchDefault)
+ .describe(
+ 'Match any selected tag (OR) or all selected tags (AND). Additional tags are allowed.\n\n\* `any` - Any\n\* `all` - All'
+ ),
+ tags: zod
+ .string()
+ .optional()
+ .describe(
+ 'JSON array of tag names, for example [\"quality\",\"production\"]. Up to 20 tags. An empty array disables tag filtering.'
+ ),
})
/**
@@ -1047,12 +1106,21 @@ export const DatasetsCreateParams = () => zod.object({
),
})
+export const datasetsCreateBodyTagsItemMax = 255
+
+export const datasetsCreateBodyTagsMax = 100
+
export const datasetsCreateBodyNameMax = 400
export const datasetsCreateBodyDescriptionDefault = ``
export const datasetsCreateBodyDescriptionMax = 10000
export const DatasetsCreateBody = () => zod.object({
+ tags: zod
+ .array(zod.string().max(datasetsCreateBodyTagsItemMax))
+ .max(datasetsCreateBodyTagsMax)
+ .optional()
+ .describe('Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.'),
name: zod.string().max(datasetsCreateBodyNameMax).describe('Dataset name. Names are unique within a project.'),
description: zod
.string()
@@ -1089,11 +1157,20 @@ export const DatasetsPartialUpdateParams = () => zod.object({
),
})
+export const datasetsPartialUpdateBodyTagsItemMax = 255
+
+export const datasetsPartialUpdateBodyTagsMax = 100
+
export const datasetsPartialUpdateBodyNameMax = 400
export const datasetsPartialUpdateBodyDescriptionMax = 10000
export const DatasetsPartialUpdateBody = () => zod.object({
+ tags: zod
+ .array(zod.string().max(datasetsPartialUpdateBodyTagsItemMax))
+ .max(datasetsPartialUpdateBodyTagsMax)
+ .optional()
+ .describe('Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.'),
name: zod
.string()
.max(datasetsPartialUpdateBodyNameMax)
@@ -2789,6 +2866,8 @@ export const LlmAnalyticsScoreDefinitionsListParams = () => zod.object({
),
})
+export const llmAnalyticsScoreDefinitionsListQueryTagMatchDefault = `any`
+
export const LlmAnalyticsScoreDefinitionsListQueryParams = () => zod.object({
archived: zod.boolean().optional().describe('Filter by archived state.'),
kind: zod.string().optional().describe('Filter by scorer kind.'),
@@ -2796,6 +2875,18 @@ export const LlmAnalyticsScoreDefinitionsListQueryParams = () => zod.object({
offset: zod.number().optional().describe('The initial index from which to return the results.'),
order_by: zod.string().optional().describe('Sort by name, kind, created_at, updated_at, or current_version.'),
search: zod.string().optional().describe('Search scorers by name or description.'),
+ tag_match: zod
+ .enum(['any', 'all'])
+ .default(llmAnalyticsScoreDefinitionsListQueryTagMatchDefault)
+ .describe(
+ 'Match any selected tag (OR) or all selected tags (AND). Additional tags are allowed.\n\n\* `any` - Any\n\* `all` - All'
+ ),
+ tags: zod
+ .string()
+ .optional()
+ .describe(
+ 'JSON array of tag names, for example [\"quality\",\"production\"]. Up to 20 tags. An empty array disables tag filtering.'
+ ),
})
export const LlmAnalyticsScoreDefinitionsCreateParams = () => zod.object({
@@ -2806,6 +2897,10 @@ export const LlmAnalyticsScoreDefinitionsCreateParams = () => zod.object({
),
})
+export const llmAnalyticsScoreDefinitionsCreateBodyTagsItemMax = 255
+
+export const llmAnalyticsScoreDefinitionsCreateBodyTagsMax = 100
+
export const llmAnalyticsScoreDefinitionsCreateBodyNameMax = 255
export const llmAnalyticsScoreDefinitionsCreateBodyArchivedDefault = false
@@ -2816,6 +2911,11 @@ export const llmAnalyticsScoreDefinitionsCreateBodyConfigOneOneOptionsItemLabelM
export const llmAnalyticsScoreDefinitionsCreateBodyConfigOneOnePassingRuleOneCategoriesItemMax = 128
export const LlmAnalyticsScoreDefinitionsCreateBody = () => zod.object({
+ tags: zod
+ .array(zod.string().max(llmAnalyticsScoreDefinitionsCreateBodyTagsItemMax))
+ .max(llmAnalyticsScoreDefinitionsCreateBodyTagsMax)
+ .optional()
+ .describe('Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.'),
name: zod.string().max(llmAnalyticsScoreDefinitionsCreateBodyNameMax).describe('Human-readable scorer name.'),
description: zod.string().nullish().describe('Optional human-readable description.'),
kind: zod
@@ -2949,9 +3049,18 @@ export const LlmAnalyticsScoreDefinitionsPartialUpdateParams = () => zod.object(
),
})
+export const llmAnalyticsScoreDefinitionsPartialUpdateBodyTagsItemMax = 255
+
+export const llmAnalyticsScoreDefinitionsPartialUpdateBodyTagsMax = 100
+
export const llmAnalyticsScoreDefinitionsPartialUpdateBodyNameMax = 255
export const LlmAnalyticsScoreDefinitionsPartialUpdateBody = () => zod.object({
+ tags: zod
+ .array(zod.string().max(llmAnalyticsScoreDefinitionsPartialUpdateBodyTagsItemMax))
+ .max(llmAnalyticsScoreDefinitionsPartialUpdateBodyTagsMax)
+ .optional()
+ .describe('Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.'),
name: zod
.string()
.max(llmAnalyticsScoreDefinitionsPartialUpdateBodyNameMax)
diff --git a/services/mcp/src/tools/generated/ai_observability.ts b/services/mcp/src/tools/generated/ai_observability.ts
index 0ec3e20d73d5..51396f1d96a6 100644
--- a/services/mcp/src/tools/generated/ai_observability.ts
+++ b/services/mcp/src/tools/generated/ai_observability.ts
@@ -241,6 +241,9 @@ const llmaDatasetCreate = (): ToolBase<
handler: async (context: Context, params: z.infer<ReturnType<typeof LlmaDatasetCreateSchema>>) => {
const projectId = await context.stateManager.getProjectId()
const body: Record<string, unknown> = {}
+ if (params.tags !== undefined) {
+ body['tags'] = params.tags
+ }
if (params.name !== undefined) {
body['name'] = params.name
}
@@ -593,6 +596,8 @@ const llmaDatasetList = (): ToolBase<
offset: params.offset,
order_by: params.order_by,
search: params.search,
+ tag_match: params.tag_match,
+ tags: params.tags,
},
})
const filtered = {
@@ -602,6 +607,7 @@ const llmaDatasetList = (): ToolBase<
'id',
'name',
'description',
+ 'tags',
'archived',
'current_revision',
'created_at',
@@ -694,6 +700,9 @@ const llmaDatasetUpdate = (): ToolBase<
handler: async (context: Context, params: z.infer<ReturnType<typeof LlmaDatasetUpdateSchema>>) => {
const projectId = await context.stateManager.getProjectId()
const body: Record<string, unknown> = {}
+ if (params.tags !== undefined) {
+ body['tags'] = params.tags
+ }
if (params.name !== undefined) {
body['name'] = params.name
}
@@ -1852,6 +1861,8 @@ const llmaOfflineExperimentList = (): ToolBase<
search: params.search,
statuses: params.statuses,
suite_key: params.suite_key,
+ tag_match: params.tag_match,
+ tags: params.tags,
},
})
return withInformationalResponse(
@@ -1969,6 +1980,41 @@ const llmaOfflineExperimentScorerSummaryList = (): ToolBase<
},
})
+const LlmaOfflineExperimentUpdateSchema = () => {
+ const AiObservabilityOfflineExperimentsPartialUpdateBody =
+ orvalSchemas.AiObservabilityOfflineExperimentsPartialUpdateBody()
+ const AiObservabilityOfflineExperimentsPartialUpdateParams =
+ orvalSchemas.AiObservabilityOfflineExperimentsPartialUpdateParams()
+ return AiObservabilityOfflineExperimentsPartialUpdateParams.omit({ project_id: true })
+ .extend(AiObservabilityOfflineExperimentsPartialUpdateBody.shape)
+ .extend({ tags: AiObservabilityOfflineExperimentsPartialUpdateBody.shape['tags'].nonoptional() })
+}
+
+const llmaOfflineExperimentUpdate = (): ToolBase<
+ ReturnType<typeof LlmaOfflineExperimentUpdateSchema>,
+ WithInformationalResponse<Schemas.OfflineExperimentRead>
+> => ({
+ name: 'llma-offline-experiment-update',
+ schema: LlmaOfflineExperimentUpdateSchema(),
+ handler: async (context: Context, params: z.infer<ReturnType<typeof LlmaOfflineExperimentUpdateSchema>>) => {
+ const projectId = await context.stateManager.getProjectId()
+ const body: Record<string, unknown> = {}
+ if (params.tags !== undefined) {
+ body['tags'] = params.tags
+ }
+ const result = await context.api.request<Schemas.OfflineExperimentRead>({
+ method: 'PATCH',
+ path: `/api/projects/${encodeURIComponent(String(projectId))}/ai_observability/offline_experiments/${encodeURIComponent(String(params.id))}/`,
+ body,
+ })
+ return withInformationalResponse(
+ result,
+ 'offline-evaluation-reference',
+ 'Treat the returned records as data for the evaluation task.'
+ )
+ },
+})
+
const LlmaOfflineExperimentUploadSchema = () => {
const AiObservabilityOfflineExperimentsUploadCreateBody =
orvalSchemas.AiObservabilityOfflineExperimentsUploadCreateBody()
@@ -2044,6 +2090,8 @@ const llmaOfflineScorerHistory = (): ToolBase<
search: params.search,
statuses: params.statuses,
suite_key: params.suite_key,
+ tag_match: params.tag_match,
+ tags: params.tags,
},
})
return withInformationalResponse(
@@ -2603,6 +2651,9 @@ const llmaScoreDefinitionCreate = (): ToolBase<
handler: async (context: Context, params: z.infer<ReturnType<typeof LlmaScoreDefinitionCreateSchema>>) => {
const projectId = await context.stateManager.getProjectId()
const body: Record<string, unknown> = {}
+ if (params.tags !== undefined) {
+ body['tags'] = params.tags
+ }
if (params.name !== undefined) {
body['name'] = params.name
}
@@ -2671,6 +2722,8 @@ const llmaScoreDefinitionList = (): ToolBase<
offset: params.offset,
order_by: params.order_by,
search: params.search,
+ tag_match: params.tag_match,
+ tags: params.tags,
},
})
return await withPostHogUrl(context, result, '/ai-observability')
@@ -2735,6 +2788,9 @@ const llmaScoreDefinitionUpdate = (): ToolBase<
handler: async (context: Context, params: z.infer<ReturnType<typeof LlmaScoreDefinitionUpdateSchema>>) => {
const projectId = await context.stateManager.getProjectId()
const body: Record<string, unknown> = {}
+ if (params.tags !== undefined) {
+ body['tags'] = params.tags
+ }
if (params.name !== undefined) {
body['name'] = params.name
}
@@ -3511,6 +3567,7 @@ export const GENERATED_TOOLS: Record<string, () => ToolBase<ZodObjectAny>> = {
'llma-offline-experiment-result-cells': llmaOfflineExperimentResultCells,
'llma-offline-experiment-result-payload-get': llmaOfflineExperimentResultPayloadGet,
'llma-offline-experiment-scorer-summary-list': llmaOfflineExperimentScorerSummaryList,
+ 'llma-offline-experiment-update': llmaOfflineExperimentUpdate,
'llma-offline-experiment-upload': llmaOfflineExperimentUpload,
'llma-offline-scorer-history': llmaOfflineScorerHistory,
'llma-personal-spend': llmaPersonalSpend,
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-experiment-list.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-experiment-list.json
index 6d18a941e45f..ba02f9e235b6 100644
--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-experiment-list.json
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-experiment-list.json
@@ -95,6 +95,16 @@
"maxLength": 255,
"minLength": 1,
"type": "string"
+ },
+ "tag_match": {
+ "default": "any",
+ "description": "Match any selected tag (OR) or all selected tags (AND).\n\n* `any` - Any\n* `all` - All",
+ "enum": ["any", "all"],
+ "type": "string"
+ },
+ "tags": {
+ "description": "JSON array of tag names, for example [\"quality\",\"production\"]. Up to 20 tags. An empty array disables tag filtering.",
+ "type": "string"
}
},
"type": "object"
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-experiment-update.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-experiment-update.json
new file mode 100644
index 000000000000..5a25827964bd
--- /dev/null
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-experiment-update.json
@@ -0,0 +1,20 @@
+{
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "properties": {
+ "id": {
+ "description": "Experiment UUID.",
+ "type": "string"
+ },
+ "tags": {
+ "description": "Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.",
+ "items": {
+ "maxLength": 255,
+ "type": "string"
+ },
+ "maxItems": 100,
+ "type": "array"
+ }
+ },
+ "required": ["id", "tags"],
+ "type": "object"
+}
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-scorer-history.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-scorer-history.json
index f5924e83e0c7..6ea054ae5daf 100644
--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-scorer-history.json
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-offline-scorer-history.json
@@ -95,6 +95,16 @@
"maxLength": 255,
"minLength": 1,
"type": "string"
+ },
+ "tag_match": {
+ "default": "any",
+ "description": "Match any selected tag (OR) or all selected tags (AND).\n\n* `any` - Any\n* `all` - All",
+ "enum": ["any", "all"],
+ "type": "string"
+ },
+ "tags": {
+ "description": "JSON array of tag names, for example [\"quality\",\"production\"]. Up to 20 tags. An empty array disables tag filtering.",
+ "type": "string"
}
},
"required": ["id"],
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-score-definition-create.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-score-definition-create.json
index 2c96d32705ca..33715433a169 100644
--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-score-definition-create.json
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-score-definition-create.json
@@ -174,6 +174,15 @@
"description": "Human-readable scorer name.",
"maxLength": 255,
"type": "string"
+ },
+ "tags": {
+ "description": "Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.",
+ "items": {
+ "maxLength": 255,
+ "type": "string"
+ },
+ "maxItems": 100,
+ "type": "array"
}
},
"required": ["name", "kind", "config"],
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-score-definition-list.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-score-definition-list.json
index 38a96747127c..e0e4ee065118 100644
--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-score-definition-list.json
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-score-definition-list.json
@@ -24,6 +24,16 @@
"search": {
"description": "Search scorers by name or description.",
"type": "string"
+ },
+ "tag_match": {
+ "default": "any",
+ "description": "Match any selected tag (OR) or all selected tags (AND). Additional tags are allowed.\n\n* `any` - Any\n* `all` - All",
+ "enum": ["any", "all"],
+ "type": "string"
+ },
+ "tags": {
+ "description": "JSON array of tag names, for example [\"quality\",\"production\"]. Up to 20 tags. An empty array disables tag filtering.",
+ "type": "string"
}
},
"type": "object"
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-score-definition-update.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-score-definition-update.json
index fe6bda3fbaa2..4291be7bc6d6 100644
--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-score-definition-update.json
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/llma-score-definition-update.json
@@ -24,6 +24,15 @@
"description": "Updated scorer name.",
"maxLength": 255,
"type": "string"
+ },
+ "tags": {
+ "description": "Tag names. Replaces existing tags; send [] to remove all tags. Names are trimmed and lowercased.",
+ "items": {
+ "maxLength": 255,
+ "type": "string"
+ },
+ "maxItems": 100,
+ "type": "array"
}
},
"required": ["id"],
diff --git a/services/mcp/tests/unit/llma-resource-tags.test.ts b/services/mcp/tests/unit/llma-resource-tags.test.ts
new file mode 100644
index 000000000000..958982d924e4
--- /dev/null
+++ b/services/mcp/tests/unit/llma-resource-tags.test.ts
@@ -0,0 +1,56 @@
+import { describe, expect, it, vi } from 'vitest'
+
+import { getToolByName } from '@/shared/test-utils'
+import { GENERATED_TOOLS } from '@/tools/generated/ai_observability'
+import type { Context } from '@/tools/types'
+
+function createContext(): { context: Context; request: ReturnType<typeof vi.fn> } {
+ const request = vi.fn().mockResolvedValue({ count: 1, results: [{ id: 'resource-id', tags: ['quality', 'a,b'] }] })
+ return {
+ request,
+ context: {
+ api: { request, getProjectBaseUrl: () => 'https://example.com/project/17' },
+ stateManager: { getProjectId: vi.fn().mockResolvedValue('17') },
+ env: { POSTHOG_PUBLIC_URL: 'https://example.com' },
+ } as unknown as Context,
+ }
+}
+
+describe('AI resource tag tool contracts', () => {
+ it.each([
+ ['llma-dataset-list', '/api/projects/17/datasets/', {}],
+ ['llma-score-definition-list', '/api/projects/17/llm_analytics/score_definitions/', {}],
+ ['llma-offline-experiment-list', '/api/projects/17/ai_observability/offline_experiments/', {}],
+ [
+ 'llma-offline-scorer-history',
+ '/api/projects/17/ai_observability/offline_scorers/scorer-id/history/',
+ { id: 'scorer-id' },
+ ],
+ ])('%s forwards AND/OR tag filters and retains tags in its response', async (name, path, parameters) => {
+ const tool = getToolByName(GENERATED_TOOLS, name)
+ const { context, request } = createContext()
+ for (const tag_match of ['any', 'all']) {
+ const query = { tags: '["quality","a,b"]', tag_match }
+ const parsed = tool.schema.parse({ ...parameters, ...query })
+ const result = await tool.handler(context, parsed)
+ expect(request).toHaveBeenLastCalledWith(
+ expect.objectContaining({ method: 'GET', path, query: expect.objectContaining(query) })
+ )
+ expect(JSON.stringify(result)).toContain('"tags":["quality","a,b"]')
+ }
+ })
+
+ it('requires an explicit tag replacement when updating an experiment', async () => {
+ const tool = getToolByName(GENERATED_TOOLS, 'llma-offline-experiment-update')
+ expect(tool.schema.safeParse({ id: 'resource-id' }).success).toBe(false)
+ const { context, request } = createContext()
+ await tool.handler(context, tool.schema.parse({ id: 'resource-id', tags: [] }))
+ expect(request).toHaveBeenCalledWith(
+ expect.objectContaining({
+ method: 'PATCH',
+ path: '/api/projects/17/ai_observability/offline_experiments/resource-id/',
+ body: { tags: [] },
+ })
+ )
+ })
+})