:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(slack-app): run classifiers on gpt-6-luna without routing loss (#119481)
Vojta Bartošb896f7a3c5c9
diff --git a/products/slack_app/backend/temporal/activities/classifiers.py b/products/slack_app/backend/temporal/activities/classifiers.pyindex da166bc15f58..dd44c89206c0 100644--- a/products/slack_app/backend/temporal/activities/classifiers.py+++ b/products/slack_app/backend/temporal/activities/classifiers.py@@ -2,7 +2,7 @@ import refrom pathlib import Pathimport structlog-from openai.types.shared_params import ResponseFormatJSONSchema+from openai.types.shared_params import ReasoningEffort, ResponseFormatJSONSchemafrom pydantic import BaseModel, ValidationErrorfrom temporalio import activity@@ -83,7 +83,7 @@ def _thread_trace_id(slack_team_id: str | None, thread_ts: str | None) -> str |# because the activity's own deadline expires first. Bounding the retries matters as much# as the timeout: the activity is sync, so a thread Temporal has stopped waiting on keeps# blocking until the client itself returns.-MODEL_OVERRIDE_CLASSIFIER_MODEL = "gpt-5.6-luna"+MODEL_OVERRIDE_CLASSIFIER_MODEL = "gpt-6-luna"MODEL_OVERRIDE_MAX_COMPLETION_TOKENS = 2048# One call per `@PostHog`, on the mention text alone. Measured mean is ~1.7s per call.MODEL_OVERRIDE_TIMEOUT_SECONDS = 10.0@@ -92,12 +92,14 @@ MODEL_OVERRIDE_MAX_RETRIES = 1# Same shape of call as the model override, so it takes the same bounds — including the# cap riding on `max_completion_tokens`. The reasoning-class chat-completions route# rejects `max_tokens` outright, and the classifier would swallow that into its fallback.-PROJECT_ROUTE_CLASSIFIER_MODEL = "gpt-5.6-luna"+PROJECT_ROUTE_CLASSIFIER_MODEL = "gpt-6-luna"PROJECT_ROUTE_MAX_COMPLETION_TOKENS = 2048PROJECT_ROUTE_TIMEOUT_SECONDS = 10.0PROJECT_ROUTE_MAX_RETRIES = 1+# With reasoning on, the model argues itself out of a project the message plainly names.+PROJECT_ROUTE_REASONING_EFFORT: ReasoningEffort = "none"-AGENT_DIRECTED_CLASSIFIER_MODEL = "gpt-5.6-luna"+AGENT_DIRECTED_CLASSIFIER_MODEL = "gpt-6-luna"AGENT_DIRECTED_MAX_COMPLETION_TOKENS = 2048# One call per reply in every thread the agent is working in, and its prompt carries the# thread the override classifier's does not. The eval suite sees 3-9s on that shape, close@@ -703,6 +705,7 @@ def classify_slack_app_project_route(model=PROJECT_ROUTE_CLASSIFIER_MODEL,messages=[{"role": "user", "content": prompt}],max_completion_tokens=PROJECT_ROUTE_MAX_COMPLETION_TOKENS,+ reasoning_effort=PROJECT_ROUTE_REASONING_EFFORT,response_format=_project_route_response_format(projects),)# Tolerant parse on top of the schema, matching the model-override classifier:diff --git a/products/slack_app/backend/temporal/activities/prompts/project_route.md.j2 b/products/slack_app/backend/temporal/activities/prompts/project_route.md.j2index dbaf35473dfe..b002a6af6651 100644--- a/products/slack_app/backend/temporal/activities/prompts/project_route.md.j2+++ b/products/slack_app/backend/temporal/activities/prompts/project_route.md.j2@@ -11,21 +11,24 @@ Only these projects can be selected, and the id is the number:Before you answer with an id, find the words in this message that name that project — its name, or its number. If you cannot point to them, the answer is null. A message that shares a subject with a project, or asks for work a project's team does, has not named it, however sure you are about whose desk it lands on.-Those words have to name the whole project, not only part of its name. Projects often share a name and differ by a qualifier such as "development", "dev", "staging", "test", "sandbox" or "local", and that kind of project holds test traffic, not real users. A message that says only the shared part and leaves the qualifier off has not named the qualified project, even when no project on the list carries the shorter name. With a project called "Fernwood sandbox", "how many orders did Fernwood take today" names no project: it asks about the real product, and the default is where that data is.+Compare names without letter case, emoji, punctuation or a leading "the". People never type the emoji in a project's name, so a name with the emoji left off still names the project: "the checkout app" names a project called "🛒 Checkout app", and "QA's" names one called "qa".++Those words have to name the whole project, not only part of its name. Projects often share a name and differ by a qualifier such as "development", "dev", "staging", "test", "sandbox" or "local", and that kind of project holds test traffic, not real users. A message that says only the shared part and leaves the qualifier off has not named the qualified project, even when no project on the list carries the shorter name. This rule is about a name with a qualifier in it. When a project's whole name is one of those words, such as a project called "Sandbox", that word alone names it. With a project called "Fernwood sandbox", "how many orders did Fernwood take today" names no project: it asks about the real product, and the default is where that data is.A number counts only where it is written as a project: "in project 42", "for 42", or a `/project/42/` link. Messages carry counts, percentages, ids, version numbers and error codes, and any of them may happen to equal a project id. A bare number names no project, however exactly it matches.Name a project when the message says where to look:- "check the error rate on Fernwood staging", "in the website project, why did signups drop", "for production, which exports failed yesterday".- - The project can be named anywhere in the sentence, and the rest of the message is still the request.+ - The project can be named anywhere in the sentence, including at the very end ("…? sandbox please"), in a possessive ("sandbox's error rate") or as a label in front of the request ("sandbox: why did …"), and the rest of the message is still the request.+ - A message can mention the project it is already on and still ask for another one. When it sets the two against each other, as in "not X, Y" or "X usually, Y this time", answer with the project it asks for now.- A project named as the place a problem lives is still where to look: "why is Fernwood production throwing 500s" is answered from Fernwood production. A project is usually the subject of the question AND the place the answer comes from.Answer with null when the project is the thing being worked on rather than the place to look:- "rename the Website project", "add staging to the project dropdown" (a project as the object of a change).- "we should split staging and production" (a proposal about projects).- "update the reply this agent sends", "fix the skill so it stops truncating" (a request to change the agent's own behaviour or output). The project whose team owns that work is not where the answer comes from, because the request asks for no data at all.- - A word that only happens to match a project name. With a project called "Website", "the website is down" names no project.- - A site, domain or app that the data is about. "On" a site says what to measure, not which project to read, unless a project carries exactly that name.+ - A word that only happens to match a project name, used as an ordinary noun with "the", "our", "a" or "my" in front of it. With a project called "Website", "the website is down" names no project. Without such a word, the bare name used as a place names the project: with a project called "Docs", "the docs are out of date" names no project, but "on docs" or "docs:" does.+ - A site, domain or app that the data is about. "On" a site says what to measure, not which project to read. This never applies to a project's own name: a message that asks for data "on <a project's name>" names that project.- "I already checked staging, look at the SDK instead" (an account of a place already ruled out — past tense is the tell).- Two projects at once, as in "compare staging and production". One task answers from one project, and picking one of the two on a guess is worse than staying on the default.diff --git a/products/slack_app/backend/temporal/tests/activities/__snapshots__/test_classify_slack_app_project_route.ambr b/products/slack_app/backend/temporal/tests/activities/__snapshots__/test_classify_slack_app_project_route.ambrindex ac5536150680..891fb6ef772e 100644--- a/products/slack_app/backend/temporal/tests/activities/__snapshots__/test_classify_slack_app_project_route.ambr+++ b/products/slack_app/backend/temporal/tests/activities/__snapshots__/test_classify_slack_app_project_route.ambr@@ -15,21 +15,24 @@Before you answer with an id, find the words in this message that name that project — its name, or its number. If you cannot point to them, the answer is null. A message that shares a subject with a project, or asks for work a project's team does, has not named it, however sure you are about whose desk it lands on.- Those words have to name the whole project, not only part of its name. Projects often share a name and differ by a qualifier such as "development", "dev", "staging", "test", "sandbox" or "local", and that kind of project holds test traffic, not real users. A message that says only the shared part and leaves the qualifier off has not named the qualified project, even when no project on the list carries the shorter name. With a project called "Fernwood sandbox", "how many orders did Fernwood take today" names no project: it asks about the real product, and the default is where that data is.+ Compare names without letter case, emoji, punctuation or a leading "the". People never type the emoji in a project's name, so a name with the emoji left off still names the project: "the checkout app" names a project called "🛒 Checkout app", and "QA's" names one called "qa".++ Those words have to name the whole project, not only part of its name. Projects often share a name and differ by a qualifier such as "development", "dev", "staging", "test", "sandbox" or "local", and that kind of project holds test traffic, not real users. A message that says only the shared part and leaves the qualifier off has not named the qualified project, even when no project on the list carries the shorter name. This rule is about a name with a qualifier in it. When a project's whole name is one of those words, such as a project called "Sandbox", that word alone names it. With a project called "Fernwood sandbox", "how many orders did Fernwood take today" names no project: it asks about the real product, and the default is where that data is.A number counts only where it is written as a project: "in project 42", "for 42", or a `/project/42/` link. Messages carry counts, percentages, ids, version numbers and error codes, and any of them may happen to equal a project id. A bare number names no project, however exactly it matches.Name a project when the message says where to look:- "check the error rate on Fernwood staging", "in the website project, why did signups drop", "for production, which exports failed yesterday".- - The project can be named anywhere in the sentence, and the rest of the message is still the request.+ - The project can be named anywhere in the sentence, including at the very end ("…? sandbox please"), in a possessive ("sandbox's error rate") or as a label in front of the request ("sandbox: why did …"), and the rest of the message is still the request.+ - A message can mention the project it is already on and still ask for another one. When it sets the two against each other, as in "not X, Y" or "X usually, Y this time", answer with the project it asks for now.- A project named as the place a problem lives is still where to look: "why is Fernwood production throwing 500s" is answered from Fernwood production. A project is usually the subject of the question AND the place the answer comes from.Answer with null when the project is the thing being worked on rather than the place to look:- "rename the Website project", "add staging to the project dropdown" (a project as the object of a change).- "we should split staging and production" (a proposal about projects).- "update the reply this agent sends", "fix the skill so it stops truncating" (a request to change the agent's own behaviour or output). The project whose team owns that work is not where the answer comes from, because the request asks for no data at all.- - A word that only happens to match a project name. With a project called "Website", "the website is down" names no project.- - A site, domain or app that the data is about. "On" a site says what to measure, not which project to read, unless a project carries exactly that name.+ - A word that only happens to match a project name, used as an ordinary noun with "the", "our", "a" or "my" in front of it. With a project called "Website", "the website is down" names no project. Without such a word, the bare name used as a place names the project: with a project called "Docs", "the docs are out of date" names no project, but "on docs" or "docs:" does.+ - A site, domain or app that the data is about. "On" a site says what to measure, not which project to read. This never applies to a project's own name: a message that asks for data "on <a project's name>" names that project.- "I already checked staging, look at the SDK instead" (an account of a place already ruled out — past tense is the tell).- Two projects at once, as in "compare staging and production". One task answers from one project, and picking one of the two on a guess is worse than staying on the default.diff --git a/products/slack_app/backend/temporal/tests/activities/test_classify_slack_app_project_route.py b/products/slack_app/backend/temporal/tests/activities/test_classify_slack_app_project_route.pyindex d0eba45f6261..f0a16d46bf7f 100644--- a/products/slack_app/backend/temporal/tests/activities/test_classify_slack_app_project_route.py+++ b/products/slack_app/backend/temporal/tests/activities/test_classify_slack_app_project_route.py@@ -55,7 +55,7 @@ class TestClassifySlackAppProjectRoute:classify_slack_app_project_route("check staging", PROJECTS)def test_token_cap_uses_the_reasoning_model_parameter(self):- # `gpt-5.6-luna` rejects `max_tokens` on chat completions, and the activity turns+ # The reasoning-class model rejects `max_tokens` on chat completions, and the activity turns# that rejection into its fallback — a mention would silently never route.fake_client = self._fake_client('{"project_id": null}')with patch(CLASSIFIER, return_value=fake_client):diff --git a/products/slack_app/evals/eval_project_classifier.py b/products/slack_app/evals/eval_project_classifier.pyindex 49a6eef4f08a..cfc917583ea0 100644--- a/products/slack_app/evals/eval_project_classifier.py+++ b/products/slack_app/evals/eval_project_classifier.py@@ -87,6 +87,7 @@ PROJECTS = [STAGING, PRODUCTION, WEBSITE, AGENT_PLATFORM, MOBILE, MAIN]# this suite is here to measure. The pull towards an owner only appears when several# owners are on offer and the message reads like their subject matter.AUTOPILOT = _project(team_id=46, integration_id=460, name="Autopilot")+TEST = _project(team_id=49, integration_id=490, name="🧪 Test")CROWD = [MAIN,STAGING,@@ -97,7 +98,7 @@ CROWD = [AUTOPILOT,_project(team_id=47, integration_id=470, name="AI labeling suite"),_project(team_id=48, integration_id=480, name="Security Review Agent"),- _project(team_id=49, integration_id=490, name="🧪 Test"),+ TEST,_project(team_id=50, integration_id=500, name="DevEx"),_project(team_id=51, integration_id=510, name="Demo"),_project(team_id=52, integration_id=520, name="Statuspage"),@@ -122,6 +123,11 @@ def _routes_to(project: Integration | None = None) -> dict:return {PROJECT_ROUTE_KEY: {"integration_id": project.id if project else None}}+def _stays_on(default: Integration) -> dict:+ """The message names the project the run is already on, so its id and null both pass."""+ return {PROJECT_ROUTE_KEY: {"integration_id": default.id, "is_default": True}}++def _in_the_crowd(**kwargs) -> BaseEvalCase:return BaseEvalCase(metadata={"workspace": "crowded"}, **kwargs)@@ -201,7 +207,7 @@ ROUTING_CASES = [BaseEvalCase(name="default_named_explicitly",prompt="@PostHog in Main, how many weekly active users did we finish the month on?",- expected=_routes_to(MAIN),+ expected=_stays_on(MAIN),),# The run is on the default already, so a project named alongside it still wins.BaseEvalCase(@@ -209,6 +215,36 @@ ROUTING_CASES = [prompt="@PostHog I know we usually look at Main, but check the paywall conversion on website",expected=_routes_to(WEBSITE),),+ BaseEvalCase(+ name="default_as_a_label",+ prompt="@PostHog Main: what was yesterday's daily active user count?",+ expected=_stays_on(MAIN),+ ),+ BaseEvalCase(+ name="label_in_front_of_the_request",+ prompt="@PostHog production: how many 5xx responses have we served since noon?",+ expected=_routes_to(PRODUCTION),+ ),+ BaseEvalCase(+ name="name_in_capitals",+ prompt="@PostHog STAGING, are session replays recording again after the SDK bump?",+ expected=_routes_to(STAGING),+ ),+ BaseEvalCase(+ name="two_word_name_in_a_lowercase_possessive",+ prompt="@PostHog what's agent platform's p95 latency today?",+ expected=_routes_to(AGENT_PLATFORM),+ ),+ BaseEvalCase(+ name="project_after_the_question",+ prompt="@PostHog what's the bounce rate on the pricing page? for website",+ expected=_routes_to(WEBSITE),+ ),+ BaseEvalCase(+ name="decorated_name_after_on",+ prompt="@PostHog on mobile app, which screens load slowest this week?",+ expected=_routes_to(MOBILE),+ ),]NO_ROUTE_CASES = [@@ -331,6 +367,22 @@ NO_ROUTE_CASES = [prompt="@PostHog keep this where it is and tell me which dashboards nobody opened this quarter",expected=_routes_to(),),+ # A project name used as an ordinary word, in the shapes the routing cases above accept.+ BaseEvalCase(+ name="environment_word_as_a_verb",+ prompt="@PostHog we're staging the 5.0 rollout this week, how did signups trend last week?",+ expected=_routes_to(),+ ),+ BaseEvalCase(+ name="label_that_is_not_a_project",+ prompt="@PostHog quick one: how many people signed up yesterday?",+ expected=_routes_to(),+ ),+ BaseEvalCase(+ name="decorated_name_as_the_object_of_a_change",+ prompt="@PostHog rename the mobile app's settings tab in our onboarding checklist copy",+ expected=_routes_to(),+ ),# The shape of the mention that misrouted, in a workspace shaped like the one it# misrouted in. Dense, numeric, unmistakably about an AI agent, and naming no project._in_the_crowd(@@ -353,6 +405,11 @@ NO_ROUTE_CASES = [),expected=_routes_to(),),+ _in_the_crowd(+ name="crowded_qualifier_word_as_a_verb",+ prompt="@PostHog can you test whether the paywall_viewed event still fires after the release?",+ expected=_routes_to(),+ ),_in_the_crowd(name="crowded_agent_evaluation_question",prompt="@PostHog which of your evaluations regressed this week, and can you make the scorer stricter?",@@ -372,6 +429,12 @@ CROWDED_ROUTING_CASES = [prompt="@PostHog how many agent runs failed overnight on Autopilot?",expected=_routes_to(AUTOPILOT),),+ # The whole name is a qualifier word, behind an emoji nobody types.+ _in_the_crowd(+ name="crowded_qualifier_word_is_the_whole_name",+ prompt="@PostHog did the onboarding_completed event fire in the test project today?",+ expected=_routes_to(TEST),+ ),]# The qualifier is what names the test project. Said in full or cut short, it still routes.diff --git a/products/slack_app/evals/scorers.py b/products/slack_app/evals/scorers.pyindex 8e238c3a0236..6408bae68135 100644--- a/products/slack_app/evals/scorers.py+++ b/products/slack_app/evals/scorers.py@@ -253,9 +253,12 @@ class ProjectRouteMatch(Scorer):got = _routes_to(output)wanted = want.get("integration_id")+ # Null keeps the run on the default, so naming the default and answering null+ # answer from the same data.+ matched = got == wanted or (got is None and want.get("is_default", False))return Score(name=self._name(),- score=1.0 if got == wanted else 0.0,+ score=1.0 if matched else 0.0,metadata={"expected_integration_id": wanted, "actual_integration_id": got},)