Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(slack_app): stop routing device-type words to a mobile project (#127391)

Vojta Bartoš2468be9549f0
diff --git a/products/slack_app/backend/temporal/activities/prompts/project_route.md.j2 b/products/slack_app/backend/temporal/activities/prompts/project_route.md.j2
index b002a6af6651..a4c54870b016 100644
--- a/products/slack_app/backend/temporal/activities/prompts/project_route.md.j2
+++ b/products/slack_app/backend/temporal/activities/prompts/project_route.md.j2
@@ -13,14 +13,14 @@ Before you answer with an id, find the words in this message that name that proj
Compare names without letter case, emoji, punctuation or a leading "the". People never type the emoji in a project's name, so a name with the emoji left off still names the project: "the checkout app" names a project called "🛒 Checkout app", and "QA's" names one called "qa".
-Those words have to name the whole project, not only part of its name. Projects often share a name and differ by a qualifier such as "development", "dev", "staging", "test", "sandbox" or "local", and that kind of project holds test traffic, not real users. A message that says only the shared part and leaves the qualifier off has not named the qualified project, even when no project on the list carries the shorter name. This rule is about a name with a qualifier in it. When a project's whole name is one of those words, such as a project called "Sandbox", that word alone names it. With a project called "Fernwood sandbox", "how many orders did Fernwood take today" names no project: it asks about the real product, and the default is where that data is.
+Those words have to name the whole project, not only part of its name. Projects often share a name and differ by a qualifier such as "development", "dev", "staging", "test", "sandbox" or "local", and that kind of project holds test traffic, not real users. A message that says only the shared part and leaves the qualifier off has not named the qualified project, even when no project on the list carries the shorter name. This rule is about a name with a qualifier in it. When a project's whole name is one of those words, such as a project called "Sandbox", that word alone names it. With a project called "Fernwood sandbox", "how many orders did Fernwood take today" names no project: it asks about the real product, and the default is where that data is. The same holds for every other word in a name: one word of a project's name, said alone, does not name it.
A number counts only where it is written as a project: "in project 42", "for 42", or a `/project/42/` link. Messages carry counts, percentages, ids, version numbers and error codes, and any of them may happen to equal a project id. A bare number names no project, however exactly it matches.
Name a project when the message says where to look:
- "check the error rate on Fernwood staging", "in the website project, why did signups drop", "for production, which exports failed yesterday".
- The project can be named anywhere in the sentence, including at the very end ("…? sandbox please"), in a possessive ("sandbox's error rate") or as a label in front of the request ("sandbox: why did …"), and the rest of the message is still the request.
- - A message can mention the project it is already on and still ask for another one. When it sets the two against each other, as in "not X, Y" or "X usually, Y this time", answer with the project it asks for now.
+ - A message can mention the project it is already on and still ask for another one. When it sets the two against each other, as in "not X, Y", "X usually, Y this time" or "I know we usually look at X, but check … on Y", answer with the project it asks for now. Y names the project even as a bare lowercase word: with a project called "Website", "… but check the signup funnel on website" asks for Website.
- A project named as the place a problem lives is still where to look: "why is Fernwood production throwing 500s" is answered from Fernwood production. A project is usually the subject of the question AND the place the answer comes from.
Answer with null when the project is the thing being worked on rather than the place to look:
@@ -29,6 +29,7 @@ Answer with null when the project is the thing being worked on rather than the p
- "update the reply this agent sends", "fix the skill so it stops truncating" (a request to change the agent's own behaviour or output). The project whose team owns that work is not where the answer comes from, because the request asks for no data at all.
- A word that only happens to match a project name, used as an ordinary noun with "the", "our", "a" or "my" in front of it. With a project called "Website", "the website is down" names no project. Without such a word, the bare name used as a place names the project: with a project called "Docs", "the docs are out of date" names no project, but "on docs" or "docs:" does.
- A site, domain or app that the data is about. "On" a site says what to measure, not which project to read. This never applies to a project's own name: a message that asks for data "on <a project's name>" names that project.
+ - A word from a project's name that says which slice of the data to look at: a device type, platform, operating system, browser or country. "Phone users", "on a tablet", "Android sessions" and "on desktop" filter the data, and the default holds that data. With a project called "🤖 Fernwood Android app", "how do people use Fernwood on Android" and "show this person's sessions on an Android device" name no project. "In the Fernwood Android app" or "Android app:" does. This is about a device word standing in for the project's name. When the message says the project's whole name, it names the project, even where that name is a device word: with a project called "📱 Lakeside mobile app", "the Lakeside mobile app" names it, the same as "the checkout app" above.
- "I already checked staging, look at the SDK instead" (an account of a place already ruled out — past tense is the tell).
- Two projects at once, as in "compare staging and production". One task answers from one project, and picking one of the two on a guess is worse than staying on the default.
diff --git a/products/slack_app/backend/temporal/tests/activities/__snapshots__/test_classify_slack_app_project_route.ambr b/products/slack_app/backend/temporal/tests/activities/__snapshots__/test_classify_slack_app_project_route.ambr
index 891fb6ef772e..179d1ed9e273 100644
--- a/products/slack_app/backend/temporal/tests/activities/__snapshots__/test_classify_slack_app_project_route.ambr
+++ b/products/slack_app/backend/temporal/tests/activities/__snapshots__/test_classify_slack_app_project_route.ambr
@@ -17,14 +17,14 @@
Compare names without letter case, emoji, punctuation or a leading "the". People never type the emoji in a project's name, so a name with the emoji left off still names the project: "the checkout app" names a project called "🛒 Checkout app", and "QA's" names one called "qa".
- Those words have to name the whole project, not only part of its name. Projects often share a name and differ by a qualifier such as "development", "dev", "staging", "test", "sandbox" or "local", and that kind of project holds test traffic, not real users. A message that says only the shared part and leaves the qualifier off has not named the qualified project, even when no project on the list carries the shorter name. This rule is about a name with a qualifier in it. When a project's whole name is one of those words, such as a project called "Sandbox", that word alone names it. With a project called "Fernwood sandbox", "how many orders did Fernwood take today" names no project: it asks about the real product, and the default is where that data is.
+ Those words have to name the whole project, not only part of its name. Projects often share a name and differ by a qualifier such as "development", "dev", "staging", "test", "sandbox" or "local", and that kind of project holds test traffic, not real users. A message that says only the shared part and leaves the qualifier off has not named the qualified project, even when no project on the list carries the shorter name. This rule is about a name with a qualifier in it. When a project's whole name is one of those words, such as a project called "Sandbox", that word alone names it. With a project called "Fernwood sandbox", "how many orders did Fernwood take today" names no project: it asks about the real product, and the default is where that data is. The same holds for every other word in a name: one word of a project's name, said alone, does not name it.
A number counts only where it is written as a project: "in project 42", "for 42", or a `/project/42/` link. Messages carry counts, percentages, ids, version numbers and error codes, and any of them may happen to equal a project id. A bare number names no project, however exactly it matches.
Name a project when the message says where to look:
- "check the error rate on Fernwood staging", "in the website project, why did signups drop", "for production, which exports failed yesterday".
- The project can be named anywhere in the sentence, including at the very end ("…? sandbox please"), in a possessive ("sandbox's error rate") or as a label in front of the request ("sandbox: why did …"), and the rest of the message is still the request.
- - A message can mention the project it is already on and still ask for another one. When it sets the two against each other, as in "not X, Y" or "X usually, Y this time", answer with the project it asks for now.
+ - A message can mention the project it is already on and still ask for another one. When it sets the two against each other, as in "not X, Y", "X usually, Y this time" or "I know we usually look at X, but check … on Y", answer with the project it asks for now. Y names the project even as a bare lowercase word: with a project called "Website", "… but check the signup funnel on website" asks for Website.
- A project named as the place a problem lives is still where to look: "why is Fernwood production throwing 500s" is answered from Fernwood production. A project is usually the subject of the question AND the place the answer comes from.
Answer with null when the project is the thing being worked on rather than the place to look:
@@ -33,6 +33,7 @@
- "update the reply this agent sends", "fix the skill so it stops truncating" (a request to change the agent's own behaviour or output). The project whose team owns that work is not where the answer comes from, because the request asks for no data at all.
- A word that only happens to match a project name, used as an ordinary noun with "the", "our", "a" or "my" in front of it. With a project called "Website", "the website is down" names no project. Without such a word, the bare name used as a place names the project: with a project called "Docs", "the docs are out of date" names no project, but "on docs" or "docs:" does.
- A site, domain or app that the data is about. "On" a site says what to measure, not which project to read. This never applies to a project's own name: a message that asks for data "on <a project's name>" names that project.
+ - A word from a project's name that says which slice of the data to look at: a device type, platform, operating system, browser or country. "Phone users", "on a tablet", "Android sessions" and "on desktop" filter the data, and the default holds that data. With a project called "🤖 Fernwood Android app", "how do people use Fernwood on Android" and "show this person's sessions on an Android device" name no project. "In the Fernwood Android app" or "Android app:" does. This is about a device word standing in for the project's name. When the message says the project's whole name, it names the project, even where that name is a device word: with a project called "📱 Lakeside mobile app", "the Lakeside mobile app" names it, the same as "the checkout app" above.
- "I already checked staging, look at the SDK instead" (an account of a place already ruled out — past tense is the tell).
- Two projects at once, as in "compare staging and production". One task answers from one project, and picking one of the two on a guess is worse than staying on the default.
diff --git a/products/slack_app/evals/eval_project_classifier.py b/products/slack_app/evals/eval_project_classifier.py
index 939af53c7aca..1240fd0b1eb6 100644
--- a/products/slack_app/evals/eval_project_classifier.py
+++ b/products/slack_app/evals/eval_project_classifier.py
@@ -120,10 +120,19 @@ NORTHWIND = _project(team_id=53, integration_id=530, name="Northwind")
SITE_DEV = _project(team_id=54, integration_id=540, name="northwind.io development")
SITES = [NORTHWIND, SITE_DEV, STAGING, AGENT_PLATFORM, MOBILE]
+# A fourth workspace, where the default and a mobile project both carry the company's
+# name, so one device word is all that tells them apart. A question about how people use
+# the product on a phone matches part of the mobile project's name, and that partial match
+# moved the run off the default, which holds the data the question asks about.
+NORTHWIND_APP = _project(team_id=55, integration_id=550, name="🎉 Northwind app + website")
+NORTHWIND_MOBILE = _project(team_id=56, integration_id=560, name="📱 Northwind mobile app")
+APPS = [NORTHWIND_APP, NORTHWIND_MOBILE, STAGING, AGENT_PLATFORM]
+
WORKSPACES: dict[str, tuple[list[WorkspaceIntegration], WorkspaceIntegration]] = {
"simple": (PROJECTS, MAIN),
"crowded": (CROWD, MAIN),
"sites": (SITES, NORTHWIND),
+ "apps": (APPS, NORTHWIND_APP),
}
@@ -144,6 +153,10 @@ def _beside_the_site(**kwargs) -> BaseEvalCase:
return BaseEvalCase(metadata={"workspace": "sites"}, **kwargs)
+def _beside_the_mobile_app(**kwargs) -> BaseEvalCase:
+ return BaseEvalCase(metadata={"workspace": "apps"}, **kwargs)
+
+
ROUTING_CASES = [
BaseEvalCase(
name="explicit_on_project",
@@ -413,6 +426,12 @@ NO_ROUTE_CASES = [
),
expected=_routes_to(),
),
+ # A device word that is also part of a project's name, used to slice the data.
+ BaseEvalCase(
+ name="device_word_as_a_filter",
+ prompt="@PostHog how many mobile users finished onboarding yesterday?",
+ expected=_routes_to(),
+ ),
_in_the_crowd(
name="crowded_qualifier_word_as_a_verb",
prompt="@PostHog can you test whether the paywall_viewed event still fires after the release?",
@@ -473,13 +492,70 @@ SITE_NO_ROUTE_CASES = [
),
]
+# A device type, platform or browser says which slice of the default's data to read. The
+# mobile project holds only what its own app sends.
+DEVICE_NO_ROUTE_CASES = [
+ _beside_the_mobile_app(
+ name="one_person_on_their_phone",
+ prompt="@PostHog look up [email protected] and summarize what they do when they use us from their phone",
+ expected=_routes_to(),
+ ),
+ _beside_the_mobile_app(
+ name="mobile_against_desktop",
+ prompt="@PostHog is checkout conversion lower on mobile than on desktop this month?",
+ expected=_routes_to(),
+ ),
+ _beside_the_mobile_app(
+ name="mobile_browsers_as_a_share",
+ prompt="@PostHog what fraction of last week's signups came in through mobile browsers?",
+ expected=_routes_to(),
+ ),
+ _beside_the_mobile_app(
+ name="mobile_as_an_adjective_on_visitors",
+ prompt="@PostHog are mobile visitors leaving the pricing page faster than they did in March?",
+ expected=_routes_to(),
+ ),
+ _beside_the_mobile_app(
+ name="device_model_as_a_cohort",
+ prompt="@PostHog show 4-week retention for people whose first visit was on an iPhone",
+ expected=_routes_to(),
+ ),
+ _beside_the_mobile_app(
+ name="company_name_beside_a_device_word",
+ prompt="@PostHog how much Northwind usage comes from tablets these days?",
+ expected=_routes_to(),
+ ),
+]
+
+# The mobile project is still one a person can ask for by name.
+DEVICE_ROUTING_CASES = [
+ _beside_the_mobile_app(
+ name="mobile_project_named_in_full",
+ prompt="@PostHog in the Northwind mobile app, which screens crash the most this week?",
+ expected=_routes_to(NORTHWIND_MOBILE),
+ ),
+ _beside_the_mobile_app(
+ name="mobile_project_as_a_label",
+ prompt="@PostHog mobile app: how many crashes since Friday's build?",
+ expected=_routes_to(NORTHWIND_MOBILE),
+ ),
+]
+
# A mention normally arrives with the resolved project on offer as the default, but not
# always: `routable_projects` drops an install the bot cannot post into, and the project
# the run is on can be that one. The model then sees the list in the auth filter's own
# order and no sentence naming where it already is.
_CASES_BY_NAME = {
case.name: case
- for case in [*ROUTING_CASES, *NO_ROUTE_CASES, *CROWDED_ROUTING_CASES, *SITE_ROUTING_CASES, *SITE_NO_ROUTE_CASES]
+ for case in [
+ *ROUTING_CASES,
+ *NO_ROUTE_CASES,
+ *CROWDED_ROUTING_CASES,
+ *SITE_ROUTING_CASES,
+ *SITE_NO_ROUTE_CASES,
+ *DEVICE_ROUTING_CASES,
+ *DEVICE_NO_ROUTE_CASES,
+ ]
}
@@ -508,6 +584,7 @@ NO_DEFAULT_CASES = [
"crowded_change_the_agents_reply",
"crowded_explicit_on_project",
"site_named_without_its_qualifier",
+ "one_person_on_their_phone",
)
]
@@ -548,8 +625,10 @@ async def eval_project_classifier(ctx: EvalContext) -> None:
*ROUTING_CASES,
*CROWDED_ROUTING_CASES,
*SITE_ROUTING_CASES,
+ *DEVICE_ROUTING_CASES,
*NO_ROUTE_CASES,
*SITE_NO_ROUTE_CASES,
+ *DEVICE_NO_ROUTE_CASES,
*NO_DEFAULT_CASES,
],
scorers=[ProjectRouteMatch(), NoUnaskedProjectSwitch()],