Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(slack-app): stop untagged follow-ups waking on confirmations (#128972)

Vojta Bartoš7fab4204b645
diff --git a/products/slack_app/backend/temporal/activities/prompts/message_is_agent_directed.md.j2 b/products/slack_app/backend/temporal/activities/prompts/message_is_agent_directed.md.j2
index ed5407e9865f..6959a735aaff 100644
--- a/products/slack_app/backend/temporal/activities/prompts/message_is_agent_directed.md.j2
+++ b/products/slack_app/backend/temporal/activities/prompts/message_is_agent_directed.md.j2
@@ -3,7 +3,7 @@ The PostHog agent is working on a task in this Slack thread. People in the threa
Answer true only when the message is addressed to the agent:
- An order or request to do something, with the agent as the only plausible audience ('also handle the empty case', 'skip the migration', 'open a PR for that too', 'try again with the other helper').
- A question put to the agent about its own work ('why did you skip the redesign commit?', 'does your PR cover the mobile breakpoint?').
- - An answer to something the PostHog agent itself just asked in this thread.
+ - An answer to something the PostHog agent itself just asked in this thread, when the answer gives it a decision or new information to act on.
- A correction of what the agent said or did ('no, the bug is in the handler, not the wrapper').
Answer false for everything else. This is the common case, and the mistake to avoid — people talk about the work far more often than they talk to the agent:
@@ -14,6 +14,8 @@ Answer false for everything else. This is the common case, and the mistake to av
- An instruction or request aimed at a named person, even with no @mention ('sam go ahead and add yourself as the owner', 'jake, can you take this one'). A name in the thread is the audience; an order is not addressed to the agent just because it is phrased as one.
- A request only a person could carry out — joining a call, meeting a customer, updating an account in a system the agent has no access to — even when it says 'you', and even when it directly follows something the agent said.
- Someone taking the work on themselves ('let me fix that', 'I'm on it', 'I'll take a look'). That stands the agent down; it is not a handoff to it.
+ - A confirmation that what the agent assumed or already did is right, when the agent asked to hear back only if it was wrong ('yep, that's the right team', 'yeah we do own that'). Nothing changes, so there is nothing to answer.
+ - An answer to a question a person asked earlier in the thread, even when it touches the agent's work.
- Acknowledgements, praise, and reactions ('thanks', 'lgtm', 'nice', '+1').
- Context dropped into the thread with no instruction attached — a link, a stack trace, a screenshot — unless the message asks the agent to act on it.
- Anything off-topic.
diff --git a/products/slack_app/backend/temporal/tests/activities/__snapshots__/test_classify_message_is_agent_directed.ambr b/products/slack_app/backend/temporal/tests/activities/__snapshots__/test_classify_message_is_agent_directed.ambr
index b24255b4e35b..2324cfbf0ce6 100644
--- a/products/slack_app/backend/temporal/tests/activities/__snapshots__/test_classify_message_is_agent_directed.ambr
+++ b/products/slack_app/backend/temporal/tests/activities/__snapshots__/test_classify_message_is_agent_directed.ambr
@@ -6,7 +6,7 @@
Answer true only when the message is addressed to the agent:
- An order or request to do something, with the agent as the only plausible audience ('also handle the empty case', 'skip the migration', 'open a PR for that too', 'try again with the other helper').
- A question put to the agent about its own work ('why did you skip the redesign commit?', 'does your PR cover the mobile breakpoint?').
- - An answer to something the PostHog agent itself just asked in this thread.
+ - An answer to something the PostHog agent itself just asked in this thread, when the answer gives it a decision or new information to act on.
- A correction of what the agent said or did ('no, the bug is in the handler, not the wrapper').
Answer false for everything else. This is the common case, and the mistake to avoid — people talk about the work far more often than they talk to the agent:
@@ -17,6 +17,8 @@
- An instruction or request aimed at a named person, even with no @mention ('sam go ahead and add yourself as the owner', 'jake, can you take this one'). A name in the thread is the audience; an order is not addressed to the agent just because it is phrased as one.
- A request only a person could carry out — joining a call, meeting a customer, updating an account in a system the agent has no access to — even when it says 'you', and even when it directly follows something the agent said.
- Someone taking the work on themselves ('let me fix that', 'I'm on it', 'I'll take a look'). That stands the agent down; it is not a handoff to it.
+ - A confirmation that what the agent assumed or already did is right, when the agent asked to hear back only if it was wrong ('yep, that's the right team', 'yeah we do own that'). Nothing changes, so there is nothing to answer.
+ - An answer to a question a person asked earlier in the thread, even when it touches the agent's work.
- Acknowledgements, praise, and reactions ('thanks', 'lgtm', 'nice', '+1').
- Context dropped into the thread with no instruction attached — a link, a stack trace, a screenshot — unless the message asks the agent to act on it.
- Anything off-topic.
diff --git a/products/slack_app/evals/eval_followup_classifier.py b/products/slack_app/evals/eval_followup_classifier.py
index 889d61197904..6ca58f2f572a 100644
--- a/products/slack_app/evals/eval_followup_classifier.py
+++ b/products/slack_app/evals/eval_followup_classifier.py
@@ -62,10 +62,27 @@ THREAD_POSTHOG_ASKED = [
),
]
+# The agent acted on an assumption and asked to hear back only if it was wrong. Then a
+# person asked the room the same question.
+THREAD_POSTHOG_ASSUMED = [
+ *THREAD,
+ SlackThreadMessage(
+ user="posthog",
+ text="I put the fix in the click handler, because I assume the analytics wrapper is shared with other pages. If that is wrong, tell me and I will move it.",
+ ts="3.0",
+ ),
+]
+THREAD_POSTHOG_ASSUMED_PERSON_ASKED = [
+ *THREAD_POSTHOG_ASSUMED,
+ SlackThreadMessage(user="dana", text="do we know who looks after that wrapper these days?", ts="4.0"),
+]
+
THREADS = {
"default": THREAD,
"other_agent_asked": THREAD_OTHER_AGENT_ASKED,
"posthog_asked": THREAD_POSTHOG_ASKED,
+ "posthog_assumed": THREAD_POSTHOG_ASSUMED,
+ "posthog_assumed_person_asked": THREAD_POSTHOG_ASSUMED_PERSON_ASKED,
}
@@ -112,6 +129,13 @@ DIRECTED_CASES = [
expected=_routes(True),
metadata=_in_thread("posthog_asked"),
),
+ # The twin of `confirmation_of_our_assumption`: the same question, answered "wrong".
+ BaseEvalCase(
+ name="correction_of_our_assumption",
+ prompt="no, that wrapper is only used on checkout, put the fix there",
+ expected=_routes(True),
+ metadata=_in_thread("posthog_assumed"),
+ ),
BaseEvalCase(
name="follow_up_ask",
prompt="can you add a test for the logged-out case as well",
@@ -203,6 +227,22 @@ CHATTER_CASES = [
expected=_routes(False),
metadata=_in_thread("other_agent_asked"),
),
+ # The agent asked to hear back only if it was wrong. A reply that says it was right
+ # reads as an answer to its question, but there is nothing for it to do.
+ BaseEvalCase(
+ name="confirmation_of_our_assumption",
+ prompt="yeah it's shared, that's the right call",
+ expected=_routes(False),
+ metadata=_in_thread("posthog_assumed"),
+ ),
+ # The same confirmation, given as the answer to a colleague and carried on into talk
+ # among people.
+ BaseEvalCase(
+ name="answer_to_a_persons_question",
+ prompt="yeah that's us, we tend to be the only ones touching it. not sure we need a dedicated owner for it though, open to ideas",
+ expected=_routes(False),
+ metadata=_in_thread("posthog_assumed_person_asked"),
+ ),
# A thread kept as a running list. The item carries an implicit work item, which is a
# sharper lure than a bare link.
BaseEvalCase(