Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

12.0 KB
Raw Blame
from collections.abc import Sequence
from typing import Any, ClassVar, Literal, Self

from langchain_core.runnables import RunnableConfig
from pydantic import BaseModel, Field

from posthog.models import Team, User

from ee.hogai.context.context import AssistantContextManager
from ee.hogai.tool import MaxTool
from ee.hogai.utils.prompt import format_prompt_string
from ee.hogai.utils.types.base import AssistantState, NodePath

TODO_WRITE_PROMPT = """
Use this tool to build and maintain a structured to-do list for the current session. It helps you monitor progress, organize complex work, and show thoroughness. It also makes both task progress and the overall status of the user’s requests clear to the user.

# When to use this tool
Use it proactively in these situations:

1. Complex, multi-step work – when a task needs 3+ distinct steps or actions
2. Non-trivial tasks – work that requires careful planning or multiple operations
3. User explicitly asks for a to-do list – when they request it directly
4. User supplies multiple tasks – e.g., a numbered or comma-separated list
5. After new instructions arrive – immediately capture the requirements as to-dos
6. When you begin a task – set it to `in_progress` BEFORE starting; ideally only one `in_progress` item at a time
7. After finishing a task – mark it `completed` and add any follow-ups discovered during execution

# When NOT to use this tool
Skip it when:
1. There's only a single, straightforward task
2. The task is trivial and tracking adds no organizational value
3. It can be finished in fewer than 3 trivial steps
4. The exchange is purely conversational or informational

NOTE: If there's just one trivial task, don't use the tool–simply do the task directly.

# Examples of when to use the todo list

{{{positive_todo_examples}}}

# Examples of when NOT to use the todo list

{{{negative_todo_examples}}}

# Task states and management

1. **Task States**: Use these states to track progress:
  - pending: Task not yet started
  - in_progress: Currently working on (limit to ONE task at a time)
  - completed: Finished successfully

2. **Managing Tasks**:
  - Update statuses in real time as you work
  - Mark tasks complete IMMEDIATELY when done–don’t batch them
  - Keep only ONE task `in_progress` at any moment
  - Finish the current task before starting another
  - Remove tasks that are no longer relevant

3. **Completion Rules**:
  - Mark a task `completed` only when it’s FULLY done
  - If you hit errors, blockers, or can’t finish, leave it `in_progress`
  - When blocked, add a new task describing what must be resolved
  - Never mark `completed` if:
    - Implementation is partial
    - Required data couldn’t be found

4. **Task Breakdown**:
  - Create specific, actionable items
  - Break complex tasks into smaller, manageable steps
  - Use clear, descriptive task names

When unsure, use this tool. Proactive task management shows attentiveness and helps ensure all requirements are met.
""".strip()

TODO_WRITE_EXAMPLE_PROMPT = """
<example>
{{{example}}}
<reasoning>
{{{reasoning}}}
</reasoning>
</example>
""".strip()


class TodoWriteExample(BaseModel):
    """
    Custom agent example to correct the agent's behavior through few-shot prompting.
    The example will be formatted as follows:
    ```
    <example>
    {example}

    <reasoning>
    {reasoning}
    </reasoning>
    </example>
    ```
    """

    example: str
    reasoning: str | None = None


POSITIVE_EXAMPLE_INSIGHT_WITH_SEGMENTATION = """
User: how many users have chatted with the AI assistant from the US?
Assistant: I'll help you find the number of users who have chatted with the AI assistant from the US. Let me create a todo list to track this implementation.
*Creates todo list with the following items:*
1. Find the relevant events to "chatted with the AI assistant"
2. Find the relevant properties of the events and persons to narrow down data to users from specific country
3. Retrieve the sample property values for found properties
4. Create the structured plan of the insight by using the data retrieved in the previous steps
5. Generate the insight
6. Analyze retrieved data
*Begins working on the first task*
""".strip()

POSITIVE_EXAMPLE_INSIGHT_WITH_SEGMENTATION_REASONING = """
The assistant used the todo list because:
1. Creating an insight requires understanding the taxonomy: events, properties, and property values are relevant to the user's query.
2. The user query requests additional segmentation.
3. Property values might require retrieving sample property values to understand the data better.
4. Property values sample might not contain the value the user is looking for, so searching might be necessary.
5. Taxonomy might have multiple combinations of data that will equally answer the question.
""".strip()

POSITIVE_EXAMPLE_COMPANY_CHURN_ANALYSIS = """
User: Has eleventy churned?
Assistant: Let me first search for a company with name "eleventy".
*Uses the search tool to find a property value with the "eleventy" value in the project*
Assistant: I've found a property value with the "Eleventy.ai" value. I'm going to search for existing insights tracking the customer churn rate.
*Uses the search tools to find insights tracking the customer churn rate in the project*
Assistant: I've found 0 matching insights. Let me find matching insights in the data.
*Uses the list_data tool with the "insights" kind to iterate over project insights*
Assistant: No matches. Let me create a new insight checking if the company "Eleventy.ai" has churned. I'm going to create a todo list to track these changes.
*Creates a todo list with specific steps to create a new insight*
""".strip()

POSITIVE_EXAMPLE_COMPANY_CHURN_ANALYSIS_REASONING = """
The assistant used the todo list because:
1. First, the assistant searched to understand the scope of the task
2. After the assistant verified that there isn't an insight tracking the customer churn rate, it determined this was a complex task with multiple steps
3. The todo list helps ensure every instance is tracked and updated systematically
""".strip()

POSITIVE_EXAMPLE_MULTIPLE_METRICS_ANALYSIS = """
User: Check why onboarding completion rate has dropped and if it is connected with a low sign-up count
Assistant: I'll help you analyze the reasons why the metrics have changed. First, let's add all the features to the todo list.
*Creates a todo list breaking down each analysis into specific tasks based on the project data*
Assistant: Let's start with analyzing the sign-up count. This will involve retrieving the events and might involve retrieving additional data.
""".strip()

POSITIVE_EXAMPLE_MULTIPLE_METRICS_ANALYSIS_REASONING = """
The assistant used the todo list because:
1. The user requested multiple complex analysis for different metrics that must be separate insights
2. The todo list helps organize these large requests into manageable tasks
3. This approach allows for tracking progress across the entire request
""".strip()

NEGATIVE_EXAMPLE_SIMPLE_QUERY_EXPLANATION = """
User: What does this query do?
Assistant: Let me analyze the query you provided.
*Reads the attached context in the conversation history*
Assistant: The query is retrieving the sign-up count for the last 30 days.
""".strip()

NEGATIVE_EXAMPLE_SIMPLE_QUERY_EXPLANATION_REASONING = """
The assistant did not use the todo list because this is a single, trivial task that can be completed in one step. There's no need to track multiple tasks or steps for such a straightforward request.
""".strip()

NEGATIVE_EXAMPLE_DOCUMENTATION_REQUEST = """
User: How can I capture exception in my Next.js application?
Assistant: Let me search for the relevant documentation.
*Uses the search tool to find the relevant documentation*
Assistant: I've found the relevant documentation.
*Summarizes and returns the answer to the user's question*
""".strip()

NEGATIVE_EXAMPLE_DOCUMENTATION_REQUEST_REASONING = """
The assistant did not use the todo list because this is an informational request. The user is simply asking for help, not for the assistant to perform multiple steps or tasks.
""".strip()

POSITIVE_TODO_EXAMPLES = [
    TodoWriteExample(
        example=POSITIVE_EXAMPLE_INSIGHT_WITH_SEGMENTATION,
        reasoning=POSITIVE_EXAMPLE_INSIGHT_WITH_SEGMENTATION_REASONING,
    ),
    TodoWriteExample(
        example=POSITIVE_EXAMPLE_COMPANY_CHURN_ANALYSIS, reasoning=POSITIVE_EXAMPLE_COMPANY_CHURN_ANALYSIS_REASONING
    ),
    TodoWriteExample(
        example=POSITIVE_EXAMPLE_MULTIPLE_METRICS_ANALYSIS,
        reasoning=POSITIVE_EXAMPLE_MULTIPLE_METRICS_ANALYSIS_REASONING,
    ),
]

NEGATIVE_TODO_EXAMPLES = [
    TodoWriteExample(
        example=NEGATIVE_EXAMPLE_SIMPLE_QUERY_EXPLANATION,
        reasoning=NEGATIVE_EXAMPLE_SIMPLE_QUERY_EXPLANATION_REASONING,
    ),
    TodoWriteExample(
        example=NEGATIVE_EXAMPLE_DOCUMENTATION_REQUEST, reasoning=NEGATIVE_EXAMPLE_DOCUMENTATION_REQUEST_REASONING
    ),
]


# Has its unique schema that doesn't match the Deep Research schema
class TodoItem(BaseModel):
    content: str = Field(..., min_length=1)
    status: Literal["pending", "in_progress", "completed"]
    id: str


class TodoWriteToolArgs(BaseModel):
    todos: list[TodoItem] = Field(..., description="The updated todo list")


class TodoWriteTool(MaxTool):
    name: Literal["todo_write"] = "todo_write"
    args_schema: type[BaseModel] = TodoWriteToolArgs

    POSITIVE_TODO_EXAMPLES: ClassVar[list[TodoWriteExample]] = POSITIVE_TODO_EXAMPLES
    NEGATIVE_TODO_EXAMPLES: ClassVar[list[TodoWriteExample]] = NEGATIVE_TODO_EXAMPLES

    async def _arun_impl(self, todos: list[TodoItem]) -> tuple[str, None]:
        return (
            "The to-dos were updated successfully. Please keep using the to-do list to track your progress, and continue with any active tasks as appropriate.",
            None,
        )

    @staticmethod
    def format_todo_list(todos: list[TodoItem] | dict[str, Any]) -> str:
        """
        Format a todo list into human-readable content.

        Args:
            todos: Either a list of TodoItem objects or a dict with 'todos' key (tool call args)

        Returns:
            Formatted string representation of the todo list
        """
        # Parse args dict if needed
        if isinstance(todos, dict):
            parsed_args = TodoWriteToolArgs(**todos)
            todos = parsed_args.todos

        if not todos:
            return "Your todo list is empty."

        lines = ["Your current todo list:"]
        for todo in todos:
            status = todo.status
            content = todo.content

            # Use status emoji/indicator
            if status == "completed":
                indicator = "✓"
            elif status == "in_progress":
                indicator = "→"
            else:  # pending
                indicator = "○"

            lines.append(f"{indicator} [{status}] {content}")

        return "\n".join(lines)

    @classmethod
    async def create_tool_class(
        cls,
        *,
        team: Team,
        user: User,
        node_path: tuple[NodePath, ...] | None = None,
        state: AssistantState | None = None,
        config: RunnableConfig | None = None,
        context_manager: AssistantContextManager | None = None,
        positive_examples: Sequence[TodoWriteExample] | None = None,
        negative_examples: Sequence[TodoWriteExample] | None = None,
    ) -> Self:
        formatted_prompt = format_prompt_string(
            TODO_WRITE_PROMPT,
            positive_todo_examples=_format_todo_write_examples(positive_examples or cls.POSITIVE_TODO_EXAMPLES),
            negative_todo_examples=_format_todo_write_examples(negative_examples or cls.NEGATIVE_TODO_EXAMPLES),
        )
        return cls(
            team=team,
            user=user,
            node_path=node_path,
            state=state,
            config=config,
            context_manager=context_manager,
            description=formatted_prompt,
        )


def _format_todo_write_examples(examples: Sequence[TodoWriteExample]) -> str:
    return "\n".join(
        [
            format_prompt_string(TODO_WRITE_EXAMPLE_PROMPT, example=example.example, reasoning=example.reasoning)
            for example in examples
        ]
    )