Building AI agents, atomically
refactor(example/deep-research): proper multi-agent research pipeline
KennyVaneetvelde6e5ccb8d8912
diff --git a/atomic-examples/deep-research/README.md b/atomic-examples/deep-research/README.mdindex ef4c603..b05d3e9 100644--- a/atomic-examples/deep-research/README.md+++ b/atomic-examples/deep-research/README.md@@ -1,6 +1,19 @@# Deep Research Agent-This directory contains the Deep Research Agent example for the Atomic Agents project. This example demonstrates how to create an intelligent research assistant that performs web searches and provides detailed answers with relevant follow-up questions.+A didactic example of a proper deep-research pipeline built out of small, single-purpose Atomic Agents.++Unlike a typical "search-and-summarise" agent — generate one set of queries, fetch results, write an answer — this example iterates: it plans sub-topics, researches each one across multiple depth levels, reflects on whether each has enough coverage, and produces a report where every claim is tied to a registered source.++## Pipeline++1. **Plan.** A `PlannerAgent` breaks the question into 3–5 durable sub-topics, each seeded with a handful of queries.+2. **Research** (per sub-topic, up to N iterations):+ - Search (SearXNG) and scrape the top new URLs.+ - `ExtractorAgent` pulls atomic, citable claims from each scraped page.+ - `ReflectorAgent` decides whether the sub-topic has enough material, or emits follow-up queries for the next iteration.+3. **Write.** `WriterAgent` drafts a cited report from the accumulated state, then runs a second pass over its own draft to strip any sentence whose citation doesn't correspond to a real source.++Every agent has a single responsibility and reads / contributes to a shared `ResearchState` object. The loop itself lives in `main.py` as plain Python — no megagent, no hidden control flow.## Getting Started@@ -26,51 +39,49 @@ This directory contains the Deep Research Agent example for the Atomic Agents prSEARXNG_BASE_URL=http://localhost:8080SEARXNG_API_KEY=your_searxng_secret_key```-- **Important**: To find your SearXNG secret key, check the `secret_key` value in your SearXNG `settings.yml` file, typically located at `/etc/searxng/settings.yml` or in the SearXNG installation directory.5. **Set up SearXNG:**- - Install SearXNG from the [official repository](https://github.com/searxng/searxng)- - Default configuration expects SearXNG at `http://localhost:8080`+ - Install from the [official repository](https://github.com/searxng/searxng).+ - Default configuration expects SearXNG at `http://localhost:8080`.+ - JSON output must be enabled in `settings.yml` (look for the `formats:` key).-6. **Run the Deep Research Agent:**+6. **Run a research query:**```bash- uv run python deep_research/main.py+ uv run python -m deep_research "What is the current state of fusion energy research?"```-## File Explanation--### 1. Agents (`agents/`)--- **ChoiceAgent** (`choice_agent.py`): Determines when new searches are needed-- **QueryAgent** (`query_agent.py`): Generates optimized search queries-- **QuestionAnsweringAgent** (`qa_agent.py`): Processes information and generates responses--### 2. Tools (`tools/`)--- **SearXNG Search** (`searxng_search.py`): Performs web searches across multiple engines-- **Webpage Scraper** (`webpage_scraper.py`): Extracts and processes web content--### 3. Main (`main.py`)--The entry point for the Deep Research Agent application. It orchestrates the research process:-- Deciding when to perform new searches-- Generating and executing search queries-- Processing information and providing answers-- Suggesting relevant follow-up questions--## Example Usage--The agent can handle various research queries and provides:-- Detailed answers based on current web information-- Relevant citations and sources-- Specific follow-up questions for deeper exploration-- Context-aware responses that build on previous interactions--## Contributing--Contributions are welcome! Please fork the repository and submit a pull request with your improvements.+## File Layout++```+deep_research/+├── config.py # Model + connectivity + research budgets+├── state.py # ResearchState dataclass — the one source of truth+├── context_providers.py # Renders state into agent system prompts+├── agents/+│ ├── planner_agent.py # Question → sub-topics (with initial queries)+│ ├── extractor_agent.py # One scraped source → atomic claims+│ ├── reflector_agent.py # Sub-topic state → sufficient? + next queries+│ └── writer_agent.py # Full state → cited report (draft + verify passes)+├── tools/+│ ├── searxng_search.py+│ └── webpage_scraper.py+└── main.py # Plain orchestrator: plan → research → write+```++## Budgets++All limits live in `ResearchBudget` inside `config.py`. Tune to taste:++| Knob | Default | Meaning |+|---|---|---|+| `num_sub_topics` | 4 | Plan width |+| `max_depth_per_sub_topic` | 2 | Max iterations per sub-topic; reflector can stop earlier |+| `search_results_per_query` | 5 | SearXNG page size |+| `scrape_top_n_per_iteration` | 3 | New URLs scraped per iteration |+| `hard_call_cap` | 80 | Global safety net on total agent calls |++Worst-case run: roughly 50 agent calls and 24 scrapes.## License-This project is licensed under the MIT License. See the [LICENSE](../../LICENSE) file for details.+MIT — see the [LICENSE](../../LICENSE) file.diff --git a/atomic-examples/deep-research/deep_research/__main__.py b/atomic-examples/deep-research/deep_research/__main__.pynew file mode 100644index 0000000..8c87d08--- /dev/null+++ b/atomic-examples/deep-research/deep_research/__main__.py@@ -0,0 +1,16 @@+"""Package entry point — so ``python -m deep_research "question"`` works.++The real orchestrator lives in ``main.py``; this file is just the Python+convention that makes the package directly runnable.+"""++import sys++from deep_research.main import run++if __name__ == "__main__":+ args = sys.argv[1:]+ if not args:+ print('Usage: python -m deep_research "your question"', file=sys.stderr)+ sys.exit(1)+ run(" ".join(args))diff --git a/atomic-examples/deep-research/deep_research/agents/choice_agent.py b/atomic-examples/deep-research/deep_research/agents/choice_agent.pydeleted file mode 100644index cf79c6d..0000000--- a/atomic-examples/deep-research/deep_research/agents/choice_agent.py+++ /dev/null@@ -1,58 +0,0 @@-import instructor-import openai-from pydantic import Field-from atomic_agents import BaseIOSchema, AtomicAgent, AgentConfig-from atomic_agents.context import SystemPromptGenerator--from deep_research.config import ChatConfig---class ChoiceAgentInputSchema(BaseIOSchema):- """Input schema for the ChoiceAgent."""-- user_message: str = Field(..., description="The user's latest message or question")- decision_type: str = Field(..., description="Explanation of the type of decision to make")---class ChoiceAgentOutputSchema(BaseIOSchema):- """Output schema for the ChoiceAgent."""-- reasoning: str = Field(..., description="Detailed explanation of the decision-making process")- decision: bool = Field(..., description="The final decision based on the analysis")---choice_agent = AtomicAgent[ChoiceAgentInputSchema, ChoiceAgentOutputSchema](- AgentConfig(- client=instructor.from_openai(openai.OpenAI(api_key=ChatConfig.api_key)),- model=ChatConfig.model,- model_api_parameters={"reasoning_effort": ChatConfig.reasoning_effort, "temperature": 0.1},- system_prompt_generator=SystemPromptGenerator(- background=[- "You are a decision-making agent that determines whether a new web search is needed to answer the user's question.",- "Your primary role is to analyze whether the existing context contains sufficient, up-to-date information to answer the question.",- "You must output a clear TRUE/FALSE decision - TRUE if a new search is needed, FALSE if existing context is sufficient.",- ],- steps=[- "1. Analyze the user's question to determine whether or not an answer warrants a new search",- "2. Review the available web search results",- "3. Determine if existing information is sufficient and relevant",- "4. Make a binary decision: TRUE for new search, FALSE for using existing context",- ],- output_instructions=[- "Your reasoning must clearly state WHY you need or don't need new information",- "If the web search context is empty or irrelevant, always decide TRUE for new search",- "If the question is time-sensitive, check the current date to ensure context is recent",- "For ambiguous cases, prefer to gather fresh information",- "Your decision must match your reasoning - don't contradict yourself",- ],- ),- )-)---if __name__ == "__main__":- # Example usage for search decision- search_example = choice_agent.run(- ChoiceAgentInputSchema(user_message="Who won the nobel prize in physics in 2024?", decision_type="needs_search")- )- print(search_example)diff --git a/atomic-examples/deep-research/deep_research/agents/extractor_agent.py b/atomic-examples/deep-research/deep_research/agents/extractor_agent.pynew file mode 100644index 0000000..f69b6f6--- /dev/null+++ b/atomic-examples/deep-research/deep_research/agents/extractor_agent.py@@ -0,0 +1,75 @@+"""+ExtractorAgent — pulls atomic claims out of one scraped source.++Called once per (sub-topic, source) pair. The orchestrator feeds in the+raw markdown content from the scraper and the agent returns a small+list of factual claims plus any follow-up questions the content raises.++We keep claims short and atomic so the writer can cite them individually+in the final report. The agent is deliberately *not* asked to assign+source IDs — the orchestrator already knows which source it passed in+and tags the claims before appending them to the state.+"""++import instructor+import openai+from pydantic import Field++from atomic_agents import AgentConfig, AtomicAgent, BaseIOSchema+from atomic_agents.context import SystemPromptGenerator++from deep_research.config import ChatConfig+++class ExtractorInput(BaseIOSchema):+ """Input schema for the ExtractorAgent."""++ sub_topic: str = Field(..., description="Which sub-topic the orchestrator is researching right now.")+ source_url: str = Field(..., description="The URL the content was scraped from (for citation context).")+ source_title: str = Field(..., description="The page's title.")+ content: str = Field(..., description="Raw scraped content in markdown form.")+++class ExtractorOutput(BaseIOSchema):+ """Output schema for the ExtractorAgent."""++ claims: list[str] = Field(+ ...,+ description=(+ "Atomic, single-sentence factual claims relevant to the sub-topic. "+ "One claim per line. Skip anything that isn't directly supported by the content."+ ),+ )+ new_questions: list[str] = Field(+ ...,+ description=(+ "Follow-up questions the content surfaces that aren't yet answered. "+ "The reflector may turn these into next-round queries."+ ),+ )+++extractor_agent = AtomicAgent[ExtractorInput, ExtractorOutput](+ AgentConfig(+ client=instructor.from_openai(openai.OpenAI(api_key=ChatConfig.api_key)),+ model=ChatConfig.model,+ model_api_parameters={"reasoning_effort": ChatConfig.reasoning_effort},+ system_prompt_generator=SystemPromptGenerator(+ background=[+ "You are a research analyst. You read one source at a time and extract the factual claims "+ "it makes that are relevant to the current sub-topic.",+ ],+ steps=[+ "Read the scraped content carefully.",+ "Extract claims that are (a) factual, (b) relevant to the sub-topic, (c) directly supported by the text.",+ "Note follow-up questions the content raises but doesn't answer.",+ ],+ output_instructions=[+ "Each claim must be a single, self-contained sentence.",+ "Do NOT include filler like 'according to the article' — just state the claim.",+ "Aim for 3–8 claims per source; fewer is fine if the source is thin.",+ "If the content is irrelevant or empty, return an empty claims list.",+ ],+ ),+ )+)diff --git a/atomic-examples/deep-research/deep_research/agents/planner_agent.py b/atomic-examples/deep-research/deep_research/agents/planner_agent.pynew file mode 100644index 0000000..01bb3b1--- /dev/null+++ b/atomic-examples/deep-research/deep_research/agents/planner_agent.py@@ -0,0 +1,81 @@+"""+PlannerAgent — decomposes a research question into durable sub-topics.++Sub-topics are the *breadth* axis of the pipeline. They are decided once+by the planner and stay fixed; only the queries *within* a sub-topic+iterate as we learn more. Keeping the plan small and durable makes the+final report easy to organise section-by-section.++The planner runs before any state exists, so it takes only the raw+question and does not register any context providers.+"""++import instructor+import openai+from pydantic import Field++from atomic_agents import AgentConfig, AtomicAgent, BaseIOSchema+from atomic_agents.context import SystemPromptGenerator++from deep_research.config import ChatConfig+++class PlannerInput(BaseIOSchema):+ """Input schema for the PlannerAgent."""++ question: str = Field(..., description="The user's research question.")+ num_sub_topics: int = Field(+ ...,+ description="How many sub-topics to produce. 3–5 is a good range for a multi-page report.",+ )+++class PlannedSubTopic(BaseIOSchema):+ """One entry in the research plan."""++ name: str = Field(+ ...,+ description="Short label (2–6 words), e.g. 'history and origins' or 'current applications'.",+ )+ initial_queries: list[str] = Field(+ ...,+ description="2–3 seed web-search queries to kick off this sub-topic. Keywords and operators, not full sentences.",+ )+++class PlannerOutput(BaseIOSchema):+ """Output schema for the PlannerAgent."""++ sub_topics: list[PlannedSubTopic] = Field(+ ...,+ description="Sub-topics that together cover the research question without overlap.",+ )+++planner_agent = AtomicAgent[PlannerInput, PlannerOutput](+ AgentConfig(+ client=instructor.from_openai(openai.OpenAI(api_key=ChatConfig.api_key)),+ model=ChatConfig.model,+ model_api_parameters={"reasoning_effort": ChatConfig.reasoning_effort},+ system_prompt_generator=SystemPromptGenerator(+ background=[+ "You are a research planner. Your job is to break a broad question into durable sub-topics.",+ "Good sub-topics are orthogonal (they don't overlap), collectively comprehensive, "+ "and each one can be researched independently of the others.",+ ],+ steps=[+ "Identify the core concept in the question.",+ "List the distinct angles a thorough report would need to cover "+ "(e.g. history, mechanics, applications, controversies, outlook — "+ "pick whatever is appropriate for the topic).",+ "Select the N most important angles, where N is the requested count.",+ "For each sub-topic, draft 2–3 seed search queries phrased as search-engine input.",+ ],+ output_instructions=[+ "Sub-topic names must be short (2–6 words).",+ "Initial queries must read like search-engine input, not natural-language sentences.",+ "Do not duplicate sub-topics or queries across the plan.",+ ],+ ),+ )+)diff --git a/atomic-examples/deep-research/deep_research/agents/qa_agent.py b/atomic-examples/deep-research/deep_research/agents/qa_agent.pydeleted file mode 100644index f587739..0000000--- a/atomic-examples/deep-research/deep_research/agents/qa_agent.py+++ /dev/null@@ -1,63 +0,0 @@-import instructor-import openai-from pydantic import Field-from atomic_agents import BaseIOSchema, AtomicAgent, AgentConfig-from atomic_agents.context import SystemPromptGenerator--from deep_research.config import ChatConfig---class QuestionAnsweringAgentInputSchema(BaseIOSchema):- """This is the input schema for the QuestionAnsweringAgent."""-- question: str = Field(..., description="The question to answer.")---class QuestionAnsweringAgentOutputSchema(BaseIOSchema):- """This is the output schema for the QuestionAnsweringAgent."""-- answer: str = Field(..., description="The answer to the question.")- follow_up_questions: list[str] = Field(- ...,- description=(- "Specific questions about the topic that would help the user learn more details about the subject matter. "- "For example, if discussing a Nobel Prize winner, suggest questions about their research, impact, or "- "related scientific concepts."- ),- )---question_answering_agent = AtomicAgent[QuestionAnsweringAgentInputSchema, QuestionAnsweringAgentOutputSchema](- AgentConfig(- client=instructor.from_openai(openai.OpenAI(api_key=ChatConfig.api_key)),- model=ChatConfig.model,- model_api_parameters={"reasoning_effort": ChatConfig.reasoning_effort},- system_prompt_generator=SystemPromptGenerator(- background=[- "You are an expert question answering agent focused on providing factual information and encouraging deeper topic exploration.",- "For general greetings or non-research questions, provide relevant questions about the system's capabilities and research functions.",- ],- steps=[- "Analyze the question and identify the core topic",- "Answer the question using available information",- "For topic-specific questions, generate follow-up questions that explore deeper aspects of the same topic",- "For general queries about the system, suggest questions about research capabilities and functionality",- ],- output_instructions=[- "Answer in a direct, informative manner",- "NEVER generate generic conversational follow-ups like 'How are you?' or 'What would you like to know?'",- "For topic questions, follow-up questions MUST be about specific aspects of that topic",- "For system queries, follow-up questions should be about specific research capabilities",- "Example good follow-ups for a Nobel Prize question:",- "- What specific discoveries led to their Nobel Prize?",- "- How has their research influenced their field?",- "- What other scientists collaborated on this research?",- "Example good follow-ups for system queries:",- "- What types of sources do you use for research?",- "- How do you verify information accuracy?",- "- What are the limitations of your search capabilities?",- ],- ),- model_api_parameters={"temperature": 0.1},- )-)diff --git a/atomic-examples/deep-research/deep_research/agents/query_agent.py b/atomic-examples/deep-research/deep_research/agents/query_agent.pydeleted file mode 100644index cdc8d1d..0000000--- a/atomic-examples/deep-research/deep_research/agents/query_agent.py+++ /dev/null@@ -1,42 +0,0 @@-from deep_research.config import ChatConfig-import instructor-import openai-from pydantic import Field-from atomic_agents import BaseIOSchema, AtomicAgent, AgentConfig-from atomic_agents.context import SystemPromptGenerator--from deep_research.tools.searxng_search import SearXNGSearchToolInputSchema---class QueryAgentInputSchema(BaseIOSchema):- """This is the input schema for the QueryAgent."""-- instruction: str = Field(..., description="A detailed instruction or request to generate search engine queries for.")- num_queries: int = Field(..., description="The number of search queries to generate.")---query_agent = AtomicAgent[QueryAgentInputSchema, SearXNGSearchToolInputSchema](- AgentConfig(- client=instructor.from_openai(openai.OpenAI(api_key=ChatConfig.api_key)),- model=ChatConfig.model,- model_api_parameters={"reasoning_effort": ChatConfig.reasoning_effort},- system_prompt_generator=SystemPromptGenerator(- background=[- (- "You are an expert search engine query generator with a deep understanding of which"- "queries will maximize the number of relevant results."- )- ],- steps=[- "Analyze the given instruction to identify key concepts and aspects that need to be researched",- "For each aspect, craft a search query using appropriate search operators and syntax",- "Ensure queries cover different angles of the topic (technical, practical, comparative, etc.)",- ],- output_instructions=[- "Return exactly the requested number of queries",- "Format each query like a search engine query, not a natural language question",- "Each query should be a concise string of keywords and operators",- ],- ),- )-)diff --git a/atomic-examples/deep-research/deep_research/agents/reflector_agent.py b/atomic-examples/deep-research/deep_research/agents/reflector_agent.pynew file mode 100644index 0000000..e7fc9f3--- /dev/null+++ b/atomic-examples/deep-research/deep_research/agents/reflector_agent.py@@ -0,0 +1,85 @@+"""+ReflectorAgent — decides, after each depth iteration, whether to keep+researching the sub-topic or call it done.++Deep research's defining move. Without the reflector we'd either+over-search easy sub-topics (wasting tokens) or under-search hard ones+(producing a shallow report). The reflector looks at the learnings+gathered so far for the sub-topic and either says "good enough" or+emits the specific follow-up queries to run next.++The reflector sees the full state via the ``ResearchStateProvider``,+so it can judge sufficiency in light of what the neighbouring+sub-topics already cover.+"""++import instructor+import openai+from pydantic import Field++from atomic_agents import AgentConfig, AtomicAgent, BaseIOSchema+from atomic_agents.context import SystemPromptGenerator++from deep_research.config import ChatConfig+++class ReflectorInput(BaseIOSchema):+ """Input schema for the ReflectorAgent."""++ sub_topic: str = Field(..., description="The sub-topic being evaluated.")+ iterations_so_far: int = Field(+ ...,+ description="How many depth iterations have been completed for this sub-topic already.",+ )+ max_iterations: int = Field(+ ...,+ description="Hard cap. After this many iterations the orchestrator stops regardless of your decision.",+ )+++class ReflectorOutput(BaseIOSchema):+ """Output schema for the ReflectorAgent."""++ reasoning: str = Field(..., description="One short paragraph explaining the decision.")+ sufficient: bool = Field(+ ...,+ description=(+ "True if the learnings for this sub-topic are rich enough to write a section of the report. "+ "False if more research is needed."+ ),+ )+ next_queries: list[str] = Field(+ ...,+ description=(+ "If sufficient is False, 2–3 new search queries that target the remaining gaps. "+ "If sufficient is True, return an empty list."+ ),+ )+++reflector_agent = AtomicAgent[ReflectorInput, ReflectorOutput](+ AgentConfig(+ client=instructor.from_openai(openai.OpenAI(api_key=ChatConfig.api_key)),+ model=ChatConfig.model,+ model_api_parameters={"reasoning_effort": ChatConfig.reasoning_effort},+ system_prompt_generator=SystemPromptGenerator(+ background=[+ "You are a research editor. After each round of searching and extraction, you decide "+ "whether the current sub-topic has enough material to stand on its own in the final report.",+ "You have full visibility into the research state — sources, learnings, and the plan.",+ ],+ steps=[+ "Look only at the learnings tagged with the given sub-topic.",+ "Ask: could a reader write a coherent, cited section from this material?",+ "If yes: mark sufficient=true and return no queries.",+ "If no: identify the specific gap and produce 2–3 queries that target it.",+ ],+ output_instructions=[+ "Be decisive. 'Maybe' is never the right answer.",+ "Prefer marking sufficient=true once you have 4+ substantive, non-duplicate claims.",+ "Prefer marking sufficient=true on the final iteration regardless of coverage — the orchestrator will stop anyway.",+ "Next queries, if any, must be keywords-and-operators style, not sentences.",+ ],+ ),+ )+)diff --git a/atomic-examples/deep-research/deep_research/agents/writer_agent.py b/atomic-examples/deep-research/deep_research/agents/writer_agent.pynew file mode 100644index 0000000..355e554--- /dev/null+++ b/atomic-examples/deep-research/deep_research/agents/writer_agent.py@@ -0,0 +1,88 @@+"""+WriterAgent — turns the accumulated research state into a cited report.++Runs twice: the first call produces a draft, the second is a cheap+verification pass that rejects any sentence whose citation marker+(``[S3]`` etc.) doesn't correspond to a real source in the state.+This is the single trick that separates our writer from the typical+open-source "deep research" agent — it guarantees every claim in the+output is backed by a registered source.++Both passes use the same agent (same schema, same prompt) but with a+different input mode — see ``WriterMode``.+"""++from typing import Literal++import instructor+import openai+from pydantic import Field++from atomic_agents import AgentConfig, AtomicAgent, BaseIOSchema+from atomic_agents.context import SystemPromptGenerator++from deep_research.config import ChatConfig++WriterMode = Literal["draft", "verify"]+++class WriterInput(BaseIOSchema):+ """Input schema for the WriterAgent."""++ question: str = Field(..., description="The original research question.")+ mode: WriterMode = Field(+ ...,+ description=(+ "'draft' to compose the report from scratch using the research state; "+ "'verify' to rewrite an existing draft, removing any sentence whose citation doesn't match a real source."+ ),+ )+ draft: str = Field(+ "",+ description="When mode='verify', the draft to audit. Leave blank for mode='draft'.",+ )+++class WriterOutput(BaseIOSchema):+ """Output schema for the WriterAgent."""++ report: str = Field(+ ...,+ description=(+ "Markdown report. Every non-trivial sentence must end with one or more citation markers "+ "like [S1] or [S2, S5], referencing sources by ID."+ ),+ )+ headline: str = Field(..., description="One-sentence top-line takeaway.")+++writer_agent = AtomicAgent[WriterInput, WriterOutput](+ AgentConfig(+ client=instructor.from_openai(openai.OpenAI(api_key=ChatConfig.api_key)),+ model=ChatConfig.model,+ model_api_parameters={"reasoning_effort": ChatConfig.reasoning_effort},+ system_prompt_generator=SystemPromptGenerator(+ background=[+ "You are a research writer. You compose cited markdown reports from a structured research state "+ "provided in your system context (sources with IDs, and learnings grouped by sub-topic).",+ ],+ steps=[+ "In 'draft' mode:",+ " 1. Read the research state (sources and learnings) from the system context.",+ " 2. Organise the report with one section per sub-topic, in a logical order.",+ " 3. Every factual sentence cites the source(s) it's based on using [S1] / [S2, S4] markers.",+ " 4. End with a '## Sources' section. Format each entry as a bullet list: `- [Sn]: <title> — <url>`. Do NOT append a trailing [Sn] after the URL.",+ "In 'verify' mode:",+ " 1. Read the draft provided in the input.",+ " 2. Remove any sentence that carries a citation marker not present in the research state's sources.",+ " 3. Remove any factual sentence with no citation at all.",+ " 4. Return the cleaned report verbatim otherwise — do not paraphrase, do not add new material.",+ ],+ output_instructions=[+ "Use markdown headings (## per sub-topic).",+ "Only cite source IDs that actually exist in the provided research state.",+ "The headline is one sentence, max 20 words, and stands on its own.",+ ],+ ),+ )+)diff --git a/atomic-examples/deep-research/deep_research/config.py b/atomic-examples/deep-research/deep_research/config.pyindex 658995e..2a109e1 100644--- a/atomic-examples/deep-research/deep_research/config.py+++ b/atomic-examples/deep-research/deep_research/config.py@@ -1,39 +1,65 @@+"""Configuration for the deep-research example."""+import osfrom dataclasses import dataclass-from typing import Setdef get_api_key() -> str:- """Retrieve API key from environment or raise error"""api_key = os.getenv("OPENAI_API_KEY")if not api_key:- raise ValueError("API key not found. Please set the OPENAI_API_KEY environment variable.")+ raise ValueError("API key not found. Set the OPENAI_API_KEY environment variable.")return api_keydef get_searxng_base_url() -> str:- """Retrieve SearXNG base URL from environment or use default"""- base_url = os.getenv("SEARXNG_BASE_URL", "http://localhost:8080")- return base_url+ return os.getenv("SEARXNG_BASE_URL", "http://localhost:8080")def get_searxng_api_key() -> str:- """Retrieve SearXNG API key from environment"""- api_key = os.getenv("SEARXNG_API_KEY")- return api_key+ return os.getenv("SEARXNG_API_KEY")@dataclassclass ChatConfig:- """Configuration for the chat application"""+ """Model and connectivity settings. Not meant to be instantiated."""- api_key: str = get_api_key() # This becomes a class variable+ api_key: str = get_api_key()model: str = "gpt-5-mini"reasoning_effort: str = "low"- exit_commands: Set[str] = frozenset({"/exit", "/quit"})searxng_base_url: str = get_searxng_base_url()searxng_api_key: str = get_searxng_api_key()def __init__(self):- # Prevent instantiationraise TypeError("ChatConfig is not meant to be instantiated")+++@dataclass+class ResearchBudget:+ """Hard and soft limits on the research loop.++ These are the knobs that decide how *deep* the deep research goes.+ The orchestrator respects each independently: you can't escape the+ loop by satisfying only one.+ """++ # Breadth — how many sub-topics the planner produces.+ num_sub_topics: int = 4++ # Depth — max iterations *per* sub-topic. The reflector can stop earlier.+ max_depth_per_sub_topic: int = 2++ # Per-search and per-iteration throttles.+ search_results_per_query: int = 5+ scrape_top_n_per_iteration: int = 3++ # Hard cap across the whole run, in case an agent goes rogue or a loop bug slips through.+ hard_call_cap: int = 80++ # Max characters of scraped content passed to the extractor. A handful+ # of claims only needs a few thousand chars of context, and some pages+ # (long Wikipedia articles, badly-parsed PDFs) can blow the model's+ # context window otherwise.+ max_extractor_content_chars: int = 12_000++ def __init__(self):+ raise TypeError("ResearchBudget is not meant to be instantiated")diff --git a/atomic-examples/deep-research/deep_research/context_providers.py b/atomic-examples/deep-research/deep_research/context_providers.pyindex 6278a30..0c6cb05 100644--- a/atomic-examples/deep-research/deep_research/context_providers.py+++ b/atomic-examples/deep-research/deep_research/context_providers.py@@ -1,33 +1,63 @@-from dataclasses import dataclass+"""+Context providers for the deep-research pipeline.++Context providers are how runtime state reaches an agent's system prompt.+Here we use one provider that renders the shared ``ResearchState`` (see+``state.py``) so the extractor, reflector, and writer all see a+consistent, up-to-date picture without having to plumb data through+their input schemas.++The planner and query agents deliberately don't use this provider — the+planner runs before any state exists, and the query agent only needs+what can fit cleanly into its input schema.+"""+from datetime import datetime-from typing import List+from atomic_agents.context import BaseDynamicContextProvider+from deep_research.state import ResearchState-@dataclass-class ContentItem:- content: str- url: str+class ResearchStateProvider(BaseDynamicContextProvider):+ """Renders the current plan, sources, and learnings for agents that need full context."""-class ScrapedContentContextProvider(BaseDynamicContextProvider):- def __init__(self, title: str):+ def __init__(self, title: str, state: ResearchState):super().__init__(title=title)- self.content_items: List[ContentItem] = []+ self.state = statedef get_info(self) -> str:- return "\n\n".join(- [- f"Source {idx}:\nURL: {item.url}\nContent:\n{item.content}\n{'-' * 80}"- for idx, item in enumerate(self.content_items, 1)- ]- )+ if not self.state.sources and not self.state.learnings:+ return "No research has been done yet."++ lines: list[str] = []+ if self.state.sources:+ lines.append("### Sources")+ for s in self.state.sources:+ lines.append(f"[{s.id}] {s.title}")+ lines.append(f" {s.url}")-class CurrentDateContextProvider(BaseDynamicContextProvider):- def __init__(self, title: str, date_format: str = "%A %B %d, %Y"):+ if self.state.learnings:+ lines.append("")+ lines.append("### Learnings so far (grouped by sub-topic)")+ seen_topics: list[str] = []+ for l in self.state.learnings:+ if l.sub_topic not in seen_topics:+ seen_topics.append(l.sub_topic)+ for sub_topic in seen_topics:+ lines.append(f"**{sub_topic}**")+ for l in self.state.learnings_for(sub_topic):+ lines.append(f"- {l.text} [{l.source_id}]")++ return "\n".join(lines)+++class CurrentDateProvider(BaseDynamicContextProvider):+ """So agents don't get confused about what counts as 'recent'."""++ def __init__(self, title: str):super().__init__(title=title)- self.date_format = date_formatdef get_info(self) -> str:- return f"The current date in the format {self.date_format} is {datetime.now().strftime(self.date_format)}."+ return datetime.utcnow().strftime("Today is %A, %B %d, %Y.")diff --git a/atomic-examples/deep-research/deep_research/main.py b/atomic-examples/deep-research/deep_research/main.pyindex 4010118..288a423 100644--- a/atomic-examples/deep-research/deep_research/main.py+++ b/atomic-examples/deep-research/deep_research/main.py@@ -1,211 +1,241 @@-from deep_research.agents.query_agent import QueryAgentInputSchema, query_agent-from deep_research.agents.qa_agent import (- QuestionAnsweringAgentInputSchema,- question_answering_agent,- QuestionAnsweringAgentOutputSchema,-)-from deep_research.agents.choice_agent import choice_agent, ChoiceAgentInputSchema-from deep_research.tools.searxng_search import SearXNGSearchTool, SearXNGSearchToolConfig, SearXNGSearchToolInputSchema-from deep_research.tools.webpage_scraper import WebpageScraperTool, WebpageScraperToolInputSchema-from deep_research.context_providers import ContentItem, CurrentDateContextProvider, ScrapedContentContextProvider+"""+Deep-research orchestrator.++Reads like a recipe: plan → (per sub-topic) search → scrape → extract →+reflect → (maybe loop) → write. Each step is a call to a single-purpose+agent (see ``deep_research/agents/``) that reads from and contributes to+the shared ``ResearchState`` (see ``state.py``).++Run: ``python -m deep_research "your question here"``+"""++import sysfrom rich.console import Console-from rich.panel import Panelfrom rich.markdown import Markdown-from rich.table import Table-from rich import box-from rich.progress import Progress, SpinnerColumn, TextColumn+from rich.panel import Panel+from deep_research.agents.extractor_agent import ExtractorInput, extractor_agent+from deep_research.agents.planner_agent import PlannerInput, planner_agent+from deep_research.agents.reflector_agent import ReflectorInput, reflector_agent+from deep_research.agents.writer_agent import WriterInput, writer_agent+from deep_research.config import ChatConfig, ResearchBudget+from deep_research.context_providers import CurrentDateProvider, ResearchStateProvider+from deep_research.state import Learning, ResearchState, SubTopic+from deep_research.tools.searxng_search import (+ SearXNGSearchTool,+ SearXNGSearchToolConfig,+ SearXNGSearchToolInputSchema,+)+from deep_research.tools.webpage_scraper import (+ WebpageScraperTool,+ WebpageScraperToolInputSchema,+)console = Console()-WELCOME_MESSAGE = (- "Welcome to Deep Research - your AI-powered research assistant! I can help you explore and "- "understand any topic through detailed research and interactive discussion."-)-STARTER_QUESTIONS = [- "Can you help me research the latest AI news?",- "Who won the Nobel Prize in Physics this year?",- "Where can I learn more about quantum computing?",-]+def wire_context_providers(state: ResearchState) -> None:+ """Register the state + current-date providers on every agent that needs them.+ The planner is the only agent that runs before any state exists, so+ it gets only the date. All others see the live ``ResearchState``.+ """+ state_provider = ResearchStateProvider("Research State", state)+ date_provider = CurrentDateProvider("Current Date")-def perform_search_and_update_context(- user_message: str, scraped_content_context_provider: ScrapedContentContextProvider-) -> None:- with Progress(- SpinnerColumn(),- TextColumn("[progress.description]{task.description}"),- console=console,- ) as progress:- # Generate search queries- task = progress.add_task("[cyan]Generating search queries...", total=None)- console.print("\n[bold yellow]🤔 Analyzing your question to generate relevant search queries...[/bold yellow]")- query_agent_output = query_agent.run(QueryAgentInputSchema(instruction=user_message, num_queries=3))- progress.remove_task(task)-- console.print("\n[bold green]🔍 Generated search queries:[/bold green]")- for i, query in enumerate(query_agent_output.queries, 1):- console.print(f" {i}. [italic]{query}[/italic]")-- # Perform the search- task = progress.add_task("[cyan]Searching the web...", total=None)- console.print("\n[bold yellow]🌐 Searching across the web using SearXNG...[/bold yellow]")- searxng_search_tool = SearXNGSearchTool(SearXNGSearchToolConfig(base_url="http://localhost:8080/"))- search_results = searxng_search_tool.run(SearXNGSearchToolInputSchema(queries=query_agent_output.queries))- progress.remove_task(task)-- # Scrape content from search results- console.print("\n[bold green]📑 Found relevant web pages:[/bold green]")- for i, result in enumerate(search_results.results[:3], 1):- console.print(f" {i}. [link={result.url}]{result.title}[/link]")-- task = progress.add_task("[cyan]Scraping webpage content...", total=None)- console.print("\n[bold yellow]📥 Extracting content from web pages...[/bold yellow]")- webpage_scraper_tool = WebpageScraperTool()- results_for_context_provider = []-- for result in search_results.results[:3]:- scraped_content = webpage_scraper_tool.run(WebpageScraperToolInputSchema(url=result.url, include_links=True))- results_for_context_provider.append(ContentItem(content=scraped_content.content, url=result.url))- progress.remove_task(task)-- # Update the context provider with new content- console.print("\n[bold green]🔄 Updating research context with new information...[/bold green]")- scraped_content_context_provider.content_items = results_for_context_provider---def initialize_conversation() -> None:- initial_answer = QuestionAnsweringAgentOutputSchema(- answer=WELCOME_MESSAGE,- follow_up_questions=STARTER_QUESTIONS,- )- question_answering_agent.history.add_message("assistant", initial_answer)+ planner_agent.register_context_provider("current_date", date_provider)+ for agent in (extractor_agent, reflector_agent, writer_agent):+ agent.register_context_provider("current_date", date_provider)+ agent.register_context_provider("research_state", state_provider)-def display_welcome() -> None:- welcome_panel = Panel(- WELCOME_MESSAGE, title="[bold blue]Deep Research Chat[/bold blue]", border_style="blue", padding=(1, 2)- )- console.print("\n")- console.print(welcome_panel)- # Create a table for starter questions- table = Table(- show_header=True, header_style="bold cyan", box=box.ROUNDED, title="[bold]Example Questions to Get Started[/bold]"+def plan_research(state: ResearchState) -> None:+ """Run the planner once to fill in ``state.plan``."""+ console.rule("[bold cyan]1. Plan")+ result = planner_agent.run(+ PlannerInput(question=state.question, num_sub_topics=ResearchBudget.num_sub_topics)+ )+ state.agent_calls += 1++ for st in result.sub_topics:+ state.plan.append(SubTopic(name=st.name, initial_queries=list(st.initial_queries)))+ state.queries_seen.update(st.initial_queries)++ for i, st in enumerate(state.plan, 1):+ console.print(f" [bold]{i}. {st.name}[/bold]")+ for q in st.initial_queries:+ console.print(f" • [dim]{q}[/dim]")+++def search_and_scrape(+ queries: list[str],+ state: ResearchState,+ search: SearXNGSearchTool,+ scraper: WebpageScraperTool,+) -> list[tuple[str, str]]:+ """Run SearXNG on the given queries, scrape the top N new URLs, return ``[(source_id, content), …]``.++ Skips URLs we've already scraped in a previous iteration. Registers+ every new URL as a ``Source`` so downstream claims can cite by ID.+ """+ results = search.run(+ SearXNGSearchToolInputSchema(queries=queries, category="general"))- table.add_column("№", style="dim", width=4)- table.add_column("Question", style="green")- for i, question in enumerate(STARTER_QUESTIONS, 1):- table.add_row(str(i), question)+ scraped: list[tuple[str, str]] = []+ for r in results.results:+ if r.url in state.urls_seen:+ continue+ if len(scraped) >= ResearchBudget.scrape_top_n_per_iteration:+ break- console.print("\n")- console.print(table)- console.print("\n" + "─" * 80 + "\n")+ page = scraper.run(WebpageScraperToolInputSchema(url=r.url, include_links=False))+ if page.error or not page.content.strip():+ continue+ source = state.register_source(url=r.url, title=r.title or page.metadata.title)+ scraped.append((source.id, page.content))-def display_search_status(is_new_search: bool, reasoning: str) -> None:- if is_new_search:- panel = Panel(- f"[white]{reasoning}[/white]",- title="[bold yellow]Performing New Search[/bold yellow]",- border_style="yellow",- padding=(1, 2),- )- else:- panel = Panel(- f"[white]{reasoning}[/white]",- title="[bold green]Using Existing Context[/bold green]",- border_style="green",- padding=(1, 2),+ return scraped+++def extract_claims(sub_topic: SubTopic, scraped: list[tuple[str, str]], state: ResearchState) -> int:+ """Call the extractor once per scraped source, append claims to state, return claim count."""+ new_claim_count = 0+ for source_id, content in scraped:+ source = next(s for s in state.sources if s.id == source_id)+ result = extractor_agent.run(+ ExtractorInput(+ sub_topic=sub_topic.name,+ source_url=source.url,+ source_title=source.title,+ content=content[: ResearchBudget.max_extractor_content_chars],+ ))- console.print("\n")- console.print(panel)+ state.agent_calls += 1+ for claim in result.claims:+ state.learnings.append(+ Learning(text=claim, source_id=source_id, sub_topic=sub_topic.name)+ )+ new_claim_count += 1+ return new_claim_count-def display_answer(answer: str, follow_up_questions: list[str]) -> None:- # Display the main answer in a panel- answer_panel = Panel(Markdown(answer), title="[bold blue]Answer[/bold blue]", border_style="blue", padding=(1, 2))- console.print("\n")- console.print(answer_panel)- # Display follow-up questions if available- if follow_up_questions:- questions_table = Table(- show_header=True, header_style="bold cyan", box=box.ROUNDED, title="[bold]Follow-up Questions[/bold]"+def reflect(sub_topic: SubTopic, iteration: int, state: ResearchState) -> tuple[bool, list[str]]:+ """Ask the reflector whether this sub-topic has enough material. Returns (sufficient, next_queries)."""+ result = reflector_agent.run(+ ReflectorInput(+ sub_topic=sub_topic.name,+ iterations_so_far=iteration,+ max_iterations=ResearchBudget.max_depth_per_sub_topic,)- questions_table.add_column("№", style="dim", width=4)- questions_table.add_column("Question", style="green")+ )+ state.agent_calls += 1- for i, question in enumerate(follow_up_questions, 1):- questions_table.add_row(str(i), question)+ console.print(f" [italic]{result.reasoning}[/italic]")- console.print("\n")- console.print(questions_table)+ # Dedup: reflector might suggest a query we've already tried.+ fresh = [q for q in result.next_queries if q not in state.queries_seen]+ state.queries_seen.update(fresh)+ return result.sufficient, fresh-def chat_loop() -> None:- console.print("\n[bold magenta]🚀 Initializing Deep Research System...[/bold magenta]")+def research_sub_topic(+ sub_topic: SubTopic,+ state: ResearchState,+ search: SearXNGSearchTool,+ scraper: WebpageScraperTool,+) -> None:+ """Run the depth loop for a single sub-topic until sufficient or out of iterations."""+ console.rule(f"[bold cyan]Sub-topic: {sub_topic.name}")- # Initialize context providers- console.print("[dim]• Creating context providers...[/dim]")- scraped_content_context_provider = ScrapedContentContextProvider("Scraped Content")- current_date_context_provider = CurrentDateContextProvider("Current Date")+ queries = sub_topic.initial_queries+ for iteration in range(1, ResearchBudget.max_depth_per_sub_topic + 1):+ if state.agent_calls >= ResearchBudget.hard_call_cap:+ console.print("[red]Hit hard call cap — stopping this sub-topic.[/red]")+ return- # Register context providers- console.print("[dim]• Registering context providers with agents...[/dim]")- choice_agent.register_context_provider("current_date", current_date_context_provider)- question_answering_agent.register_context_provider("current_date", current_date_context_provider)- query_agent.register_context_provider("current_date", current_date_context_provider)+ console.print(f"\n [bold]Iteration {iteration}/{ResearchBudget.max_depth_per_sub_topic}[/bold]")+ console.print(f" queries: {queries}")- choice_agent.register_context_provider("scraped_content", scraped_content_context_provider)- question_answering_agent.register_context_provider("scraped_content", scraped_content_context_provider)- query_agent.register_context_provider("scraped_content", scraped_content_context_provider)+ scraped = search_and_scrape(queries, state, search, scraper)+ console.print(f" scraped {len(scraped)} new source(s)")- console.print("[dim]• Initializing conversation history...[/dim]")- initialize_conversation()+ if not scraped:+ # No new information to extract from — further iterations won't help either.+ sub_topic.sufficient = True+ return- console.print("[bold green]✨ System initialized successfully![/bold green]\n")- display_welcome()+ new_claims = extract_claims(sub_topic, scraped, state)+ console.print(f" extracted {new_claims} claim(s)")- while True:- user_message = console.input("\n[bold blue]Your question:[/bold blue] ").strip()+ sufficient, next_queries = reflect(sub_topic, iteration, state)+ if sufficient or iteration == ResearchBudget.max_depth_per_sub_topic or not next_queries:+ sub_topic.sufficient = sufficient+ return- if user_message.lower() in ["/exit", "/quit"]:- console.print("\n[bold]👋 Goodbye! Thanks for using Deep Research.[/bold]")- break+ queries = next_queries- console.print("\n[bold yellow]🤖 Processing your question...[/bold yellow]")-- # Determine if we need a new search- console.print("[dim]• Evaluating if new research is needed...[/dim]")- choice_agent_output = choice_agent.run(- ChoiceAgentInputSchema(- user_message=user_message,- decision_type=(- "Should we perform a new web search? TRUE if we need new or updated information, FALSE if existing "- "context is sufficient. Consider: 1) Is the context empty? 2) Is the existing information relevant? "- "3) Is the information recent enough?"- ),- )- )- # Display search status with new formatting- display_search_status(choice_agent_output.decision, choice_agent_output.reasoning)+def write_report(state: ResearchState) -> tuple[str, str]:+ """Draft the report, then run a cheap verification pass over it. Returns (headline, report)."""+ console.rule("[bold cyan]3. Write")++ draft = writer_agent.run(+ WriterInput(question=state.question, mode="draft", draft="")+ )+ state.agent_calls += 1+ console.print(" [dim]draft written, verifying citations…[/dim]")++ verified = writer_agent.run(+ WriterInput(question=state.question, mode="verify", draft=draft.report)+ )+ state.agent_calls += 1+ return verified.headline, verified.report+++def run(question: str) -> None:+ """Top-level pipeline. Reads like an outline of what 'deep research' means in this project."""+ console.print(Panel.fit(+ f"[bold]Deep Research[/bold]\n{question}",+ border_style="blue",+ ))- if choice_agent_output.decision:- perform_search_and_update_context(user_message, scraped_content_context_provider)+ state = ResearchState(question=question)+ wire_context_providers(state)- # Get and display the answer with new formatting- console.print("\n[bold yellow]🎯 Generating comprehensive answer...[/bold yellow]")- question_answering_agent_output = question_answering_agent.run(- QuestionAnsweringAgentInputSchema(question=user_message)+ search = SearXNGSearchTool(+ SearXNGSearchToolConfig(+ base_url=ChatConfig.searxng_base_url,+ max_results=ResearchBudget.search_results_per_query,)+ )+ scraper = WebpageScraperTool()++ plan_research(state)- display_answer(question_answering_agent_output.answer, question_answering_agent_output.follow_up_questions)+ console.rule("[bold cyan]2. Research")+ for sub_topic in state.plan:+ research_sub_topic(sub_topic, state, search, scraper)++ headline, report = write_report(state)++ console.rule("[bold green]Report")+ console.print(Panel(f"[bold]{headline}[/bold]", border_style="green"))+ console.print(Markdown(report))++ console.print(+ f"\n[dim]Stats: {state.agent_calls} agent calls, {len(state.sources)} sources, "+ f"{len(state.learnings)} learnings.[/dim]"+ )if __name__ == "__main__":- chat_loop()+ args = sys.argv[1:]+ if not args:+ console.print("[red]Usage:[/red] python -m deep_research \"your question\"")+ sys.exit(1)+ run(" ".join(args))diff --git a/atomic-examples/deep-research/deep_research/state.py b/atomic-examples/deep-research/deep_research/state.pynew file mode 100644index 0000000..01f4f84--- /dev/null+++ b/atomic-examples/deep-research/deep_research/state.py@@ -0,0 +1,82 @@+"""+Shared state for the deep-research pipeline.++Every agent in the pipeline reads from — and contributes to — a single+`ResearchState` object. Passing it explicitly through function arguments+(instead of hiding it in globals or on an agent) makes the data flow+inspectable and each pipeline stage easy to reason about in isolation.++The state holds three kinds of data:++- The plan: durable sub-topics the planner produced.+- Accumulated findings: sources we've seen and learnings extracted from them.+- Deduplication sets: queries and URLs we've already touched, so the+ query agent and search loop don't re-do work.++Source IDs (``S1``, ``S2``, ...) are assigned when a source is first+registered and are used throughout the pipeline as citation anchors.+"""++from dataclasses import dataclass, field+from datetime import datetime+++@dataclass+class Source:+ """A web page we've scraped. ``id`` is referenced by learnings and the final report."""++ id: str+ url: str+ title: str+++@dataclass+class Learning:+ """One atomic claim extracted from a single source."""++ text: str+ source_id: str # must match some Source.id+ sub_topic: str # the sub-topic this was gathered under+++@dataclass+class SubTopic:+ """One durable branch of the research plan. Queries iterate; sub-topics don't."""++ name: str+ initial_queries: list[str]+ sufficient: bool = False # set by the reflector when further research is unnecessary+++@dataclass+class ResearchState:+ question: str+ plan: list[SubTopic] = field(default_factory=list)+ learnings: list[Learning] = field(default_factory=list)+ sources: list[Source] = field(default_factory=list)++ # Dedup sets — keep the search loop and query agent from repeating themselves.+ queries_seen: set[str] = field(default_factory=set)+ urls_seen: set[str] = field(default_factory=set)++ # Budget counter — see config.HARD_CALL_CAP.+ agent_calls: int = 0++ started_at: datetime = field(default_factory=datetime.utcnow)++ def learnings_for(self, sub_topic: str) -> list[Learning]:+ return [l for l in self.learnings if l.sub_topic == sub_topic]++ def register_source(self, url: str, title: str) -> Source:+ """Register a source if new, return the (new or existing) record.++ IDs are stable within a run — once a URL has an ID, it keeps it even+ if the source is looked up again later.+ """+ for s in self.sources:+ if s.url == url:+ return s+ source = Source(id=f"S{len(self.sources) + 1}", url=url, title=title)+ self.sources.append(source)+ self.urls_seen.add(url)+ return source