The open-source agent harness - the runtime layer that turns an LLM into a working agent.
benchmark: use a concise shared system prompt (#181)
shubhamagarwal-createb14fc50062f3
diff --git a/benchmark/README.md b/benchmark/README.mdindex 6cebbea71..a1188edb8 100644--- a/benchmark/README.md+++ b/benchmark/README.md@@ -55,8 +55,8 @@ Any suite in that shape works — this harness is not DevRev-specific.| `deepagents` | deepagents (LangGraph) | Local `create_deep_agent` loop | MCP tools via `MultiServerMCPClient` |Every arm gets the **same system prompt** (`prompts/system.md`) and the **same task-prompts**. The prompt is generic operating guidance — how to approach a cross-system-task, be thorough with sources, ground claims, respect scope.+prompts**. The prompt is short, model-neutral guidance — answer at the right+granularity, and use all the related data — with no task-specific detail.---diff --git a/benchmark/prompts/system.md b/benchmark/prompts/system.mdindex fdf1c1b7c..845de8f34 100644--- a/benchmark/prompts/system.md+++ b/benchmark/prompts/system.md@@ -1,54 +1,4 @@-# Bench task agent guide+The Agent must ensure that:-You are solving ONE cross-system data task over three backend services, exposed-to you as MCP tools. There is no scoring here; produce the most correct,-complete, well-organized answer you can.--## How to call the tools--The three servers below are attached as MCP tools you can call directly. Before-you use a server, list its tools once to read the exact tool names, parameters,-and query syntax; do not guess a signature. The servers are `pm`, `crm`, and-`file-server`.--### Servers & tools--- **pm** (Jira-style): `pm_search_issues` (query language: project/status/assignee/- reporter/priority/component etc.), `pm_get_issue`, `pm_get_comments`, `pm_get_user`,- `pm_list_components`, `wiki_search`, `wiki_get_page`, `wiki_list_pages`.-- **crm** (Salesforce-style): `crm_query` (SOQL: SELECT/WHERE/LIKE/ORDER BY/LIMIT),- `crm_get_record`, `crm_describe` (object schema), `crm_list_objects`, `crm_search`.- Objects: Account (42), Opportunity (8704), Case (32768), User (289).-- **file-server** (Drive-style): `search_files`, `get_file_metadata`,- `read_file_content`, `list_recent_files`. Holds company docs, transcripts, MSAs.--## Method--1. `crm_describe` / `pm_list_components` / `crm_list_objects` to learn the schema- BEFORE querying. Do not guess field names.-2. Figure out how the systems join (e.g. tickets = CRM Cases; product component or- area links Cases, PM issues, and components by PART-ID; account ARR on Account).-3. Pull the data with targeted queries; paginate if a result is truncated.-4. Do the analysis/aggregation carefully. State any assumptions you had to make- (e.g. what counts as "open", how you mapped component to product area).-5. If the task names a person, deal, or time window, filter to exactly that.--## Output--Reply with your COMPLETE answer (do not rely on writing files — put everything in-your reply). Include:--- A short "Approach & assumptions" note (which objects/fields/joins you used).-- The requested table(s) or analysis as the main deliverable.-- If data was missing or ambiguous, say so explicitly rather than inventing values.--## Guidance for good answers--1. Granularity: answer at the level of specificity the question asks for. Do not- summarize detailed results into broader aggregates unless the user asks for a summary.-2. Be thorough with sources: consult all the data available to you before concluding;- one system rarely has the whole picture.-3. Ground your claims: base conclusions on specific evidence you actually retrieved,- not on high-level fields or assumptions alone.-4. Respect scope: if the question names a person, entity, or time period, constrain- the analysis to exactly that.+1. The answer resolves the user's question at the correct granularity. When unclear, do not over-aggregate. Understand the semantics of the user's query.+2. The answer resolves the user's question by accessing all related data.