SWE-agent takes a GitHub issue and tries to automatically fix it, using your LM of choice. It can also be employed for offensive cybersecurity or competitive coding challenges. [NeurIPS 2024]
Restored review & retry
Kilian Lieretd7f47114be10
diff --git a/config/function_calling_anthropic_replace_late_repro_review_submit_retry.yaml b/config/function_calling_anthropic_replace_late_repro_review_submit_retry.yamlnew file mode 100644index 00000000..a125ee4d--- /dev/null+++ b/config/function_calling_anthropic_replace_late_repro_review_submit_retry.yaml@@ -0,0 +1,342 @@+agent:+ templates:+ system_template: |-+ SETTING: You are an autonomous programmer, and you're working directly in the command line with a special interface.++ The special interface consists of a file editor that shows you {{WINDOW}} lines of a file at a time.+ In addition to typical bash commands, you can also use specific commands to help you navigate and edit files.+ To call a command, you need to invoke it with a function call/tool call.++ Please note that THE EDIT COMMAND REQUIRES PROPER INDENTATION.++ For example, if you are looking at this file:++ def fct():+ print("Hello world")++ and you want to edit the file to read:++ def fct():+ print("Hello")+ print("world")++ you search string should be `Hello world` and your replace string should be `"Hello"\n print("world")`+ (note the extra spaces before the print statement!).++ You could also get the same result by search for ` print("Hello world")` and replace with ` print("Hello")\n print("world")`.++ RESPONSE FORMAT:+ Your shell prompt is formatted as follows:+ (Open file: <path>)+ (Current directory: <cwd>)+ bash-$++ First, you should _always_ include a general thought about what you're going to do next.+ Then, for every response, you must include exactly _ONE_ tool call/function call.++ Remember, you should always include a _SINGLE_ tool call/function call and then wait for a response from the shell before continuing with more discussion and commands. Everything you include in the DISCUSSION section will be saved for future reference.+ If you'd like to issue two commands at once, PLEASE DO NOT DO THAT! Please instead first submit just the first tool call, and then after receiving a response you'll be able to issue the second .+ Note that the environment does NOT support interactive session commands (e.g. python, vim), so please do not invoke them.+ instance_template: |-+ We're currently solving the following issue within our repository. Here's the issue text:+ ISSUE:+ {{problem_statement}}++ INSTRUCTIONS:+ Now, you're going to solve this issue on your own. Your terminal session has started and you're in the repository's root directory. You can use any bash commands or the special interface to help you. Edit all the files you need to and run any checks or tests that you want.+ Remember, YOU SHOULD ALWAYS INCLUDE EXACTLY ONE TOOL CALL/FUNCTION CALL PER RESPONSE.+ When you're satisfied with all of the changes you've made, you can submit your changes to the code base by simply running the submit command.+ Note however that you cannot use any interactive session commands (e.g. python, vim) in this environment, but you can write scripts and run them. E.g. you can write a python script and then run it with the python command.++ NOTE ABOUT THE EDIT COMMAND: Indentation really matters! When editing a file, make sure to insert appropriate indentation before each line!++ GENERAL IMPORTANT TIPS:++ 1. If you run a command and it doesn't work, try running a different command. A command that did not work once will not work the second time unless you modify it!++ 2. If you open a file and need to get to an area around a specific line that is not in the first 100 lines, say line 583, don't just use the scroll_down command multiple times. Instead, use the goto 583 command. It's much quicker.++ 3. If the bug reproduction script requires inputting/reading a specific file, such as buggy-input.png, and you'd like to understand how to input that file, conduct a search in the existing repo code, to see whether someone else has already done that. Do this by running the command: find_file "buggy-input.png" If that doesn't work, use the linux 'find' command.++ 4. Always make sure to look at the currently open file and the current working directory (which appears right after the currently open file). The currently open file might be in a different directory than the working directory! Note that some commands, such as 'create', open files, so they might change the current open file.++ 5. When editing files, it is easy to accidentally to write code with incorrect indentation or make other mistakes. Always check the code after you issue an edit to make sure that it reflects what you wanted to accomplish. If it didn't, issue another command to fix it.++ 6. When editing files, first explain the code you want to edit and why it is causing the problem. Then explain the edit you want to make and how it fixes the problem. Explain how the edit does not break existing functionality.++ 7. The environment should be set up so that you do not need to install any packages with `pip`, `conda`, or any other way.+ If the environment is not set up correctly, try to fix the issue without executing python code or running any tests that require the package installed.+ The only exception is installing pytest, or similar utilities for testing: You may use `pip` to install it.++ STRATEGY:++ 1. Locate relevant files using the find and search commands, then read the code related to the issue.++ 2. Try to replicate the bug that the issues discusses.+ If the issue includes code for reproducing the bug, we recommend that you re-implement that in your environment, and run it to make sure you can reproduce the bug.+ Then start trying to fix it.++ If the bug reproduction script does not print anything when it successfully runs, we recommend adding a print("Script completed successfully, no errors.") command at the end of the file,+ so that you can be sure that the script indeed ran fine all the way through.++ 3. Use the `edit` command to perform edits on the files that cause the issue.++ 4. When you think you've fixed the bug, re-run the bug reproduction script to make sure that the bug has indeed been fixed.++ 5. Create additional tests to verify the fix in a style similar to the existing reproduction script. In particular, make sure to test edge cases.+ If you find any issues, go back to the file you edited and perform further edits.+ Put these tests in the same file as the bug reproduction script.++ 6. Many packages have tests. Please also take a look at any relevant tests related to the problem you are fixing or the+ code you are editing. Do not try to run them yet, just look at them to understand how the code is tested and what+ functionality is tested. Do NOT modify the existing tests!++ <IMPORTANT>+ * Never modify existing tests! Only add new tests to a reproduce.py script or similar (NOT to the existing test directory).+ * Make sure to remove these tests before submitting your changes.+ </IMPORTANT>++ (Open file: {{open_file}})+ (Current directory: {{working_dir}})+ bash-$+ next_step_template: |-+ {{observation}}+ (Open file: {{open_file}})+ (Current directory: {{working_dir}})+ bash-$+ next_step_no_output_template: |-+ Your command ran successfully and did not produce any output.+ (Open file: {{open_file}})+ (Current directory: {{working_dir}})+ bash-$+ demonstrations:+ - trajectories/demonstrations/str_replace_anthropic_demo.yaml+ put_demos_in_history: true+ tools:+ submit_command: "submit -f"+ execution_timeout: 120+ env_variables:+ WINDOW: 100+ OVERLAP: 2+ SUBMIT_REVIEW_MESSAGE: |-+ Thank you for your work on this issue.++ As a refresher, this is the issue we're solving:++ <issue_text>+ {{problem_statement}}+ </issue_text>++ Here are your changes:++ <diff>+ {{diff}}+ </diff>++ Before we submit your changes, here are some final instructions to review:++ 1. Does the diff <diff> look correct or could it introduce any problems?+ 2. Does the diff <diff> solve the problem explained in the issue <issue_text>?+ 3. Were you able to reproduce and then fix the bug? If you had trouble with the reproduction, are there any other+ ways you could double-check that the bug is fixed?+ 4. Is there any existing functionality that could be broken by your changes?+ 5. Did you test or think of any edge cases?+ 6. Many packages come with tests. Have you tried to run them? You might have to install pytest (`pip install pytest`) to run them.+ first. VERY IMPORTANT THOUGH: First locate all relevant tests and only run these (`pytest -k ...`). DO NOT RUN ALL TESTS!+ Running all tests will take a long time and probably not work.++ Please write a DETAILED response to the above questions.+ You can also run additional checks and tests if you want.+ If you find anything to be problematic, please fix it.++ Once you're done, run the submit command to submit your changes.++ <IMPORTANT>+ 1. Please be verbose and think things through step-by-step.+ 2. Do not run pytest without specifying the tests you want to run. It will take too long.+ 3. Never modify existing tests! Only add new tests to a reproduce.py script or similar (NOT to the existing test directory).+ Make sure to remove these tests before submitting your changes.+ </IMPORTANT>+ bundles:+ - path: tools/registry+ - path: tools/edit_anthropic+ - path: tools/review_on_submit+ enable_bash_tool: true+ parse_function:+ type: function_calling+ history_processors:+ - type: last_n_observations+ n: 5+ review_loop:+ review_loop_classname: ReviewLoop+ reviewer_classname: Reviewer+ reviewer_config:+ system_template: |+ You are checking whether a problem in a code base has been resolved.++ You will be presented with++ INPUTS++ **INPUT 1**: Problem statement: A GitHub issue that describes an issue+ **INPUT 2**: Code: Relevant lines from a file++ TASK++ Determine if the problem persists.++ If you believe the problem has been resolved, answer "success". If you believe the problem persists, answer "fail".++ THE LAST LINE OF YOUR RESPONSE MUST BE EITHER "SUCCESS" OR "FAIL".+ instance_template: |+ INPUT 1: Problem statement++ {{problem_statement}}+++ INPUT 2: Code++ ```+ {{edited_files70}}+ ```+ binary_reviewer_classname: BinaryReviewer+ binary_reviewer_config:+ system_template: |+ You are comparing two attempts to fix a problem in a code base to determine which one resolves the problem better.++ You will be presented with++ 1. INPUTS++ **INPUT 1: Problem statement**: A GitHub issue that describes an issue+ **INPUT 2: Investigation 1**: History of how the first developer tried to fix the problem+ **INPUT 3: Investigation 2**: History of how the second developer tried to fix the problem++ 2. INPUT FORMATS++ 2.1 The investigation format++ The investigation is structured as a series of actions ("Step {i} action: "), where each action is a thought followed by a command in triple-backticks.+ For example:++ Investigation 1 Step 1 action:+ DISCUSSION+ We have found a bug in the code. Let's try to reproduce it.+ ```+ create reproduce.py+ ```++ Investigation 1 Step 1 output:+ reproduce.py created.++ After the command, there is a description of the output of the command ("Step {i} output: ").+ The most important actions are++ - `open {file}`: This action opens the file {file}. The output of the command shows part of the content of the file.+ - `edit {from}:{to}`, followed by the content to replace lines {from} to {to}. This action always affects the currently+ opened file. The output of the command is the content of the file after the edit.+ - `python {file}`: This action executes the Python script {file}. The output of the command is the output of the script.+ - `submit`: This action submits the code for review. The output of the command is the result of the submission.+ The format of the output is a patch file/git diff (described in the next section)++ 2.2 The patch/git diff format++ The patch/git diff format is formatted as follows:++ 2.1.1 File Paths: Each diff begins with two lines indicating the file paths of the old and new files.++ ```+ --- a/path/to/file+ +++ b/path/to/file+ ```++ 2.1.2 Hunks: The changes within the files are divided into “hunks.” Each hunk shows the line numbers affected and the actual changes. The hunk header looks like this:++ ```+ @@ -start,line_count +start,line_count @@+ ```+ The -start,line_count refers to the starting line number and the number of lines in the old file.+ The +start,line_count refers to the starting line number and the number of lines in the new file.++ 2.1.3 Changes:++ Removed Lines: Lines removed from the file are prefixed with a -.+ Added Lines: Lines added to the file are prefixed with a +.+ Unchanged Lines: Lines that remain unchanged around the changes are shown for context and are prefixed with a space.++ 3. TASK++ The last line of your reply MUST be "first" or "second", depending on which patch resolves the problem.++ Compare the two investigations (INPUT 2 and INPUT 3) and determine which one is better based on how they address+ the following points:++ - The history should attempt to reproduce the problem if it was a bug, for example by creating a `reproduce.py` file.+ Make sure that the problem reproduced is the same as the problem described in INPUT 1.+ - Fixing the isuee: If the problem was reproduced, the edits should fix the problem without modifying+ the reproduction script.+ - Any attempt to fix the issue should not introduce new issues or remove important checks that might serve other purposes.+ For example, when trying to fix a case where an exception is raised, the fix should not remove all of the exception handling,+ but understand why it was raised in this specific scenario and fix the root cause. If the logic is significantly simplified+ or shortened, you should be extra suspicious.+ - Generally, the less steps and the fewer API calls are needed to fix the problem, the better the solution is.+ Investigation 1 used {model_stats_api_calls1} API calls to fix the problem. Investigation 2 used {model_stats_api_calls2} API calls to fix the problem.+ - Make sure that the explanation of the actions is clear and concise and that they make sense in the context of the previous+ steps, actions, and outputs. You are actively looking for inconsistencies.+ - The explanation of the actions must match the actual code changes.++ Based on these questions, determine which patch is the better solution.++ If INPUT 2 is the better solution, answer "first"+ If INPUT 3 is the better solution, answer "second"++ If you cannot determine which solution is better, answer "first".++ YOU MUST DECIDE. THE LAST LINE OF YOUR REPLY MUST BE "first" OR "second".++ instance_template: |+ ==========================+ INPUT 1: Problem statement+ ==========================++ {{problem_statement}}++ ========================+ INPUT 2: Investigation 1+ ========================++ Investigation 1 used {{model_stats_api_calls1}} API calls to fix the problem. The investigation is as follows:++ {{traj1}}++ This is the final patch of investigation 1:++ ```+ {submission1}+ ```++ ========================+ INPUT 3: Investigation 2+ ========================++ Investigation 2 used {{model_stats_api_calls2}} API calls to fix the problem. The investigation is as follows:++ {{traj2}}++ This is the final patch of investigation 2:++ ```+ {{submission2}}+ ```++ traj_only_show_last_n_output: 3+ traj_item_template: |+ Investigation {{i_traj}} Step {{i_step}}:+ {{response}}++ Investigation {{i_traj}} Step {{i_step}} output:+ {{observation}}+ max_samples: 5+ min_draws: 2+ max_accepted_draws: 0+ max_n_consec_exit_cost: 3\ No newline at end of filediff --git a/sweagent/agent/agents.py b/sweagent/agent/agents.pyindex 53047a28..d28cedb7 100644--- a/sweagent/agent/agents.py+++ b/sweagent/agent/agents.py@@ -27,13 +27,14 @@ from sweagent.agent.models import (get_model,)from sweagent.agent.problem_statement import ProblemStatement, ProblemStatementConfig+from sweagent.agent.reviewer import ReviewLoopConfig, get_review_loop_from_configfrom sweagent.environment.swe_env import SWEEnvfrom sweagent.exceptions import ContextWindowExceededError, CostLimitExceededError, FormatErrorfrom sweagent.tools.parsing import (ThoughtActionParser,)from sweagent.tools.tools import ToolConfig, ToolHandler-from sweagent.types import AgentInfo, AgentRunResult, History, StepOutput, Trajectory, TrajectoryStep+from sweagent.types import AgentInfo, AgentRunResult, History, ReviewSubmission, StepOutput, Trajectory, TrajectoryStepfrom sweagent.utils.config import _convert_paths_to_abspath, _strip_abspath_from_dictfrom sweagent.utils.jinja_warnings import _warn_probably_wrong_jinja_syntaxfrom sweagent.utils.log import get_logger@@ -115,6 +116,7 @@ class AgentConfig(BaseModel):"""Maximum number of times to requery the model after an error, such as aformatting error, a blocked action, or a bash syntax error."""+ review_loop: ReviewLoopConfig | None = None# pydantic configmodel_config = ConfigDict(extra="forbid")@@ -146,6 +148,7 @@ class Agent:name: str = "main",_catch_errors: bool = True,_always_require_zero_exit_code: bool = False,+ review_loop_config: ReviewLoopConfig | None = None,):"""The agent handles the behaviour of the model and how it interacts with the environment.@@ -160,7 +163,7 @@ class Agent:self.history_processors = history_processorsself.max_requeries = max_requeriesself.logger = get_logger("swea-agent", emoji="🤠")-+ self.review_loop_config = review_loop_config# Set in run methodself._env: SWEEnv | None = Noneself._problem_statement: ProblemStatement | ProblemStatementConfig | None = None@@ -195,6 +198,7 @@ class Agent:history_processors=config.history_processors,model=model,max_requeries=config.max_requeries,+ review_loop_config=config.review_loop,)def add_hook(self, hook: AbstractAgentHook) -> None:@@ -862,13 +866,6 @@ class Agent:self._chook.on_step_done(step=step_output, info=self.info)return step_output- def _should_attempt_retry(self, step_output: StepOutput) -> bool:- if step_output.exit_status in ["exit_cost"]:- return False- if self.model.config.name in ["human", "human_thought"] and step_output.exit_status in ["exit_command"]:- return False- return True-def run(self,env: SWEEnv,@@ -888,15 +885,19 @@ class Agent:# Run action/observation loopself._chook.on_run_start()+ review_loop = get_review_loop_from_config(self.review_loop_config, problem_statement, self.model)step_output = StepOutput()while not step_output.done:step_output = self.step()self.save_trajectory()- if step_output.done and not step_output.submission and self._should_attempt_retry(step_output):- assert self._env is not None- self.logger.warning("No submission found in the trajectory. Going for a new attempt.")- self.setup_attempt(hard_reset=True)- step_output.done = False+ if step_output.done:+ if review_loop is not None:+ review_loop.on_submit(ReviewSubmission(trajectory=self.trajectory, info=self.info))+ self.info["review"] = review_loop.reviews[-1].to_dict()+ if review_loop.retry():+ assert self._env is not None+ self.setup_attempt(hard_reset=True)+ step_output.done = Falseself._chook.on_run_done(trajectory=self.trajectory, info=self.info)self.logger.info("Trajectory saved to %s", self.traj_path)diff --git a/sweagent/agent/reviewer.py b/sweagent/agent/reviewer.pynew file mode 100644index 00000000..76544ed2--- /dev/null+++ b/sweagent/agent/reviewer.py@@ -0,0 +1,549 @@+"""The reviewer implements a retry loop for the agent to retry+solving the issue and to select the best solution.+"""++from __future__ import annotations++import copy+from abc import ABC, abstractmethod+from typing import Any, Literal++from jinja2 import Template+from pydantic import BaseModel++from sweagent.agent.models import AbstractModel, InstanceStats+from sweagent.agent.problem_statement import ProblemStatement+from sweagent.types import BinaryReviewerResult, History, ReviewerResult, ReviewSubmission, Trajectory, TrajectoryStep+from sweagent.utils.log import get_logger++logger = get_logger("reviewer")++# --- INTERFACES ---+++class AbstractReviewer(ABC):+ """The reviewer checks a single solution and tries to predict+ if it successfully solves the issue.+ """++ @abstractmethod+ def review(self, instance: ProblemStatement, submission: ReviewSubmission) -> ReviewerResult:+ """Returns True if the submission is believed to be correct"""+++class AbstractBinaryReviewer(ABC):+ """The binary reviewer checks two solutions and tries to predict+ which one is better.+ """++ @abstractmethod+ def compare_submissions(+ self,+ instance: ProblemStatement,+ sub1: ReviewSubmission,+ sub2: ReviewSubmission,+ rev1: ReviewerResult,+ rev2: ReviewerResult,+ ) -> BinaryReviewerResult:+ """Returns 0 if sub1 is better, 1 if sub2 is better"""+++class AbstractGraveToCradle(ABC):+ """Forward messages from past attempts to the next one"""++ @abstractmethod+ def get_forwarded_vars(+ self,+ submissions: list[ReviewSubmission],+ reviews: list[ReviewerResult],+ breviews: list[tuple[int, int, BinaryReviewerResult]],+ ) -> dict[str, Any]:+ """Get the variables that should be forwarded to the next iteration.++ Note: Must return a dictionary with the correct keys even when called+ with empty lists. This is because else we cannot use the variables in the template+ when we call for the first attempt.++ Returns:+ A dictionary of variables that should be forwarded to the next iteration.+ """+++class AbstractReviewLoop(ABC):+ """The review loop controls how often the agent tries to solve+ the issue and how it selects the best solution.+ """++ @abstractmethod+ def retry(self) -> bool:+ """Returns True if the agent should retry solving the issue"""++ @abstractmethod+ def on_submit(self, submission: ReviewSubmission) -> None:+ """Called when the agent submits a solution"""++ @abstractmethod+ def get_best(self) -> int:+ """Returns the best solution"""++ @property+ @abstractmethod+ def model_stats(self) -> InstanceStats:+ """Returns the API stats of the model (if any)"""++ @property+ @abstractmethod+ def reviews(self) -> list[ReviewerResult]: ...++ @property+ @abstractmethod+ def comparisons(self) -> list[tuple[int, int, BinaryReviewerResult]]:+ """Get information about comparisons++ Returns:+ A list of tuples, where each tuple contains the indices of the+ compared submissions and the result of the comparison.+ """++ def get_forwarded_vars(self) -> dict[str, Any]:+ """Get the variables that should be forwarded to the next iteration.++ Returns:+ A dictionary of variables that should be forwarded to the next iteration.+ """+ return {}+++# --- CONFIGS ---+++class ReviewerConfig(BaseModel):+ """The configuration for the reviewer"""++ system_template: str+ instance_template: str+ #: If a submission autosubmits because of total cost or a similar exit status,+ #: it will be desk rejected+ reject_exit_status: bool = True+ #: Filter the following actions from the trajectory+ traj_filter: list[str] = []+ #: Filter outputs from the following actions from the trajectory+ traj_output_filter: list[str] = []+ #: Format of the trajectory item+ traj_item_template: str = "Model: {response}\n\nObservation: {observation}"+ filter_failed_edits: bool = False+++class BinaryReviewerConfig(BaseModel):+ """The configuration for the binary reviewer"""++ system_template: str+ instance_template: str+ #: Filter the following actions from the trajectory+ traj_filter: list[str] = []+ #: Format of the trajectory item+ traj_item_template: str = "Model: {response}\n\nObservation: {observation}"+ traj_output_filter: list[str] = []+ traj_filter_failed_edits: bool = False+ traj_only_show_last_n_output: int = 0+++class GTCConfig(BaseModel):+ """The configuration for the GraveToCradle"""+++class ReviewLoopConfig(BaseModel):+ """The configuration for the review loop"""++ review_loop_classname: str+ reviewer_config: ReviewerConfig | None = None+ binary_reviewer_config: BinaryReviewerConfig | None = None+ gtc_config: GTCConfig | None = None+ reviewer_classname: str | None = None+ binary_reviewer_classname: str | None = None+ gtc_classname: str = ""+ #: Maximal number of attempts+ max_samples: int = 2+ #: Minimal number of attempts.+ min_draws: int = 1+ #: For use together with a `min_draws > 1`. Even if we have+ #: not reached the `min_draws`, we will stop if we have accepted+ #: `max_accepted_draws` submissions.+ max_accepted_draws: int = 0+ #: If set > 0 and there are more than this number of consecutive attempts+ #: with an 'exit cost' exit stats, the review loop will quit.+ max_n_consec_exit_cost: int = 0++ def validate(self):+ """Checks config. Raises `ValueError` in case of misconfiguration"""+ if self.min_draws < 1:+ msg = "`min_draws` must be at least 1"+ raise ValueError(msg)+ if self.max_samples < 1:+ msg = "`max_samples` must be at least 1"+ raise ValueError(msg)+ if self.max_samples < self.min_draws:+ msg = "`max_samples` must be greater than or equal to `min_draws`"+ raise ValueError(msg)+ if self.max_accepted_draws > self.min_draws:+ msg = "`max_accepted_draws` must be less than or equal to `min_draws`, else it has no effect"+ raise ValueError(msg)++ def __post_init__(self):+ self.validate()+++# --- IMPLEMENTATIONS ---+++class Reviewer(AbstractReviewer):+ LOG_PREFIX = "🧑⚖️ Reviewer: "++ def __init__(self, config: ReviewerConfig, model):+ self._config = config+ self._model = model+ self._traj_formatter = TrajectoryFormatter(+ traj_filter=config.traj_filter,+ traj_item_template=config.traj_item_template,+ traj_output_filter=config.traj_output_filter,+ filter_failed_edits=config.filter_failed_edits,+ )++ def format_messages(self, instance: ProblemStatement, submission: ReviewSubmission):+ system_message = self._config.system_template+ logger.debug(f"{self.LOG_PREFIX}MODEL INPUT (system)\n{system_message}")+ ps_format_dict = {+ "problem_statement": instance.get_problem_statement(),+ **instance.get_extra_fields(),+ }+ user_message = Template(self._config.instance_template).render(+ **ps_format_dict,+ **submission.to_format_dict(),+ traj=self._traj_formatter.format_trajectory(submission.trajectory),+ )+ logger.debug(f"{self.LOG_PREFIX}MODEL INPUT (user)\n{user_message}")+ return [+ {"role": "system", "content": system_message},+ {"role": "user", "content": user_message},+ ]++ def interpret(self, response: str) -> bool:+ last_line = response.strip().split("\n")[-1].strip()+ if "success" in last_line.lower():+ return True+ elif "fail" in last_line.lower():+ return False+ logger.warning("Could not interpret response: %s, will reject submission.", response)+ return False++ def review(self, instance: ProblemStatement, submission: ReviewSubmission) -> ReviewerResult:+ exit_status = submission.info.get("exit_status")+ messages = []+ if not exit_status:+ answer = "No exit status in submission, will reject."+ accept = False+ elif self._config.reject_exit_status and exit_status.strip() != "submitted":+ answer = f"Submission desk-rejected because of exit status {exit_status!r}."+ accept = False+ else:+ messages = self.format_messages(instance, submission)+ answer = self._model.query(messages)["message"]+ accept = self.interpret(answer)+ accept_emoji = "✅" if accept else "❌"+ logger.info(f"{self.LOG_PREFIX}{accept_emoji}\n{answer}")+ return ReviewerResult(accept, answer, messages=messages)+++# todo: Couldn't I just replace the whole thing with Jinja templates?+class TrajectoryFormatter:+ def __init__(+ self,+ *,+ traj_filter: list[str] | None = None,+ traj_output_filter: list[str] | None = None,+ traj_item_template: str = "Model: {response}\n\nObservation: {observation}",+ filter_failed_edits: bool = False,+ only_show_last_n_output: int = 0,+ ):+ """Formats trajectories for the use in prompts"""+ self._traj_filter = traj_filter or []+ self._traj_output_filter = traj_output_filter or []+ self._traj_item_template = traj_item_template+ self._filter_failed_edits = filter_failed_edits+ self._only_show_last_n_output = only_show_last_n_output++ def _include_step(self, item: TrajectoryStep) -> bool:+ action = item["action"].strip()+ for f in self._traj_filter:+ if action.startswith(f):+ return False+ if self._filter_failed_edits and "Your proposed edit has introduced new syntax error(s)" in item["observation"]:+ return False+ return True++ def _include_step_output(self, item: TrajectoryStep, i_step: int, n_steps: int) -> bool:+ if self._only_show_last_n_output > 0 and i_step < n_steps - self._only_show_last_n_output:+ return False+ action = item["action"].strip()+ for f in self._traj_output_filter:+ if action.startswith(f):+ return False+ return True++ def _format_trajectory_step(self, step: TrajectoryStep, i_step: int, *, n_steps: int, i_traj: int = 1) -> str:+ step = copy.deepcopy(step)+ if not self._include_step_output(step, i_step, n_steps=n_steps):+ step["observation"] = "[Output omitted]"+ return Template(self._traj_item_template).render(+ **step,+ i_step=i_step,+ i_traj=i_traj,+ )++ def format_trajectory(self, trajectory: Trajectory, i_traj: int = 1) -> str:+ traj_messages = [step for step in trajectory if self._include_step(step)]+ return "\n\n".join(+ [+ self._format_trajectory_step(step, i_step, i_traj=i_traj, n_steps=len(traj_messages))+ for i_step, step in enumerate(traj_messages)+ ]+ )+++class BinaryReviewer(AbstractBinaryReviewer):+ LOG_PREFIX = "⚖️ Binary Reviewer: "++ def __init__(self, config: BinaryReviewerConfig, model: AbstractModel):+ self._config = config+ self._model = model+ self._traj_formatter = TrajectoryFormatter(+ traj_filter=config.traj_filter,+ traj_item_template=config.traj_item_template,+ traj_output_filter=config.traj_output_filter,+ filter_failed_edits=config.traj_filter_failed_edits,+ only_show_last_n_output=config.traj_only_show_last_n_output,+ )++ def format_messages(self, instance: ProblemStatement, sub1: ReviewSubmission, sub2: ReviewSubmission):+ system_message = self._config.system_template+ logger.debug(f"{self.LOG_PREFIX}MODEL INPUT (system)\n{system_message}")+ ps_format_dict = {+ "problem_statement": instance.get_problem_statement(),+ **instance.get_extra_fields(),+ }+ user_message = self._config.instance_template.format(+ **ps_format_dict,+ **sub1.to_format_dict(suffix="1"),+ **sub2.to_format_dict(suffix="2"),+ traj1=self._traj_formatter.format_trajectory(sub1.trajectory, i_traj=1),+ traj2=self._traj_formatter.format_trajectory(sub2.trajectory, i_traj=2),+ )+ logger.debug(f"{self.LOG_PREFIX}MODEL INPUT (user)\n{user_message}")+ return [+ {"role": "system", "content": system_message},+ {"role": "user", "content": user_message},+ ]++ def interpret(self, response: str) -> Literal[0, 1]:+ """Interpret response from LM. Note: 1-based indexing"""+ last_line = response.strip().split("\n")[-1].strip()+ if "first" in last_line.lower():+ return 0+ elif "second" in last_line.lower():+ return 1+ logger.warning("Could not interpret response: %s, will choose first submission.", response)+ return 0++ def compare_submissions(+ self,+ instance: ProblemStatement,+ sub1: ReviewSubmission,+ sub2: ReviewSubmission,+ rev1: ReviewerResult,+ rev2: ReviewerResult,+ ) -> BinaryReviewerResult:+ messages: History = self.format_messages(instance, sub1, sub2) # type: ignore+ answer = self._model.query(messages)["message"]+ idx = self.interpret(answer)+ # Use words because else confusion with 0-based vs 1-based indices+ choice_emoji = "first" if idx == 0 else "second"+ logger.info(f"{self.LOG_PREFIX}{choice_emoji}\n{answer}")+ return BinaryReviewerResult(idx, output=answer, messages=messages) # type: ignore+++class GraveToCradle(AbstractGraveToCradle):+ def __init__(self, config: GTCConfig, model: AbstractModel):+ self._config = config+ self._model = model++ def get_forwarded_vars(+ self,+ submissions: list[ReviewSubmission],+ reviews: list[ReviewerResult],+ breviews: list[tuple[int, int, BinaryReviewerResult]],+ ) -> dict[str, Any]:+ assert len(submissions) == len(reviews)+ failed_idxs = [i for i, r in enumerate(reviews) if not r.accept]+ if not failed_idxs:+ return {"failed_verdicts_with_submissions": ""}+ msg_lines = ["The following previous submissions were deemed to be incorrect:"]+ for i, idx in enumerate(failed_idxs):+ info = submissions[idx].info+ if not info.get("submission"):+ continue+ submission = info["submission"]+ review = reviews[idx].output+ msg_lines.append(f"Submission {i+1}:\n\n{submission}\n\nReview {i+1}:\n\n{review}")+ msg = "\n\n".join(msg_lines)+ return {"failed_verdicts_with_submissions": msg}+++class ReviewLoop(AbstractReviewLoop):+ LOG_PREFIX = "🔄 Review Loop: "++ def __init__(+ self,+ loop_config: ReviewLoopConfig,+ instance: ProblemStatement,+ model: AbstractModel,+ ):+ self._model = model+ self._instance = instance+ self._reviewer: AbstractReviewer = globals()[loop_config.reviewer_classname](loop_config.reviewer_config, model) # type: ignore+ if loop_config.binary_reviewer_classname is not None:+ self._breviewer: AbstractBinaryReviewer | None = globals()[loop_config.binary_reviewer_classname](+ loop_config.binary_reviewer_config, model+ )+ else:+ self._breviewer = None+ self._gtc: AbstractGraveToCradle | None = None+ if loop_config.gtc_classname:+ self._gtc = globals()[loop_config.gtc_classname](loop_config.gtc_config, model)+ self._loop_config = loop_config+ # Note: These are "cumulative" submissions, i.e., they include all retries+ # up to that point.+ self._submissions: list[ReviewSubmission] = []+ self._reviews: list[ReviewerResult] = []+ self._comparisons: list[tuple[int, int, BinaryReviewerResult]] = []+ # Once we have k submissions, there will always be one voted at the+ # top through k calls to the binary reviewer. Here, we store the+ # corresponding index+ self._best_idx: int = 0+ #: Number of consecutive exit cost submissions+ self._n_consec_exit_cost: int = 0++ @property+ def reviews(self) -> list[ReviewerResult]:+ return self._reviews++ @property+ def comparisons(self) -> list[tuple[int, int, BinaryReviewerResult]]:+ return self._comparisons++ @property+ def _n_samples(self) -> int:+ return len(self._submissions)++ @property+ def _n_accepted(self) -> int:+ return sum([r.accept for r in self._reviews])++ @property+ def model_stats(self) -> InstanceStats:+ return self._model.stats++ def on_submit(self, submission: ReviewSubmission) -> None:+ self._submissions.append(submission)+ self._review()+ self._compare()++ def _review(self) -> bool:+ review = self._reviewer.review(self._instance, self._submissions[-1])+ self._reviews.append(review)+ if "exit_cost" in self._submissions[-1].info.get("exit_status", "").lower():+ self._n_consec_exit_cost += 1+ else:+ self._n_consec_exit_cost = 0+ return review.accept++ def _compare(self) -> None:+ if self._breviewer is None:+ self._best_idx = self._n_samples - 1+ return+ if self._n_samples < 2:+ return+ if self._reviews[self._best_idx].accept and not self._reviews[-1].accept:+ # Require that the best submission is accepted, so don't+ # even need to compare here+ return+ if not self._reviews[self._best_idx].accept and self._reviews[-1].accept:+ # If the best submission is not accepted, but the last one is,+ # then the last one is the new best+ self._best_idx = self._n_samples - 1+ return+ sub1 = self._submissions[self._best_idx]+ sub2 = self._submissions[-1]+ rev1 = self._reviews[self._best_idx]+ rev2 = self._reviews[-1]+ cresult = self._breviewer.compare_submissions(self._instance, sub1, sub2, rev1, rev2)+ self._comparisons.append((self._n_samples - 2, self._n_samples - 1, cresult))+ assert cresult.choice in [0, 1]+ # this was a comparison between the current best and the last one+ self._best_idx = [self._best_idx, self._n_samples - 1][cresult.choice]++ def retry(self) -> bool:+ stat_str = f"n_samples={self._n_samples}, n_accepted={self._n_accepted}"+ # n_samples is 1-based+ if self._n_samples >= self._loop_config.max_samples:+ # We've exceeded our budget. Returning best solution no matter what.+ logger.info(f"{self.LOG_PREFIX}Exiting retry loop ({stat_str}): `max_samples` reached")+ return False++ if self._n_accepted and self._n_samples >= self._loop_config.min_draws:+ # We have an accepted submission and have reached the minimum number of draws+ logger.info(+ f"{self.LOG_PREFIX}Existing retry loop ({stat_str}): `min_draws` reached and last submission was accepted"+ )+ return False++ if self._n_accepted >= self._loop_config.max_accepted_draws > 0:+ # We have reached more than the required number of accepted submissions.+ # Exiting even if we haven't reached the minimum number of draws.+ logger.info(f"{self.LOG_PREFIX}Exiting retry loop ({stat_str}): `max_accepted_draws` reached")+ return False++ max_n_exit_cost = self._loop_config.max_n_consec_exit_cost+ if self._n_consec_exit_cost >= max_n_exit_cost > 0:+ logger.info(+ f"{self.LOG_PREFIX}Exiting retry loop ({stat_str}): {max_n_exit_cost} exit cost attempts reached"+ )+ return False++ return True++ def get_best(self) -> int:+ if self._breviewer is not None:+ assert len(self._reviews) == len(self._submissions)+ return self._best_idx++ def get_forwarded_vars(self) -> dict[str, Any]:+ if self._gtc is None:+ return {}+ return self._gtc.get_forwarded_vars(self._submissions, self._reviews, self._comparisons)+++def get_review_loop_from_config(+ config: ReviewLoopConfig | None, instance: ProblemStatement, model: AbstractModel+) -> AbstractReviewLoop | None:+ if config is None:+ logger.debug("Running without review loop")+ return None+ if not isinstance(config, ReviewLoopConfig):+ msg = (+ f"Expected a `ReviewLoopConfig`, got {config!r} of type {type(config)}. "+ "This usually happens for misconfiguration issues: Make sure that all validation "+ "conditions are satisfied (e.g., min draws <= max draws etc.) and that all "+ "required keys are there."+ )+ raise ValueError(msg)+ return globals()[config.review_loop_classname](config, instance, model)