SWE-agent takes a GitHub issue and tries to automatically fix it, using your LM of choice. It can also be employed for offensive cybersecurity or competitive coding challenges. [NeurIPS 2024]
Always save replay config
Kilian Lieretf3a5a2579f76
diff --git a/docs/usage/trajectories.md b/docs/usage/trajectories.mdindex 24c066cf..397fb8c9 100644--- a/docs/usage/trajectories.md+++ b/docs/usage/trajectories.md@@ -1,57 +1,69 @@-# Trajectories+# Output files-The [`trajectories/`](https://github.com/princeton-nlp/SWE-agent/tree/main/trajectories) folder is the default location that experiment results (invocations of [`run.py`](cl_tutorial.md)) will be written to.+The [`trajectories/`](https://github.com/princeton-nlp/SWE-agent/tree/main/trajectories) folder is the default location that experiment results (for example invocations of [`sweagent run`](cl_tutorial.md)) will be written to.++The main output file is `<instance_id>.traj`, which is a `.json` formatted file containing the (thought, action, observation) turns generated by SWE-agent towards solving `<instance_id>`.<details><summary>Example trajectory</summary>+Note: This trajectory is from SWE-agent v0.7.0. It might look different in newer versions.+```json--8<-- "trajectories/demonstrations/replay__marshmallow-code__marshmallow-1867__default__t-0.20__p-0.95__c-2.00__install-1___install_from_source/marshmallow-code__marshmallow-1867.traj"```</details>+Together with the trajectory, SWE-agent will also generate the following files per instance:++* `config.yaml`: A yaml file containing the configuration used for the experiment. You can directly repeat the run of this+ exact instance by running `sweagent run --config config.yaml`.+* `*.log`: Log files at different levels+At a high level, the experiments folder is organized in the following manner:+```-trajectories-├── <user 1> 👩💻-│ ├── <experiment 1> 🧪-│ │ ├── all_preds.jsonl-│ │ ├── args.yaml-│ │ ├── *.html (Webpage Files)-│ │ └── *.traj (Trajectories)-│ └── <experiment 2> 🧪-│ ├── all_preds.jsonl-│ ├── args.yaml-│ ├── *.html (Webpage Files)-│ └── *.traj (Trajectories)-├── <user 2> 👨💻-│ ├── <experiment 1> 🧪-│ │ └── ...-│ └── <experiment 2> 🧪-│ └── ...-...+trajectories:+ user 1:+ experiment 1:+ run_batch.config.yaml+ preds.json+ run_batch.debug.log+ run_batch.info.log+ run_batch.warning.log+ run_batch_exit_statuses.yaml+ instance_1:+ instance_1.traj+ instance_1.config.yaml+ instance_1.debug.log+ instance_1.info.log+ instance_1.warning.log+ instance 2:+ ...+ experiment 2:+ ...+ user 2:+ ...```-Where every experiment follows the pattern `trajectories/<user name>/<experiment name>`. The `<user name>` is automatically inferred from your system, and the `experiment name` is inferred from the arguments of the `run.py`.++Where every experiment follows the pattern `trajectories/<user name>/<experiment name>`. The `<user name>` is automatically inferred from your system, and the `experiment name` is inferred from the model names and other settings to the run command.+++Here are a few more files that are generated with `sweagent run-batch`:++* `run_batch.config.yaml`: A yaml file containing the configuration used for the experiment. You can directly repeat the run of this+ exact experiment by running `sweagent run-batch --config run_batch.config.yaml`.+* `preds.json`: All predictions (i.e., patches produced by the agent) from all instances of the experiment.+ If this file goes out of date, you can regenerate it by running `sweagent merge-preds`.+* `run_batch.*.log`: Log files at different levels.+* `run_batch_exit_statuses.yaml`: A yaml file containing the exit statuses of all instances in the experiment (for this run of `sweagent run-batch`).+!!! tip "Viewing trajectories"We provide a [trajectory viewer](inspector.md) for an easy viewing of trajectories.## How an Experiment Folder is Generated-Each call to `run.py` produces a single `trajectories/<user name>/<experiment name>` folder containing the following assets:--* `all_preds.jsonl`: A single file containing all of the predictions generated for the experiment (1 prediction per task instance), where each line is formatted as:-```-{- "instance_id": "<Unique task instance ID>",- "model_patch": "<.patch file content string>",- "model_name_or_path": "<Model name here (Inferred from experiment configs)>",-}-```-* `args.yaml`: A summary of the configurations for the experiment run.-* `<instance_id>.traj`: A `.json` formatted file containing the (thought, action, observation) turns generated by SWE-agent towards solving `<instance_id>`.-* `<instance_id>.html`: An `.html` single webpage render of the trajectory, which can be directly opened in the browser for easier viewing of the trajectory.-!!! tip* Evaluation is not completed by `run.py`, it is a separate step (see [benchmarking](benchmarking.md))* `all_preds.jsonl` can be referenced directly into SWE-bench to run evaluation (see [benchmarking](benchmarking.md))diff --git a/sweagent/agent/agents.py b/sweagent/agent/agents.pyindex 33003fa8..264450b9 100644--- a/sweagent/agent/agents.py+++ b/sweagent/agent/agents.py@@ -206,12 +206,15 @@ class Agent:# ----------@property- def replay_config(self) -> dict | None:+ def replay_config(self) -> BaseModel | None:return self._replay_config@replay_config.setter- def replay_config(self, value: BaseModel | None):- self._replay_config = _strip_abspath_from_dict(value.model_dump(mode="json")) if value else None+ def replay_config(self, value: BaseModel):+ # Do import here to avoid circular dependency+ from sweagent.run.run_single import RunSingleConfig++ self._replay_config = RunSingleConfig.model_validate(_strip_abspath_from_dict(value.model_dump()))@propertydef history(self) -> History:@@ -474,7 +477,9 @@ class Agent:"info": self._info_by_attempt[attempt_idx],})- attempt_data["replay_config"] = self.replay_config+ attempt_data["replay_config"] = (+ self.replay_config.model_dump_json() if self.replay_config is not None else None+ )attempt_data["environment"] = self._env.namereturn attempt_datadiff --git a/sweagent/run/run_batch.py b/sweagent/run/run_batch.pyindex eeb1ae89..4e0473ca 100644--- a/sweagent/run/run_batch.py+++ b/sweagent/run/run_batch.py@@ -41,6 +41,7 @@ from contextlib import ExitStackfrom pathlib import Pathfrom typing import Self+import yamlfrom pydantic import Fieldfrom pydantic_settings import BaseSettingsfrom rich.live import Live@@ -162,6 +163,7 @@ class RunBatch:load_environment_variables(config.env_var_path)config.set_default_output_dir()config.output_dir.mkdir(parents=True, exist_ok=True)+ (config.output_dir / "run_batch.config.yaml").write_text(yaml.dump(config.model_dump_json(), indent=2))logger = get_logger("run", emoji="🏃")logger.debug("Loading instances from %s", f"{config.instances!r}")instances = config.instances.get_instance_configs()@@ -287,6 +289,8 @@ class RunBatch:self._remove_instance_log_file_handlers(instance.problem_statement.id)def _run_instance(self, instance: BatchInstance) -> AgentRunResult:+ output_dir = Path(self.output_dir) / instance.problem_statement.id+ output_dir.mkdir(parents=True, exist_ok=True)self.agent_config.name = f"{instance.problem_statement.id}"agent = Agent.from_config(self.agent_config)single_run_replay_config = RunSingleConfig(@@ -294,6 +298,9 @@ class RunBatch:problem_statement=instance.problem_statement,env=instance.env,)+ (output_dir / f"{instance.problem_statement.id}.config.yaml").write_text(+ yaml.dump(single_run_replay_config.model_dump_json(), indent=2)+ )agent.replay_config = single_run_replay_configagent.add_hook(SetStatusAgentHook(instance.problem_statement.id, self._progress_manager.update_instance_status))self._progress_manager.update_instance_status(instance.problem_statement.id, "Starting environment")@@ -312,7 +319,7 @@ class RunBatch:result = agent.run(problem_statement=instance.problem_statement,env=env,- output_dir=Path(self.output_dir) / instance.problem_statement.id,+ output_dir=output_dir,)except Exception:# The actual handling is happening in `run_instance`, but we need to make sure thatdiff --git a/sweagent/run/run_single.py b/sweagent/run/run_single.pyindex 041719c8..8338553e 100644--- a/sweagent/run/run_single.py+++ b/sweagent/run/run_single.py@@ -32,6 +32,7 @@ import sysfrom pathlib import Pathfrom typing import Self+import yamlfrom pydantic import BaseModel, ConfigDict, Fieldfrom pydantic_settings import BaseSettings@@ -158,10 +159,14 @@ class RunSingle:self.env.start()self.logger.info("Running agent")self._chooks.on_instance_start(index=0, env=self.env, problem_statement=self.problem_statement)+ output_dir = self.output_dir / self.problem_statement.id+ output_dir.mkdir(parents=True, exist_ok=True)+ if self.agent.replay_config is not None:+ (output_dir / "config.yaml").write_text(yaml.dump(self.agent.replay_config.model_dump_json(), indent=2))result = self.agent.run(problem_statement=self.problem_statement,env=self.env,- output_dir=Path(self.output_dir),+ output_dir=output_dir,)self._chooks.on_instance_completed(result=result)self.logger.info("Done")diff --git a/tests/test_run_single.py b/tests/test_run_single.pyindex f3422dff..bb30317f 100644--- a/tests/test_run_single.py+++ b/tests/test_run_single.py@@ -36,6 +36,7 @@ def agent_config_with_commands():Bundle(path=TOOLS_DIR / "defaults"),Bundle(path=TOOLS_DIR / "submit"),]+ ac.tools.env_variables = {"WINDOW": 100}assert (TOOLS_DIR / "submit").exists()# Make sure dependent properties are setac.tools.model_post_init(None)@@ -45,6 +46,7 @@ def agent_config_with_commands():@pytest.mark.slowdef test_hidden_tools(tmpdir):ac = AgentConfig(model=InstantEmptySubmitModelConfig())+ ac.tools.env_variables = {"WINDOW": 100}ac.tools.bundles = [Bundle(path=TOOLS_DIR / "defaults", hidden_tools=["scroll_up"]),Bundle(path=TOOLS_DIR / "submit"),@@ -117,5 +119,5 @@ def test_run_ies_repo_ps_matrix(# Test that we can run run.py also independently from repo dirrs.run()for fmt in output_formats:- assert len(list(Path(tmpdir).glob(f"*.{fmt}"))) == 1+ assert len(list(Path(tmpdir).rglob(f"*.{fmt}"))) == 1print(fmt, list(Path(tmpdir).iterdir()))