Skip to content

SWE-agent takes a GitHub issue and tries to automatically fix it, using your LM of choice. It can also be employed for offensive cybersecurity or competitive coding challenges. [NeurIPS 2024]

Always save replay config

Kilian Lieretf3a5a2579f76
diff --git a/docs/usage/trajectories.md b/docs/usage/trajectories.md
index 24c066cf..397fb8c9 100644
--- a/docs/usage/trajectories.md
+++ b/docs/usage/trajectories.md
@@ -1,57 +1,69 @@
-# Trajectories
+# Output files
-The [`trajectories/`](https://github.com/princeton-nlp/SWE-agent/tree/main/trajectories) folder is the default location that experiment results (invocations of [`run.py`](cl_tutorial.md)) will be written to.
+The [`trajectories/`](https://github.com/princeton-nlp/SWE-agent/tree/main/trajectories) folder is the default location that experiment results (for example invocations of [`sweagent run`](cl_tutorial.md)) will be written to.
+
+The main output file is `<instance_id>.traj`, which is a `.json` formatted file containing the (thought, action, observation) turns generated by SWE-agent towards solving `<instance_id>`.
<details>
<summary>Example trajectory</summary>
+Note: This trajectory is from SWE-agent v0.7.0. It might look different in newer versions.
+
```json
--8<-- "trajectories/demonstrations/replay__marshmallow-code__marshmallow-1867__default__t-0.20__p-0.95__c-2.00__install-1___install_from_source/marshmallow-code__marshmallow-1867.traj"
```
</details>
+Together with the trajectory, SWE-agent will also generate the following files per instance:
+
+* `config.yaml`: A yaml file containing the configuration used for the experiment. You can directly repeat the run of this
+ exact instance by running `sweagent run --config config.yaml`.
+* `*.log`: Log files at different levels
+
At a high level, the experiments folder is organized in the following manner:
+
```
-trajectories
-├── <user 1> 👩‍💻
-│ ├── <experiment 1> 🧪
-│ │ ├── all_preds.jsonl
-│ │ ├── args.yaml
-│ │ ├── *.html (Webpage Files)
-│ │ └── *.traj (Trajectories)
-│ └── <experiment 2> 🧪
-│ ├── all_preds.jsonl
-│ ├── args.yaml
-│ ├── *.html (Webpage Files)
-│ └── *.traj (Trajectories)
-├── <user 2> 👨‍💻
-│ ├── <experiment 1> 🧪
-│ │ └── ...
-│ └── <experiment 2> 🧪
-│ └── ...
-...
+trajectories:
+ user 1:
+ experiment 1:
+ run_batch.config.yaml
+ preds.json
+ run_batch.debug.log
+ run_batch.info.log
+ run_batch.warning.log
+ run_batch_exit_statuses.yaml
+ instance_1:
+ instance_1.traj
+ instance_1.config.yaml
+ instance_1.debug.log
+ instance_1.info.log
+ instance_1.warning.log
+ instance 2:
+ ...
+ experiment 2:
+ ...
+ user 2:
+ ...
```
-Where every experiment follows the pattern `trajectories/<user name>/<experiment name>`. The `<user name>` is automatically inferred from your system, and the `experiment name` is inferred from the arguments of the `run.py`.
+
+Where every experiment follows the pattern `trajectories/<user name>/<experiment name>`. The `<user name>` is automatically inferred from your system, and the `experiment name` is inferred from the model names and other settings to the run command.
+
+
+Here are a few more files that are generated with `sweagent run-batch`:
+
+* `run_batch.config.yaml`: A yaml file containing the configuration used for the experiment. You can directly repeat the run of this
+ exact experiment by running `sweagent run-batch --config run_batch.config.yaml`.
+* `preds.json`: All predictions (i.e., patches produced by the agent) from all instances of the experiment.
+ If this file goes out of date, you can regenerate it by running `sweagent merge-preds`.
+* `run_batch.*.log`: Log files at different levels.
+* `run_batch_exit_statuses.yaml`: A yaml file containing the exit statuses of all instances in the experiment (for this run of `sweagent run-batch`).
+
!!! tip "Viewing trajectories"
We provide a [trajectory viewer](inspector.md) for an easy viewing of trajectories.
## How an Experiment Folder is Generated
-Each call to `run.py` produces a single `trajectories/<user name>/<experiment name>` folder containing the following assets:
-
-* `all_preds.jsonl`: A single file containing all of the predictions generated for the experiment (1 prediction per task instance), where each line is formatted as:
-```
-{
- "instance_id": "<Unique task instance ID>",
- "model_patch": "<.patch file content string>",
- "model_name_or_path": "<Model name here (Inferred from experiment configs)>",
-}
-```
-* `args.yaml`: A summary of the configurations for the experiment run.
-* `<instance_id>.traj`: A `.json` formatted file containing the (thought, action, observation) turns generated by SWE-agent towards solving `<instance_id>`.
-* `<instance_id>.html`: An `.html` single webpage render of the trajectory, which can be directly opened in the browser for easier viewing of the trajectory.
-
!!! tip
* Evaluation is not completed by `run.py`, it is a separate step (see [benchmarking](benchmarking.md))
* `all_preds.jsonl` can be referenced directly into SWE-bench to run evaluation (see [benchmarking](benchmarking.md))
diff --git a/sweagent/agent/agents.py b/sweagent/agent/agents.py
index 33003fa8..264450b9 100644
--- a/sweagent/agent/agents.py
+++ b/sweagent/agent/agents.py
@@ -206,12 +206,15 @@ class Agent:
# ----------
@property
- def replay_config(self) -> dict | None:
+ def replay_config(self) -> BaseModel | None:
return self._replay_config
@replay_config.setter
- def replay_config(self, value: BaseModel | None):
- self._replay_config = _strip_abspath_from_dict(value.model_dump(mode="json")) if value else None
+ def replay_config(self, value: BaseModel):
+ # Do import here to avoid circular dependency
+ from sweagent.run.run_single import RunSingleConfig
+
+ self._replay_config = RunSingleConfig.model_validate(_strip_abspath_from_dict(value.model_dump()))
@property
def history(self) -> History:
@@ -474,7 +477,9 @@ class Agent:
"info": self._info_by_attempt[attempt_idx],
}
)
- attempt_data["replay_config"] = self.replay_config
+ attempt_data["replay_config"] = (
+ self.replay_config.model_dump_json() if self.replay_config is not None else None
+ )
attempt_data["environment"] = self._env.name
return attempt_data
diff --git a/sweagent/run/run_batch.py b/sweagent/run/run_batch.py
index eeb1ae89..4e0473ca 100644
--- a/sweagent/run/run_batch.py
+++ b/sweagent/run/run_batch.py
@@ -41,6 +41,7 @@ from contextlib import ExitStack
from pathlib import Path
from typing import Self
+import yaml
from pydantic import Field
from pydantic_settings import BaseSettings
from rich.live import Live
@@ -162,6 +163,7 @@ class RunBatch:
load_environment_variables(config.env_var_path)
config.set_default_output_dir()
config.output_dir.mkdir(parents=True, exist_ok=True)
+ (config.output_dir / "run_batch.config.yaml").write_text(yaml.dump(config.model_dump_json(), indent=2))
logger = get_logger("run", emoji="🏃")
logger.debug("Loading instances from %s", f"{config.instances!r}")
instances = config.instances.get_instance_configs()
@@ -287,6 +289,8 @@ class RunBatch:
self._remove_instance_log_file_handlers(instance.problem_statement.id)
def _run_instance(self, instance: BatchInstance) -> AgentRunResult:
+ output_dir = Path(self.output_dir) / instance.problem_statement.id
+ output_dir.mkdir(parents=True, exist_ok=True)
self.agent_config.name = f"{instance.problem_statement.id}"
agent = Agent.from_config(self.agent_config)
single_run_replay_config = RunSingleConfig(
@@ -294,6 +298,9 @@ class RunBatch:
problem_statement=instance.problem_statement,
env=instance.env,
)
+ (output_dir / f"{instance.problem_statement.id}.config.yaml").write_text(
+ yaml.dump(single_run_replay_config.model_dump_json(), indent=2)
+ )
agent.replay_config = single_run_replay_config
agent.add_hook(SetStatusAgentHook(instance.problem_statement.id, self._progress_manager.update_instance_status))
self._progress_manager.update_instance_status(instance.problem_statement.id, "Starting environment")
@@ -312,7 +319,7 @@ class RunBatch:
result = agent.run(
problem_statement=instance.problem_statement,
env=env,
- output_dir=Path(self.output_dir) / instance.problem_statement.id,
+ output_dir=output_dir,
)
except Exception:
# The actual handling is happening in `run_instance`, but we need to make sure that
diff --git a/sweagent/run/run_single.py b/sweagent/run/run_single.py
index 041719c8..8338553e 100644
--- a/sweagent/run/run_single.py
+++ b/sweagent/run/run_single.py
@@ -32,6 +32,7 @@ import sys
from pathlib import Path
from typing import Self
+import yaml
from pydantic import BaseModel, ConfigDict, Field
from pydantic_settings import BaseSettings
@@ -158,10 +159,14 @@ class RunSingle:
self.env.start()
self.logger.info("Running agent")
self._chooks.on_instance_start(index=0, env=self.env, problem_statement=self.problem_statement)
+ output_dir = self.output_dir / self.problem_statement.id
+ output_dir.mkdir(parents=True, exist_ok=True)
+ if self.agent.replay_config is not None:
+ (output_dir / "config.yaml").write_text(yaml.dump(self.agent.replay_config.model_dump_json(), indent=2))
result = self.agent.run(
problem_statement=self.problem_statement,
env=self.env,
- output_dir=Path(self.output_dir),
+ output_dir=output_dir,
)
self._chooks.on_instance_completed(result=result)
self.logger.info("Done")
diff --git a/tests/test_run_single.py b/tests/test_run_single.py
index f3422dff..bb30317f 100644
--- a/tests/test_run_single.py
+++ b/tests/test_run_single.py
@@ -36,6 +36,7 @@ def agent_config_with_commands():
Bundle(path=TOOLS_DIR / "defaults"),
Bundle(path=TOOLS_DIR / "submit"),
]
+ ac.tools.env_variables = {"WINDOW": 100}
assert (TOOLS_DIR / "submit").exists()
# Make sure dependent properties are set
ac.tools.model_post_init(None)
@@ -45,6 +46,7 @@ def agent_config_with_commands():
@pytest.mark.slow
def test_hidden_tools(tmpdir):
ac = AgentConfig(model=InstantEmptySubmitModelConfig())
+ ac.tools.env_variables = {"WINDOW": 100}
ac.tools.bundles = [
Bundle(path=TOOLS_DIR / "defaults", hidden_tools=["scroll_up"]),
Bundle(path=TOOLS_DIR / "submit"),
@@ -117,5 +119,5 @@ def test_run_ies_repo_ps_matrix(
# Test that we can run run.py also independently from repo dir
rs.run()
for fmt in output_formats:
- assert len(list(Path(tmpdir).glob(f"*.{fmt}"))) == 1
+ assert len(list(Path(tmpdir).rglob(f"*.{fmt}"))) == 1
print(fmt, list(Path(tmpdir).iterdir()))