Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

Let a cloud machine move to another instance type and keep its disk

Petr Bauch2febc976a97b
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2199b6e26..e676b2e38 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -111,6 +111,14 @@ version of their own to them without also giving them a release of their own.
compares it only when someone signs in.
#### Changed
+- **A cloud machine can move to another instance type and keep its disk.**
+ The hosted controller now launches every new instance of a machine as the
+ instance type in its `gateway.json`, at the same moment it already moves the
+ machine onto the configured image, and `upgrade --gateway-config` moves a
+ stopped machine onto both. `allowed_instance_types` is no longer bound to the
+ controller's state database, so it can change while machines exist; a
+ database bound before this release is accepted and rebound on first start,
+ as long as `allowed_instance_types` has not changed in the same deploy.
- **Following the conversation is described the same way everywhere, and the
chart can turn it on.** The feature added in 0.31.0 is off unless
`follow_the_conversation` is among the enabled feature flags
diff --git a/deploy/hosted/README.md b/deploy/hosted/README.md
index a7568a28a..4720e913a 100644
--- a/deploy/hosted/README.md
+++ b/deploy/hosted/README.md
@@ -74,9 +74,10 @@ means the server cannot run them. Keep it no higher than the hosted controller's
Mount a secret with `gateway.json` for the hosted controller, holding `origin`,
the matching `token` and `instance_type`, and set the chart's
-`gatewaySecretName` to it. `instance_type` is used for every new machine and
-must be one of `allowed_instance_types` in both Terraform and the controller
-configuration; `c7i.2xlarge` is recommended. The token authorizes only the
+`gatewaySecretName` to it. `instance_type` is used for every new machine, and
+an existing machine moves onto it the next time it launches an instance (see
+"Operator commands"). It must be one of `allowed_instance_types` in both
+Terraform and the controller configuration; `c7i.2xlarge` is recommended. The token authorizes only the
hosted controller routes, for every machine on the server: their lifecycle
and the controllers they run, never a workspace's agents, rooms or logins. It
is not a user or agent API key.
@@ -141,13 +142,18 @@ switch-hosted-controller --config /etc/switch-hosted/controller.json status <mac
switch-hosted-controller --config /etc/switch-hosted/controller.json stop <machine-id>
switch-hosted-controller --config /etc/switch-hosted/controller.json start <machine-id>
switch-hosted-controller --config /etc/switch-hosted/controller.json delete <machine-id> --confirm-machine-id <machine-id> --retain-volume
-switch-hosted-controller --config /etc/switch-hosted/controller.json upgrade <machine-id> --confirm-instance-id i-0123456789abcdef0
+switch-hosted-controller --config /etc/switch-hosted/controller.json --gateway-config /etc/switch-hosted-gateway/gateway.json upgrade <machine-id> --confirm-instance-id i-0123456789abcdef0
```
-Every new instance of a machine launches from the configured image: a
-relaunch for new user data, a recovery, or a retained machine starting again
-moves it onto a new `image_id`, keeping its disk. `upgrade` does it for a
-stopped machine whose instance is terminated. A queued command is not confirmation that
+Every new instance of a machine launches from the configured image, as
+`gateway.json`'s `instance_type`: a relaunch for new user data, a recovery, or
+a retained machine starting again moves it onto a new `image_id` and instance
+type, keeping its disk. `upgrade` does it for a stopped machine whose instance
+is terminated; without `--gateway-config` it keeps the machine's instance type.
+`allowed_instance_types` may change while machines exist: add the new type,
+move the machines, then drop the old one. The other placement settings
+(installation, region, zone, subnet, security groups, root device and volume
+sizes) are bound to the state database and cannot change. A queued command is not confirmation that
AWS has done it. Deletion needs `--confirm-machine-id` and exactly one of
`--retain-volume` or `--delete-volume`. Deleting a machine does not remove
snapshots, roles, the NAT gateway or the controller's database.
diff --git a/deploy/hosted/controller/src/switch_hosted_controller/cli.py b/deploy/hosted/controller/src/switch_hosted_controller/cli.py
index c196caa42..590879efe 100644
--- a/deploy/hosted/controller/src/switch_hosted_controller/cli.py
+++ b/deploy/hosted/controller/src/switch_hosted_controller/cli.py
@@ -22,7 +22,7 @@ from .health import check_health
from .lock import ControllerAlreadyRunning, ControllerLock
from .model import DesiredState, Machine
from .reconciler import Reconciler
-from .store import MachineStore, StoreError
+from .store import MachineStore, StoreError, rebind_superseded_fingerprint
def parser() -> argparse.ArgumentParser:
@@ -46,7 +46,9 @@ def parser() -> argparse.ArgumentParser:
cleanup.add_argument("--delete-volume", action="store_true")
upgrade = subparsers.add_parser(
- "upgrade", help="use the configured image after the old instance is stopped and terminated"
+ "upgrade",
+ help="use the configured image, and the cloud gateway's instance type with"
+ " --gateway-config, after the old instance is stopped and terminated",
)
upgrade.add_argument("machine_id")
upgrade.add_argument("--confirm-instance-id", required=True)
@@ -75,7 +77,7 @@ def main(argv: list[str] | None = None) -> int:
def _state_command(config: ControllerConfig, args: argparse.Namespace) -> int:
- store = MachineStore(config.state_db_path, config.fingerprint())
+ store = _open_store(config)
try:
if args.command == "create":
if args.instance_type not in config.allowed_instance_types:
@@ -136,24 +138,41 @@ def _state_command(config: ControllerConfig, args: argparse.Namespace) -> int:
raise StoreError(
"the old instance must be confirmed terminated and its retained disk detached"
)
+ instance_type = (
+ GatewayConfig.load(args.gateway_config).instance_type
+ if args.gateway_config
+ else machine.instance_type
+ )
+ if instance_type not in config.allowed_instance_types:
+ raise ConfigError("instance type is not in allowed_instance_types")
cloud.validate_image(replace(machine, image_id=config.image_id))
claim = store.mark_instance_terminal_observed(machine.machine_id, machine.instance_id)
- _print_machine(store.upgrade_terminated(claim, config.image_id))
+ _print_machine(store.upgrade_terminated(claim, config.image_id, instance_type))
return 0
raise AssertionError(f"unhandled command {args.command}")
finally:
store.close()
+def _open_store(config: ControllerConfig) -> MachineStore:
+ rebind_superseded_fingerprint(
+ config.state_db_path, config.superseded_fingerprint(), config.fingerprint()
+ )
+ return MachineStore(config.state_db_path, config.fingerprint())
+
+
def _reconcile_command(config: ControllerConfig, command: str, gateway_path: Path | None) -> int:
with ControllerLock(config.lock_path):
- store = MachineStore(config.state_db_path, config.fingerprint())
+ store = _open_store(config)
try:
ec2 = boto3.client("ec2", region_name=config.region)
- reconciler = Reconciler(store, Ec2Cloud(ec2, config))
- gateway = (
- Gateway(GatewayConfig.load(gateway_path), config, store) if gateway_path else None
+ settings = GatewayConfig.load(gateway_path) if gateway_path else None
+ reconciler = Reconciler(
+ store,
+ Ec2Cloud(ec2, config),
+ instance_type=settings.instance_type if settings else None,
)
+ gateway = Gateway(settings, config, store) if settings else None
_touch_health()
if command == "reconcile-once":
try:
diff --git a/deploy/hosted/controller/src/switch_hosted_controller/config.py b/deploy/hosted/controller/src/switch_hosted_controller/config.py
index 1a3a6fcb0..78e741f39 100644
--- a/deploy/hosted/controller/src/switch_hosted_controller/config.py
+++ b/deploy/hosted/controller/src/switch_hosted_controller/config.py
@@ -138,19 +138,33 @@ class ControllerConfig:
)
def fingerprint(self) -> str:
- immutable = {
+ """What the state database is bound to: where machines and their disks are.
+ The allowed instance types are not part of it; IAM enforces them on launch,
+ and a machine moves onto the configured type when it next launches."""
+ return _fingerprint_of(self._placement())
+
+ def superseded_fingerprint(self) -> str:
+ """The fingerprint before it left out the allowed instance types."""
+ return _fingerprint_of(
+ {**self._placement(), "allowed_instance_types": sorted(self.allowed_instance_types)}
+ )
+
+ def _placement(self) -> dict[str, Any]:
+ return {
"installation_id": self.installation_id,
"region": self.region,
"availability_zone": self.availability_zone,
"subnet_id": self.subnet_id,
"security_group_ids": sorted(self.security_group_ids),
"root_device_name": self.root_device_name,
- "allowed_instance_types": sorted(self.allowed_instance_types),
"root_volume_gib": self.root_volume_gib,
"data_volume_gib": self.data_volume_gib,
}
- payload = json.dumps(immutable, separators=(",", ":"), sort_keys=True)
- return hashlib.sha256(payload.encode()).hexdigest()
+
+
+def _fingerprint_of(immutable: dict[str, Any]) -> str:
+ payload = json.dumps(immutable, separators=(",", ":"), sort_keys=True)
+ return hashlib.sha256(payload.encode()).hexdigest()
def _validated_string(raw: dict[str, Any], key: str, pattern: re.Pattern[str]) -> str:
diff --git a/deploy/hosted/controller/src/switch_hosted_controller/reconciler.py b/deploy/hosted/controller/src/switch_hosted_controller/reconciler.py
index 9894c6be1..7aaa6a44a 100644
--- a/deploy/hosted/controller/src/switch_hosted_controller/reconciler.py
+++ b/deploy/hosted/controller/src/switch_hosted_controller/reconciler.py
@@ -18,9 +18,12 @@ LAUNCH_VISIBILITY_GRACE = timedelta(minutes=15)
class Reconciler:
- def __init__(self, store: MachineStore, cloud: Ec2Cloud):
+ def __init__(self, store: MachineStore, cloud: Ec2Cloud, *, instance_type: str | None):
+ """`instance_type` is the type every machine moves onto when it next
+ launches, the cloud gateway's; None keeps each on the type it was created with."""
self._store = store
self._cloud = cloud
+ self._instance_type = instance_type
def reconcile_all(self) -> None:
for listed in self._store.list():
@@ -91,10 +94,16 @@ class Reconciler:
if not machine.instance_launch_issued:
if not self._unchanged(claim, DesiredState.RUNNING):
return self._store.cancel_queued_instance_launch(claim)
- if machine.image_id != self._cloud.image_id:
- # No instance is left, and the data volume is kept: the
- # one moment the machine can move onto the configured image.
- machine = self._store.use_image(claim, self._cloud.image_id)
+ instance_type = self._instance_type or machine.instance_type
+ if (
+ machine.image_id != self._cloud.image_id
+ or machine.instance_type != instance_type
+ ):
+ # No instance is left, and the data volume is kept: the one
+ # moment the machine can move onto the configured image and type.
+ machine = self._store.use_launch_spec(
+ claim, self._cloud.image_id, instance_type
+ )
if not self._same_claim(claim, machine):
return machine
self._cloud.validate_image(machine)
diff --git a/deploy/hosted/controller/src/switch_hosted_controller/store.py b/deploy/hosted/controller/src/switch_hosted_controller/store.py
index 4d7d9882e..c3fb8adc8 100644
--- a/deploy/hosted/controller/src/switch_hosted_controller/store.py
+++ b/deploy/hosted/controller/src/switch_hosted_controller/store.py
@@ -31,6 +31,26 @@ class MachineNotFoundError(StoreError):
pass
+def rebind_superseded_fingerprint(path: Path, superseded: str, fingerprint: str) -> None:
+ """Moves a state database bound to the controller's superseded fingerprint
+ onto its current one, before MachineStore checks it. The two describe the
+ same placement; a database bound to anything else is left for MachineStore
+ to refuse."""
+ if not path.exists():
+ return
+ connection = sqlite3.connect(path, timeout=30, isolation_level=None)
+ try:
+ if connection.execute(
+ "SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'controller_metadata'"
+ ).fetchone():
+ connection.execute(
+ "UPDATE controller_metadata SET value = ? WHERE key = ? AND value = ?",
+ (fingerprint, "controller_fingerprint", superseded),
+ )
+ finally:
+ connection.close()
+
+
class MachineStore:
def __init__(self, path: Path, controller_fingerprint: str):
path.parent.mkdir(parents=True, exist_ok=True)
@@ -296,21 +316,21 @@ class MachineStore:
def mark_volume_create_intent(self, claim: Machine) -> Machine:
return self._mark_intent(claim, "volume_create_intent")
- def use_image(self, claim: Machine, image_id: str) -> Machine:
- """Launch the machine's next instance from `image_id`: only while it has
- no instance and no launch is issued, so nothing launched is told apart
- from its launch request by the image."""
+ def use_launch_spec(self, claim: Machine, image_id: str, instance_type: str) -> Machine:
+ """Launch the machine's next instance from `image_id` as `instance_type`:
+ only while it has no instance and no launch is issued, so nothing
+ launched is told apart from its launch request by either."""
self._connection.execute(
"""
- UPDATE machines SET image_id = ?, updated_at = CURRENT_TIMESTAMP
+ UPDATE machines SET image_id = ?, instance_type = ?, updated_at = CURRENT_TIMESTAMP
WHERE machine_id = ? AND desired_revision = ? AND operation_id = ?
AND instance_id IS NULL AND instance_launch_issued = 0
""",
- (image_id, claim.machine_id, claim.desired_revision, claim.operation_id),
+ (image_id, instance_type, claim.machine_id, claim.desired_revision, claim.operation_id),
)
return self.get(claim.machine_id)
- def upgrade_terminated(self, claim: Machine, image_id: str) -> Machine:
+ def upgrade_terminated(self, claim: Machine, image_id: str, instance_type: str) -> Machine:
if (
claim.desired_state is not DesiredState.STOPPED
or not claim.instance_id
@@ -319,12 +339,12 @@ class MachineStore:
raise StoreError(
"image upgrades require a stopped machine and confirmed terminated predecessor"
)
- if image_id == claim.image_id:
- raise StoreError("the machine already uses this image")
+ if image_id == claim.image_id and instance_type == claim.instance_type:
+ raise StoreError("the machine already uses this image and instance type")
cursor = self._connection.execute(
"""UPDATE machines SET previous_instance_id = instance_id, instance_id = NULL,
instance_bundle = NULL,
- image_id = ?, instance_seq = instance_seq + 1,
+ image_id = ?, instance_type = ?, instance_seq = instance_seq + 1,
desired_revision = desired_revision + 1, operation_id = ?,
instance_launch_intent = 0, instance_launch_issued = 0, instance_launch_issued_at = NULL,
instance_terminate_issued = 0,
@@ -333,6 +353,7 @@ class MachineStore:
AND desired_state = 'stopped' AND instance_id = ? AND instance_terminal_observed = 1""",
(
image_id,
+ instance_type,
str(uuid.uuid4()),
claim.machine_id,
claim.desired_revision,
@@ -341,7 +362,7 @@ class MachineStore:
),
)
if cursor.rowcount != 1:
- raise StoreError("machine state changed during image upgrade; refresh before retrying")
+ raise StoreError("machine state changed during upgrade; refresh before retrying")
return self.get(claim.machine_id)
def replace_terminated(self, claim: Machine, *, unexpected: bool) -> Machine:
diff --git a/deploy/hosted/controller/tests/test_bundle_revisions.py b/deploy/hosted/controller/tests/test_bundle_revisions.py
index 7d929661d..f030e3b91 100644
--- a/deploy/hosted/controller/tests/test_bundle_revisions.py
+++ b/deploy/hosted/controller/tests/test_bundle_revisions.py
@@ -314,7 +314,7 @@ def test_a_bundle_for_a_superseded_revision_is_not_recorded(tmp_path):
def test_sleep_wake_refreshes_the_bundle_before_the_instance_starts(tmp_path):
store, core, gateway = harness(tmp_path)
cloud = FakeCloud(store)
- reconciler = Reconciler(store, cloud)
+ reconciler = Reconciler(store, cloud, instance_type=None)
def poll() -> None:
gateway.sync_machines(gateway.machines())
@@ -381,7 +381,7 @@ def test_sleep_wake_refreshes_the_bundle_before_the_instance_starts(tmp_path):
def test_an_enrolled_machine_wakes_on_the_user_data_it_enrolled_with(tmp_path):
store, core, gateway = harness(tmp_path)
cloud = FakeCloud(store)
- reconciler = Reconciler(store, cloud)
+ reconciler = Reconciler(store, cloud, instance_type=None)
def poll() -> None:
gateway.sync_machines(gateway.machines())
@@ -421,7 +421,7 @@ def test_an_enrolled_machine_wakes_on_the_user_data_it_enrolled_with(tmp_path):
def running_machine(tmp_path):
store, core, gateway = harness(tmp_path)
cloud = FakeCloud(store)
- reconciler = Reconciler(store, cloud)
+ reconciler = Reconciler(store, cloud, instance_type=None)
def poll() -> None:
gateway.sync_machines(gateway.machines())
diff --git a/deploy/hosted/controller/tests/test_cli.py b/deploy/hosted/controller/tests/test_cli.py
index d5327f256..8033db54d 100644
--- a/deploy/hosted/controller/tests/test_cli.py
+++ b/deploy/hosted/controller/tests/test_cli.py
@@ -176,6 +176,62 @@ def test_upgrade_moves_the_machine_onto_the_new_image(capsys, config_path):
assert machine.instance_bundle is None
+def test_upgrade_moves_the_machine_onto_the_gateway_instance_type(tmp_path, capsys, config_path):
+ raw = json.loads(config_path.read_text())
+ raw["allowed_instance_types"] = ["m6i.large", "r7i.2xlarge"]
+ config_path.write_text(json.dumps(raw))
+ _, created = run(capsys, config_path, "create", "--instance-type", "m6i.large")
+ stopped_with_instance(config_path, created["machine_id"])
+ gateway_path = tmp_path / "gateway.json"
+ gateway_path.write_text(
+ json.dumps(
+ {
+ "origin": "https://switch.example.test",
+ "token": "SYNTHETIC-CONTROLLER-CREDENTIAL-0123456789",
+ "instance_type": "r7i.2xlarge",
+ }
+ )
+ )
+ with (
+ patch("switch_hosted_controller.cli.boto3"),
+ patch("switch_hosted_controller.cli.Ec2Cloud") as cloud,
+ ):
+ cloud.return_value.get_instance.return_value = {"State": {"Name": "terminated"}}
+ cloud.return_value.get_volume.return_value = {"State": "available", "Attachments": []}
+ code, upgraded = run(
+ capsys,
+ config_path,
+ "--gateway-config",
+ str(gateway_path),
+ "upgrade",
+ created["machine_id"],
+ "--confirm-instance-id",
+ "i-0123456789abcdef0",
+ )
+ assert code == 0
+ assert upgraded["instance_type"] == "r7i.2xlarge"
+ assert upgraded["instance_id"] is None
+
+
+def test_a_database_bound_to_the_superseded_fingerprint_still_opens(capsys, config_path):
+ config = ControllerConfig.load(config_path)
+ store = MachineStore(config.state_db_path, config.superseded_fingerprint())
+ machine_id = str(uuid.uuid4())
+ store.insert(
+ machine_id=machine_id,
+ core_revision=0,
+ instance_type="m6i.large",
+ image_id=config.image_id,
+ max_machines=config.max_machines,
+ )
+ store.close()
+
+ code, status = run(capsys, config_path, "status", machine_id)
+
+ assert code == 0
+ assert status["machine_id"] == machine_id
+
+
class OneIteration:
def __init__(self):
self.checks = 0
diff --git a/deploy/hosted/controller/tests/test_controller.py b/deploy/hosted/controller/tests/test_controller.py
index 9379df3b5..586f50966 100644
--- a/deploy/hosted/controller/tests/test_controller.py
+++ b/deploy/hosted/controller/tests/test_controller.py
@@ -19,6 +19,7 @@ from switch_hosted_controller.store import (
CapacityError,
MachineStore,
StoreError,
+ rebind_superseded_fingerprint,
)
MACHINE_ID = "3f1c2b4a-0000-4000-8000-000000000001"
@@ -292,7 +293,7 @@ def test_uncertain_launch_then_stop_waits_for_late_instance(tmp_path: Path):
{"InstanceIds": [worker["InstanceId"]]},
)
- reconciler = Reconciler(store, cloud)
+ reconciler = Reconciler(store, cloud, instance_type=None)
assert (
reconciler.reconcile(machine.machine_id).observed_state is ObservedState.NEEDS_ATTENTION
)
@@ -411,7 +412,9 @@ def test_an_instance_with_a_stale_bundle_makes_way_for_one_launched_with_the_new
{"TerminatingInstances": []},
{"InstanceIds": [machine.instance_id]},
)
- result = Reconciler(store, Ec2Cloud(client, cfg)).reconcile(machine.machine_id)
+ result = Reconciler(store, Ec2Cloud(client, cfg), instance_type=None).reconcile(
+ machine.machine_id
+ )
stubber.assert_no_pending_responses()
assert result.observed_state is ObservedState.PROVISIONING
assert result.instance_terminate_issued
@@ -438,7 +441,9 @@ def test_starting_an_instance_with_the_current_bundle_only_starts_it(tmp_path: P
stubber.add_response(
"start_instances", {"StartingInstances": []}, {"InstanceIds": [machine.instance_id]}
)
- result = Reconciler(store, Ec2Cloud(client, cfg)).reconcile(machine.machine_id)
+ result = Reconciler(store, Ec2Cloud(client, cfg), instance_type=None).reconcile(
+ machine.machine_id
+ )
stubber.assert_no_pending_responses()
assert result.observed_state is ObservedState.PROVISIONING
assert result.instance_bundle == REVISION_2_BUNDLE
@@ -607,7 +612,9 @@ def test_starting_an_enrolled_instance_keeps_the_user_data_it_enrolled_with(tmp_
stubber.add_response(
"start_instances", {"StartingInstances": []}, {"InstanceIds": [machine.instance_id]}
)
- result = Reconciler(store, Ec2Cloud(client, cfg)).reconcile(machine.machine_id)
+ result = Reconciler(store, Ec2Cloud(client, cfg), instance_type=None).reconcile(
+ machine.machine_id
+ )
stubber.assert_no_pending_responses()
assert result.instance_bundle == REVISION_1_BUNDLE
store.close()
@@ -807,7 +814,7 @@ def test_slot_era_database_with_a_live_machine_is_refused(tmp_path, row):
connection.close()
-def test_capacity_and_image_can_change_but_placement_cannot(tmp_path):
+def test_capacity_image_and_instance_types_can_change_but_placement_cannot(tmp_path):
cfg = config(tmp_path)
MachineStore(cfg.state_db_path, cfg.fingerprint()).close()
expanded = config(tmp_path, max_machines=2)
@@ -815,23 +822,72 @@ def test_capacity_and_image_can_change_but_placement_cannot(tmp_path):
current.close()
new_image = replace(expanded, image_id="ami-11111111111111111")
MachineStore(new_image.state_db_path, new_image.fingerprint()).close()
+ new_types = replace(new_image, allowed_instance_types=frozenset({"r7i.2xlarge"}))
+ MachineStore(new_types.state_db_path, new_types.fingerprint()).close()
incompatible = replace(expanded, subnet_id="subnet-11111111111111111")
with pytest.raises(StoreError, match="immutable"):
MachineStore(incompatible.state_db_path, incompatible.fingerprint())
+def test_a_database_bound_to_the_superseded_fingerprint_is_rebound(tmp_path):
+ cfg = config(tmp_path)
+ MachineStore(cfg.state_db_path, cfg.superseded_fingerprint()).close()
+ with pytest.raises(StoreError, match="immutable"):
+ MachineStore(cfg.state_db_path, cfg.fingerprint())
+ rebind_superseded_fingerprint(
+ cfg.state_db_path, cfg.superseded_fingerprint(), cfg.fingerprint()
+ )
+ MachineStore(cfg.state_db_path, cfg.fingerprint()).close()
+
+
+def test_rebinding_leaves_another_placement_refused(tmp_path):
+ cfg = config(tmp_path)
+ other = replace(cfg, subnet_id="subnet-11111111111111111")
+ MachineStore(cfg.state_db_path, other.superseded_fingerprint()).close()
+ rebind_superseded_fingerprint(
+ cfg.state_db_path, cfg.superseded_fingerprint(), cfg.fingerprint()
+ )
+ with pytest.raises(StoreError, match="immutable"):
+ MachineStore(cfg.state_db_path, cfg.fingerprint())
+
+
+def test_rebinding_a_missing_database_creates_nothing(tmp_path):
+ cfg = config(tmp_path)
+ rebind_superseded_fingerprint(
+ cfg.state_db_path, cfg.superseded_fingerprint(), cfg.fingerprint()
+ )
+ assert not cfg.state_db_path.exists()
+
+
+def test_an_upgrade_can_change_only_the_instance_type(tmp_path):
+ cfg = config(tmp_path)
+ store, machine = store_and_machine(cfg)
+ store.record_volume(machine.machine_id, "vol-0123456789abcdef0", cfg.availability_zone)
+ machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0", "bundle-1")
+ machine = store.set_desired(machine.machine_id, DesiredState.STOPPED, None)
+ machine = store.mark_instance_terminal_observed(machine.machine_id, machine.instance_id)
+ with pytest.raises(StoreError, match="already uses this image and instance type"):
+ store.upgrade_terminated(machine, machine.image_id, machine.instance_type)
+ upgraded = store.upgrade_terminated(machine, machine.image_id, "r7i.2xlarge")
+ assert upgraded.instance_type == "r7i.2xlarge"
+ assert upgraded.image_id == machine.image_id
+ assert upgraded.instance_id is None
+ assert upgraded.data_volume_id == machine.data_volume_id
+ store.close()
+
+
def test_image_upgrade_requires_stopped_terminal_claim_and_preserves_disk(tmp_path):
cfg = config(tmp_path)
store, machine = store_and_machine(cfg)
store.record_volume(machine.machine_id, "vol-0123456789abcdef0", cfg.availability_zone)
machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0", "bundle-1")
with pytest.raises(StoreError, match="stopped"):
- store.upgrade_terminated(machine, "ami-11111111111111111")
+ store.upgrade_terminated(machine, "ami-11111111111111111", machine.instance_type)
machine = store.set_desired(machine.machine_id, DesiredState.STOPPED, None)
with pytest.raises(StoreError, match="terminated"):
- store.upgrade_terminated(machine, "ami-11111111111111111")
+ store.upgrade_terminated(machine, "ami-11111111111111111", machine.instance_type)
machine = store.mark_instance_terminal_observed(machine.machine_id, machine.instance_id)
- upgraded = store.upgrade_terminated(machine, "ami-11111111111111111")
+ upgraded = store.upgrade_terminated(machine, "ami-11111111111111111", machine.instance_type)
assert upgraded.instance_id is None
assert upgraded.previous_instance_id == machine.instance_id
assert upgraded.data_volume_id == machine.data_volume_id
@@ -839,7 +895,7 @@ def test_image_upgrade_requires_stopped_terminal_claim_and_preserves_disk(tmp_pa
assert upgraded.image_id != machine.image_id
assert upgraded.instance_bundle is None
with pytest.raises(StoreError, match="changed"):
- store.upgrade_terminated(machine, "ami-11111111111111111")
+ store.upgrade_terminated(machine, "ami-11111111111111111", machine.instance_type)
store.close()
@@ -868,7 +924,9 @@ def test_terminating_worker_waits_without_profile_or_replacement(tmp_path, desir
{"Reservations": [{"Instances": [worker]}]},
{"InstanceIds": [machine.instance_id]},
)
- result = Reconciler(store, Ec2Cloud(client, cfg)).reconcile(machine.machine_id)
+ result = Reconciler(store, Ec2Cloud(client, cfg), instance_type=None).reconcile(
+ machine.machine_id
+ )
expected = (
ObservedState.PROVISIONING
if desired is DesiredState.RUNNING
diff --git a/deploy/hosted/controller/tests/test_lifecycle_fencing.py b/deploy/hosted/controller/tests/test_lifecycle_fencing.py
index 57ad661b6..c2f56e32d 100644
--- a/deploy/hosted/controller/tests/test_lifecycle_fencing.py
+++ b/deploy/hosted/controller/tests/test_lifecycle_fencing.py
@@ -105,7 +105,7 @@ def test_terminated_then_not_found_remains_safely_stopped_and_deletable(
{"Reservations": [{"Instances": [terminal_instance(cloud, claim)]}]},
{"InstanceIds": [machine.instance_id]},
)
- first = Reconciler(store, cloud).reconcile(machine.machine_id)
+ first = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)
assert first.instance_terminal_observed
assert first.observed_state is ObservedState.STOPPED
@@ -115,7 +115,7 @@ def test_terminated_then_not_found_remains_safely_stopped_and_deletable(
service_error_code="InvalidInstanceID.NotFound",
expected_params={"InstanceIds": [machine.instance_id]},
)
- second = Reconciler(store, cloud).reconcile(machine.machine_id)
+ second = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)
assert second.observed_state is ObservedState.STOPPED
assert second.observed_revision == second.desired_revision
assert second.observed_operation_id == second.operation_id
@@ -175,7 +175,7 @@ def test_start_callback_cannot_publish_stale_success_or_error(tmp_path: Path, st
store.set_observed(machine, ObservedState.STOPPED)
cloud = StartRaceCloud(cfg.state_db_path, cfg.fingerprint(), fail=start_fails)
- Reconciler(store, cloud).reconcile_all()
+ Reconciler(store, cloud, instance_type=None).reconcile_all()
current = store.get(machine.machine_id)
assert cloud.start_calls == 1
@@ -243,7 +243,7 @@ def test_retained_stops_running_before_termination_and_keeps_volume(
machine = record_compute(store, machine, cfg.availability_zone)
store.set_desired(machine.machine_id, DesiredState.RETAINED, None)
cloud = DeletionCloud(["running", "stopped", "terminated"])
- reconciler = Reconciler(store, cloud)
+ reconciler = Reconciler(store, cloud, instance_type=None)
assert reconciler.reconcile(machine.machine_id).observed_state is ObservedState.STOPPING
assert cloud.calls == ["stop-instance"]
@@ -269,7 +269,7 @@ def test_retained_with_a_missing_volume_needs_attention(tmp_path: Path):
store.set_desired(machine.machine_id, DesiredState.RETAINED, None)
cloud = DeletionCloud(["terminated"])
cloud.volume_exists = False
- result = Reconciler(store, cloud).reconcile(machine.machine_id)
+ result = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)
assert result.observed_state is ObservedState.NEEDS_ATTENTION
assert result.error == "recorded data volume cannot be found"
store.close()
@@ -302,12 +302,14 @@ def test_running_after_retained_launches_the_next_instance_on_the_kept_volume(tm
store, machine = store_and_machine(cfg)
first = record_compute(store, machine, cfg.availability_zone)
store.set_desired(machine.machine_id, DesiredState.RETAINED, None)
- Reconciler(store, DeletionCloud(["terminated"])).reconcile(machine.machine_id)
+ Reconciler(store, DeletionCloud(["terminated"]), instance_type=None).reconcile(
+ machine.machine_id
+ )
bundled = with_bundle(store, machine.machine_id, 2)
store.set_desired(machine.machine_id, DesiredState.RUNNING, None)
cloud = LaunchCloud()
- relaunched = Reconciler(store, cloud).reconcile(machine.machine_id)
+ relaunched = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)
assert relaunched.instance_id == "i-0000000000000000b"
assert relaunched.data_volume_id == first.data_volume_id
@@ -329,12 +331,14 @@ def test_a_new_instance_moves_onto_the_configured_image(tmp_path: Path):
store, machine = store_and_machine(cfg)
first = record_compute(store, machine, cfg.availability_zone)
store.set_desired(machine.machine_id, DesiredState.RETAINED, None)
- Reconciler(store, DeletionCloud(["terminated"])).reconcile(machine.machine_id)
+ Reconciler(store, DeletionCloud(["terminated"]), instance_type=None).reconcile(
+ machine.machine_id
+ )
with_bundle(store, machine.machine_id, 2)
store.set_desired(machine.machine_id, DesiredState.RUNNING, None)
cloud = LaunchCloud(image_id="ami-22222222222222222")
- relaunched = Reconciler(store, cloud).reconcile(machine.machine_id)
+ relaunched = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)
[launched] = cloud.launched
assert first.image_id == cfg.image_id
@@ -343,13 +347,40 @@ def test_a_new_instance_moves_onto_the_configured_image(tmp_path: Path):
store.close()
-def test_the_image_of_an_issued_launch_is_not_changed(tmp_path: Path):
+ "configured, expected", [("r7i.2xlarge", "r7i.2xlarge"), (None, "m6i.large")]
+)
+def test_a_new_instance_moves_onto_the_configured_instance_type(
+ tmp_path: Path, configured, expected
+):
+ cfg = config(tmp_path)
+ store, machine = store_and_machine(cfg)
+ first = record_compute(store, machine, cfg.availability_zone)
+ store.set_desired(machine.machine_id, DesiredState.RETAINED, None)
+ Reconciler(store, DeletionCloud(["terminated"]), instance_type=None).reconcile(
+ machine.machine_id
+ )
+ with_bundle(store, machine.machine_id, 2)
+ store.set_desired(machine.machine_id, DesiredState.RUNNING, None)
+ cloud = LaunchCloud()
+
+ relaunched = Reconciler(store, cloud, instance_type=configured).reconcile(machine.machine_id)
+
+ [launched] = cloud.launched
+ assert first.instance_type == "m6i.large"
+ assert launched.instance_type == relaunched.instance_type == expected
+ assert relaunched.data_volume_id == first.data_volume_id
+ store.close()
+
+
+def test_the_image_and_type_of_an_issued_launch_are_not_changed(tmp_path: Path):
cfg = config(tmp_path)
store, machine = store_and_machine(cfg)
machine = store.record_volume(machine.machine_id, "vol-0123456789abcdef0", "us-east-1a")
machine = store.mark_instance_launch_intent(machine)
machine = store.mark_instance_launch_issued(machine, datetime.now(UTC))
- assert store.use_image(machine, "ami-22222222222222222").image_id == cfg.image_id
+ unchanged = store.use_launch_spec(machine, "ami-22222222222222222", "r7i.2xlarge")
+ assert (unchanged.image_id, unchanged.instance_type) == (cfg.image_id, "m6i.large")
store.close()
@@ -360,7 +391,7 @@ def test_deleted_keeps_the_volume_until_retain_until(tmp_path: Path, monkeypatch
retain_until = datetime(2026, 1, 8, tzinfo=UTC)
accept_delete(store, machine, retain_until)
cloud = DeletionCloud(["terminated", None, None, None])
- reconciler = Reconciler(store, cloud)
+ reconciler = Reconciler(store, cloud, instance_type=None)
monkeypatch.setattr(reconciler_module, "utcnow", lambda: retain_until - timedelta(seconds=1))
assert reconciler.reconcile(machine.machine_id).observed_state is ObservedState.DELETING
@@ -384,7 +415,7 @@ def test_accepted_delete_waits_for_pending_and_deletes_volume_only_after_termina
machine = record_compute(store, machine, cfg.availability_zone)
accept_delete(store, machine, None)
cloud = DeletionCloud(["pending", "stopped", "terminated", None])
- reconciler = Reconciler(store, cloud)
+ reconciler = Reconciler(store, cloud, instance_type=None)
assert reconciler.reconcile(machine.machine_id).observed_state is ObservedState.DELETING
assert cloud.calls == []
@@ -453,7 +484,7 @@ def test_deletion_completes_after_terminated_instance_is_purged(tmp_path: Path):
{"VolumeIds": [machine.data_volume_id]},
)
stubber.add_response("delete_volume", {}, {"VolumeId": machine.data_volume_id})
- deleting = Reconciler(store, cloud).reconcile(machine.machine_id)
+ deleting = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)
stubber.add_response(
"describe_instances", {"Reservations": []}, {"InstanceIds": [machine.instance_id]}
)
@@ -462,14 +493,14 @@ def test_deletion_completes_after_terminated_instance_is_purged(tmp_path: Path):
service_error_code="InvalidVolume.NotFound",
expected_params={"VolumeIds": [machine.data_volume_id]},
)
- deleted = Reconciler(store, cloud).reconcile(machine.machine_id)
+ deleted = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)
stubber.assert_no_pending_responses()
assert deleting.observed_state is ObservedState.DELETING
assert deleted.observed_state is ObservedState.DELETED
assert deleted.error is None
with Stubber(client):
- again = Reconciler(store, cloud).reconcile(machine.machine_id)
+ again = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)
assert again.observed_state is ObservedState.DELETED
store.close()
@@ -483,7 +514,7 @@ def test_purged_instance_without_terminal_evidence_needs_attention(tmp_path: Pat
stubber.add_response(
"describe_instances", {"Reservations": []}, {"InstanceIds": [machine.instance_id]}
)
- result = Reconciler(store, cloud).reconcile(machine.machine_id)
+ result = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)
assert result.observed_state is ObservedState.NEEDS_ATTENTION
assert result.error == "recorded instance state is unknown during deletion"
store.close()
@@ -496,7 +527,7 @@ def test_completed_deletion_is_not_reconciled_again(tmp_path: Path):
accept_delete(store, machine, None)
cloud = DeletionCloud(["terminated"])
cloud.volume_exists = False
- reconciler = Reconciler(store, cloud)
+ reconciler = Reconciler(store, cloud, instance_type=None)
assert reconciler.reconcile(machine.machine_id).observed_state is ObservedState.DELETED
calls = list(cloud.calls)
diff --git a/deploy/hosted/controller/tests/test_retries.py b/deploy/hosted/controller/tests/test_retries.py
index 2963f6870..853d811ec 100644
--- a/deploy/hosted/controller/tests/test_retries.py
+++ b/deploy/hosted/controller/tests/test_retries.py
@@ -29,7 +29,7 @@ def test_create_volume_error_is_recovered_by_tag_discovery(tmp_path: Path):
with Stubber(client) as stubber:
stubber.add_response("describe_volumes", {"Volumes": []}, {"Filters": filters})
stubber.add_client_error("create_volume", service_error_code="RequestTimeout")
- Reconciler(store, cloud).reconcile_all()
+ Reconciler(store, cloud, instance_type=None).reconcile_all()
uncertain = store.get(machine.machine_id)
assert uncertain.volume_create_intent
@@ -39,7 +39,7 @@ def test_create_volume_error_is_recovered_by_tag_discovery(tmp_path: Path):
owned_volume = volume(cfg, uncertain)
with Stubber(client) as stubber:
stubber.add_response("describe_volumes", {"Volumes": [owned_volume]}, {"Filters": filters})
- recovered = Reconciler(store, cloud).reconcile(machine.machine_id)
+ recovered = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)
assert recovered.data_volume_id == owned_volume["VolumeId"]
store.close()
@@ -58,7 +58,7 @@ def test_stop_cancels_queued_launch_before_any_api_write(tmp_path: Path):
{"Reservations": []},
{"Filters": cloud._resource_filters(machine, "worker")},
)
- stopped = Reconciler(store, cloud).reconcile(machine.machine_id)
+ stopped = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)
assert stopped.observed_state is ObservedState.STOPPED
assert not stopped.instance_launch_intent
assert not stopped.instance_launch_issued
@@ -101,7 +101,7 @@ def ready_to_launch(tmp_path: Path):
def test_definitely_rejected_launch_is_forgotten(tmp_path: Path, code, status, cleared):
_, store, machine, cloud = ready_to_launch(tmp_path)
cloud.run_instance.side_effect = client_error(code, status, "RunInstances")
- Reconciler(store, cloud).reconcile_all()
+ Reconciler(store, cloud, instance_type=None).reconcile_all()
failed = store.get(machine.machine_id)
assert failed.observed_state is ObservedState.NEEDS_ATTENTION
@@ -110,7 +110,7 @@ def test_definitely_rejected_launch_is_forgotten(tmp_path: Path, code, status, c
store.set_desired(machine.machine_id, DesiredState.RETAINED, None)
cloud.run_instance.reset_mock()
- retained = Reconciler(store, cloud).reconcile(machine.machine_id)
+ retained = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)
expected = ObservedState.RETAINED if cleared else ObservedState.STOPPING
assert retained.observed_state is expected
assert retained.instance_launch_issued is not cleared
@@ -121,12 +121,12 @@ def test_definitely_rejected_launch_is_forgotten(tmp_path: Path, code, status, c
def test_ambiguous_launch_is_replayed_with_the_same_sequence(tmp_path: Path):
_, store, machine, cloud = ready_to_launch(tmp_path)
cloud.run_instance.side_effect = client_error("InternalError", 500, "RunInstances")
- Reconciler(store, cloud).reconcile_all()
+ Reconciler(store, cloud, instance_type=None).reconcile_all()
assert store.get(machine.machine_id).instance_launch_issued
cloud.run_instance.side_effect = None
cloud.run_instance.return_value = "i-0123456789abcdef0"
- launched = Reconciler(store, cloud).reconcile(machine.machine_id)
+ launched = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)
assert launched.instance_id == "i-0123456789abcdef0"
assert launched.instance_seq == 0
cloud.validate_image.assert_called_once()
@@ -149,7 +149,7 @@ def test_recent_unresolved_launch_is_not_forgotten(tmp_path: Path, desired):
store, machine, cloud = unresolved_launch(tmp_path, desired, timedelta(minutes=1))
busy = ObservedState.STOPPING if desired is DesiredState.RETAINED else ObservedState.DELETING
- waiting = Reconciler(store, cloud).reconcile(machine.machine_id)
+ waiting = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)
assert waiting.observed_state is busy
assert waiting.instance_launch_issued
assert waiting.instance_launch_issued_at is not None
@@ -161,7 +161,7 @@ def test_recent_unresolved_launch_is_not_forgotten(tmp_path: Path, desired):
def test_old_unresolved_launch_is_forgotten(tmp_path: Path, desired):
store, machine, cloud = unresolved_launch(tmp_path, desired, timedelta(minutes=20))
- forgotten = Reconciler(store, cloud).reconcile(machine.machine_id)
+ forgotten = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)
assert not forgotten.instance_launch_issued
assert forgotten.instance_launch_issued_at is None
if desired is DesiredState.RETAINED:
@@ -179,7 +179,7 @@ def test_retention_adopts_an_instance_found_by_launch_token(tmp_path: Path):
worker = instance(cfg, machine, "terminated")
cloud.find_launched_instance.return_value = worker
- adopted = Reconciler(store, cloud).reconcile(machine.machine_id)
+ adopted = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)
assert adopted.instance_id == worker["InstanceId"]
assert adopted.instance_launch_issued
store.close()
@@ -192,14 +192,14 @@ def test_definitely_rejected_volume_create_lets_retention_finish(tmp_path: Path)
cloud.image_id = cfg.image_id
cloud.get_volume.return_value = None
cloud.create_volume.side_effect = client_error("VolumeLimitExceeded", 400, "CreateVolume")
- Reconciler(store, cloud).reconcile_all()
+ Reconciler(store, cloud, instance_type=None).reconcile_all()
failed = store.get(machine.machine_id)
assert failed.observed_state is ObservedState.NEEDS_ATTENTION
assert not failed.volume_create_issued
store.set_desired(machine.machine_id, DesiredState.RETAINED, None)
cloud.get_instance.return_value = None
- retained = Reconciler(store, cloud).reconcile(machine.machine_id)
+ retained = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)
assert retained.observed_state is ObservedState.RETAINED
assert retained.error is None
store.close()
@@ -237,13 +237,13 @@ def test_controller_termination_is_not_a_recovery(tmp_path: Path):
cfg, store, machine, cloud = launched_instance(tmp_path)
store.set_desired(machine.machine_id, DesiredState.RETAINED, None)
cloud.get_instance.return_value = instance(cfg, machine, "stopped")
- terminating = Reconciler(store, cloud).reconcile(machine.machine_id)
+ terminating = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)
assert terminating.observed_state is ObservedState.STOPPING
cloud.terminate_instance.assert_called_once()
store.set_desired(machine.machine_id, DesiredState.RUNNING, None)
cloud.get_instance.return_value = instance(cfg, machine, "terminated")
- replaced = Reconciler(store, cloud).reconcile(machine.machine_id)
+ replaced = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)
assert replaced.instance_id is None
assert replaced.instance_seq == 1
assert replaced.recovery_count == 0
@@ -252,7 +252,7 @@ def test_controller_termination_is_not_a_recovery(tmp_path: Path):
def test_recovery_limit_applies_per_operation(tmp_path: Path):
cfg, store, machine, cloud = launched_instance(tmp_path)
- reconciler = Reconciler(store, cloud)
+ reconciler = Reconciler(store, cloud, instance_type=None)
cloud.get_instance.return_value = instance(cfg, machine, "terminated")
for index in range(3):
replaced = reconciler.reconcile(machine.machine_id)