Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.
Python agent-orchestrationaiai-agentschatopsdiscordlangchainmcpmicrosoft-teamsmulti-agentopen-sourceopenaiself-hostedslack
Let a cloud machine move to another instance type and keep its disk
Petr Bauch2febc976a97b
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 2199b6e26..e676b2e38 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -111,6 +111,14 @@ version of their own to them without also giving them a release of their own.compares it only when someone signs in.#### Changed+- **A cloud machine can move to another instance type and keep its disk.**+ The hosted controller now launches every new instance of a machine as the+ instance type in its `gateway.json`, at the same moment it already moves the+ machine onto the configured image, and `upgrade --gateway-config` moves a+ stopped machine onto both. `allowed_instance_types` is no longer bound to the+ controller's state database, so it can change while machines exist; a+ database bound before this release is accepted and rebound on first start,+ as long as `allowed_instance_types` has not changed in the same deploy.- **Following the conversation is described the same way everywhere, and thechart can turn it on.** The feature added in 0.31.0 is off unless`follow_the_conversation` is among the enabled feature flagsdiff --git a/deploy/hosted/README.md b/deploy/hosted/README.mdindex a7568a28a..4720e913a 100644--- a/deploy/hosted/README.md+++ b/deploy/hosted/README.md@@ -74,9 +74,10 @@ means the server cannot run them. Keep it no higher than the hosted controller'sMount a secret with `gateway.json` for the hosted controller, holding `origin`,the matching `token` and `instance_type`, and set the chart's-`gatewaySecretName` to it. `instance_type` is used for every new machine and-must be one of `allowed_instance_types` in both Terraform and the controller-configuration; `c7i.2xlarge` is recommended. The token authorizes only the+`gatewaySecretName` to it. `instance_type` is used for every new machine, and+an existing machine moves onto it the next time it launches an instance (see+"Operator commands"). It must be one of `allowed_instance_types` in both+Terraform and the controller configuration; `c7i.2xlarge` is recommended. The token authorizes only thehosted controller routes, for every machine on the server: their lifecycleand the controllers they run, never a workspace's agents, rooms or logins. Itis not a user or agent API key.@@ -141,13 +142,18 @@ switch-hosted-controller --config /etc/switch-hosted/controller.json status <macswitch-hosted-controller --config /etc/switch-hosted/controller.json stop <machine-id>switch-hosted-controller --config /etc/switch-hosted/controller.json start <machine-id>switch-hosted-controller --config /etc/switch-hosted/controller.json delete <machine-id> --confirm-machine-id <machine-id> --retain-volume-switch-hosted-controller --config /etc/switch-hosted/controller.json upgrade <machine-id> --confirm-instance-id i-0123456789abcdef0+switch-hosted-controller --config /etc/switch-hosted/controller.json --gateway-config /etc/switch-hosted-gateway/gateway.json upgrade <machine-id> --confirm-instance-id i-0123456789abcdef0```-Every new instance of a machine launches from the configured image: a-relaunch for new user data, a recovery, or a retained machine starting again-moves it onto a new `image_id`, keeping its disk. `upgrade` does it for a-stopped machine whose instance is terminated. A queued command is not confirmation that+Every new instance of a machine launches from the configured image, as+`gateway.json`'s `instance_type`: a relaunch for new user data, a recovery, or+a retained machine starting again moves it onto a new `image_id` and instance+type, keeping its disk. `upgrade` does it for a stopped machine whose instance+is terminated; without `--gateway-config` it keeps the machine's instance type.+`allowed_instance_types` may change while machines exist: add the new type,+move the machines, then drop the old one. The other placement settings+(installation, region, zone, subnet, security groups, root device and volume+sizes) are bound to the state database and cannot change. A queued command is not confirmation thatAWS has done it. Deletion needs `--confirm-machine-id` and exactly one of`--retain-volume` or `--delete-volume`. Deleting a machine does not removesnapshots, roles, the NAT gateway or the controller's database.diff --git a/deploy/hosted/controller/src/switch_hosted_controller/cli.py b/deploy/hosted/controller/src/switch_hosted_controller/cli.pyindex c196caa42..590879efe 100644--- a/deploy/hosted/controller/src/switch_hosted_controller/cli.py+++ b/deploy/hosted/controller/src/switch_hosted_controller/cli.py@@ -22,7 +22,7 @@ from .health import check_healthfrom .lock import ControllerAlreadyRunning, ControllerLockfrom .model import DesiredState, Machinefrom .reconciler import Reconciler-from .store import MachineStore, StoreError+from .store import MachineStore, StoreError, rebind_superseded_fingerprintdef parser() -> argparse.ArgumentParser:@@ -46,7 +46,9 @@ def parser() -> argparse.ArgumentParser:cleanup.add_argument("--delete-volume", action="store_true")upgrade = subparsers.add_parser(- "upgrade", help="use the configured image after the old instance is stopped and terminated"+ "upgrade",+ help="use the configured image, and the cloud gateway's instance type with"+ " --gateway-config, after the old instance is stopped and terminated",)upgrade.add_argument("machine_id")upgrade.add_argument("--confirm-instance-id", required=True)@@ -75,7 +77,7 @@ def main(argv: list[str] | None = None) -> int:def _state_command(config: ControllerConfig, args: argparse.Namespace) -> int:- store = MachineStore(config.state_db_path, config.fingerprint())+ store = _open_store(config)try:if args.command == "create":if args.instance_type not in config.allowed_instance_types:@@ -136,24 +138,41 @@ def _state_command(config: ControllerConfig, args: argparse.Namespace) -> int:raise StoreError("the old instance must be confirmed terminated and its retained disk detached")+ instance_type = (+ GatewayConfig.load(args.gateway_config).instance_type+ if args.gateway_config+ else machine.instance_type+ )+ if instance_type not in config.allowed_instance_types:+ raise ConfigError("instance type is not in allowed_instance_types")cloud.validate_image(replace(machine, image_id=config.image_id))claim = store.mark_instance_terminal_observed(machine.machine_id, machine.instance_id)- _print_machine(store.upgrade_terminated(claim, config.image_id))+ _print_machine(store.upgrade_terminated(claim, config.image_id, instance_type))return 0raise AssertionError(f"unhandled command {args.command}")finally:store.close()+def _open_store(config: ControllerConfig) -> MachineStore:+ rebind_superseded_fingerprint(+ config.state_db_path, config.superseded_fingerprint(), config.fingerprint()+ )+ return MachineStore(config.state_db_path, config.fingerprint())++def _reconcile_command(config: ControllerConfig, command: str, gateway_path: Path | None) -> int:with ControllerLock(config.lock_path):- store = MachineStore(config.state_db_path, config.fingerprint())+ store = _open_store(config)try:ec2 = boto3.client("ec2", region_name=config.region)- reconciler = Reconciler(store, Ec2Cloud(ec2, config))- gateway = (- Gateway(GatewayConfig.load(gateway_path), config, store) if gateway_path else None+ settings = GatewayConfig.load(gateway_path) if gateway_path else None+ reconciler = Reconciler(+ store,+ Ec2Cloud(ec2, config),+ instance_type=settings.instance_type if settings else None,)+ gateway = Gateway(settings, config, store) if settings else None_touch_health()if command == "reconcile-once":try:diff --git a/deploy/hosted/controller/src/switch_hosted_controller/config.py b/deploy/hosted/controller/src/switch_hosted_controller/config.pyindex 1a3a6fcb0..78e741f39 100644--- a/deploy/hosted/controller/src/switch_hosted_controller/config.py+++ b/deploy/hosted/controller/src/switch_hosted_controller/config.py@@ -138,19 +138,33 @@ class ControllerConfig:)def fingerprint(self) -> str:- immutable = {+ """What the state database is bound to: where machines and their disks are.+ The allowed instance types are not part of it; IAM enforces them on launch,+ and a machine moves onto the configured type when it next launches."""+ return _fingerprint_of(self._placement())++ def superseded_fingerprint(self) -> str:+ """The fingerprint before it left out the allowed instance types."""+ return _fingerprint_of(+ {**self._placement(), "allowed_instance_types": sorted(self.allowed_instance_types)}+ )++ def _placement(self) -> dict[str, Any]:+ return {"installation_id": self.installation_id,"region": self.region,"availability_zone": self.availability_zone,"subnet_id": self.subnet_id,"security_group_ids": sorted(self.security_group_ids),"root_device_name": self.root_device_name,- "allowed_instance_types": sorted(self.allowed_instance_types),"root_volume_gib": self.root_volume_gib,"data_volume_gib": self.data_volume_gib,}- payload = json.dumps(immutable, separators=(",", ":"), sort_keys=True)- return hashlib.sha256(payload.encode()).hexdigest()+++def _fingerprint_of(immutable: dict[str, Any]) -> str:+ payload = json.dumps(immutable, separators=(",", ":"), sort_keys=True)+ return hashlib.sha256(payload.encode()).hexdigest()def _validated_string(raw: dict[str, Any], key: str, pattern: re.Pattern[str]) -> str:diff --git a/deploy/hosted/controller/src/switch_hosted_controller/reconciler.py b/deploy/hosted/controller/src/switch_hosted_controller/reconciler.pyindex 9894c6be1..7aaa6a44a 100644--- a/deploy/hosted/controller/src/switch_hosted_controller/reconciler.py+++ b/deploy/hosted/controller/src/switch_hosted_controller/reconciler.py@@ -18,9 +18,12 @@ LAUNCH_VISIBILITY_GRACE = timedelta(minutes=15)class Reconciler:- def __init__(self, store: MachineStore, cloud: Ec2Cloud):+ def __init__(self, store: MachineStore, cloud: Ec2Cloud, *, instance_type: str | None):+ """`instance_type` is the type every machine moves onto when it next+ launches, the cloud gateway's; None keeps each on the type it was created with."""self._store = storeself._cloud = cloud+ self._instance_type = instance_typedef reconcile_all(self) -> None:for listed in self._store.list():@@ -91,10 +94,16 @@ class Reconciler:if not machine.instance_launch_issued:if not self._unchanged(claim, DesiredState.RUNNING):return self._store.cancel_queued_instance_launch(claim)- if machine.image_id != self._cloud.image_id:- # No instance is left, and the data volume is kept: the- # one moment the machine can move onto the configured image.- machine = self._store.use_image(claim, self._cloud.image_id)+ instance_type = self._instance_type or machine.instance_type+ if (+ machine.image_id != self._cloud.image_id+ or machine.instance_type != instance_type+ ):+ # No instance is left, and the data volume is kept: the one+ # moment the machine can move onto the configured image and type.+ machine = self._store.use_launch_spec(+ claim, self._cloud.image_id, instance_type+ )if not self._same_claim(claim, machine):return machineself._cloud.validate_image(machine)diff --git a/deploy/hosted/controller/src/switch_hosted_controller/store.py b/deploy/hosted/controller/src/switch_hosted_controller/store.pyindex 4d7d9882e..c3fb8adc8 100644--- a/deploy/hosted/controller/src/switch_hosted_controller/store.py+++ b/deploy/hosted/controller/src/switch_hosted_controller/store.py@@ -31,6 +31,26 @@ class MachineNotFoundError(StoreError):pass+def rebind_superseded_fingerprint(path: Path, superseded: str, fingerprint: str) -> None:+ """Moves a state database bound to the controller's superseded fingerprint+ onto its current one, before MachineStore checks it. The two describe the+ same placement; a database bound to anything else is left for MachineStore+ to refuse."""+ if not path.exists():+ return+ connection = sqlite3.connect(path, timeout=30, isolation_level=None)+ try:+ if connection.execute(+ "SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'controller_metadata'"+ ).fetchone():+ connection.execute(+ "UPDATE controller_metadata SET value = ? WHERE key = ? AND value = ?",+ (fingerprint, "controller_fingerprint", superseded),+ )+ finally:+ connection.close()++class MachineStore:def __init__(self, path: Path, controller_fingerprint: str):path.parent.mkdir(parents=True, exist_ok=True)@@ -296,21 +316,21 @@ class MachineStore:def mark_volume_create_intent(self, claim: Machine) -> Machine:return self._mark_intent(claim, "volume_create_intent")- def use_image(self, claim: Machine, image_id: str) -> Machine:- """Launch the machine's next instance from `image_id`: only while it has- no instance and no launch is issued, so nothing launched is told apart- from its launch request by the image."""+ def use_launch_spec(self, claim: Machine, image_id: str, instance_type: str) -> Machine:+ """Launch the machine's next instance from `image_id` as `instance_type`:+ only while it has no instance and no launch is issued, so nothing+ launched is told apart from its launch request by either."""self._connection.execute("""- UPDATE machines SET image_id = ?, updated_at = CURRENT_TIMESTAMP+ UPDATE machines SET image_id = ?, instance_type = ?, updated_at = CURRENT_TIMESTAMPWHERE machine_id = ? AND desired_revision = ? AND operation_id = ?AND instance_id IS NULL AND instance_launch_issued = 0""",- (image_id, claim.machine_id, claim.desired_revision, claim.operation_id),+ (image_id, instance_type, claim.machine_id, claim.desired_revision, claim.operation_id),)return self.get(claim.machine_id)- def upgrade_terminated(self, claim: Machine, image_id: str) -> Machine:+ def upgrade_terminated(self, claim: Machine, image_id: str, instance_type: str) -> Machine:if (claim.desired_state is not DesiredState.STOPPEDor not claim.instance_id@@ -319,12 +339,12 @@ class MachineStore:raise StoreError("image upgrades require a stopped machine and confirmed terminated predecessor")- if image_id == claim.image_id:- raise StoreError("the machine already uses this image")+ if image_id == claim.image_id and instance_type == claim.instance_type:+ raise StoreError("the machine already uses this image and instance type")cursor = self._connection.execute("""UPDATE machines SET previous_instance_id = instance_id, instance_id = NULL,instance_bundle = NULL,- image_id = ?, instance_seq = instance_seq + 1,+ image_id = ?, instance_type = ?, instance_seq = instance_seq + 1,desired_revision = desired_revision + 1, operation_id = ?,instance_launch_intent = 0, instance_launch_issued = 0, instance_launch_issued_at = NULL,instance_terminate_issued = 0,@@ -333,6 +353,7 @@ class MachineStore:AND desired_state = 'stopped' AND instance_id = ? AND instance_terminal_observed = 1""",(image_id,+ instance_type,str(uuid.uuid4()),claim.machine_id,claim.desired_revision,@@ -341,7 +362,7 @@ class MachineStore:),)if cursor.rowcount != 1:- raise StoreError("machine state changed during image upgrade; refresh before retrying")+ raise StoreError("machine state changed during upgrade; refresh before retrying")return self.get(claim.machine_id)def replace_terminated(self, claim: Machine, *, unexpected: bool) -> Machine:diff --git a/deploy/hosted/controller/tests/test_bundle_revisions.py b/deploy/hosted/controller/tests/test_bundle_revisions.pyindex 7d929661d..f030e3b91 100644--- a/deploy/hosted/controller/tests/test_bundle_revisions.py+++ b/deploy/hosted/controller/tests/test_bundle_revisions.py@@ -314,7 +314,7 @@ def test_a_bundle_for_a_superseded_revision_is_not_recorded(tmp_path):def test_sleep_wake_refreshes_the_bundle_before_the_instance_starts(tmp_path):store, core, gateway = harness(tmp_path)cloud = FakeCloud(store)- reconciler = Reconciler(store, cloud)+ reconciler = Reconciler(store, cloud, instance_type=None)def poll() -> None:gateway.sync_machines(gateway.machines())@@ -381,7 +381,7 @@ def test_sleep_wake_refreshes_the_bundle_before_the_instance_starts(tmp_path):def test_an_enrolled_machine_wakes_on_the_user_data_it_enrolled_with(tmp_path):store, core, gateway = harness(tmp_path)cloud = FakeCloud(store)- reconciler = Reconciler(store, cloud)+ reconciler = Reconciler(store, cloud, instance_type=None)def poll() -> None:gateway.sync_machines(gateway.machines())@@ -421,7 +421,7 @@ def test_an_enrolled_machine_wakes_on_the_user_data_it_enrolled_with(tmp_path):def running_machine(tmp_path):store, core, gateway = harness(tmp_path)cloud = FakeCloud(store)- reconciler = Reconciler(store, cloud)+ reconciler = Reconciler(store, cloud, instance_type=None)def poll() -> None:gateway.sync_machines(gateway.machines())diff --git a/deploy/hosted/controller/tests/test_cli.py b/deploy/hosted/controller/tests/test_cli.pyindex d5327f256..8033db54d 100644--- a/deploy/hosted/controller/tests/test_cli.py+++ b/deploy/hosted/controller/tests/test_cli.py@@ -176,6 +176,62 @@ def test_upgrade_moves_the_machine_onto_the_new_image(capsys, config_path):assert machine.instance_bundle is None+def test_upgrade_moves_the_machine_onto_the_gateway_instance_type(tmp_path, capsys, config_path):+ raw = json.loads(config_path.read_text())+ raw["allowed_instance_types"] = ["m6i.large", "r7i.2xlarge"]+ config_path.write_text(json.dumps(raw))+ _, created = run(capsys, config_path, "create", "--instance-type", "m6i.large")+ stopped_with_instance(config_path, created["machine_id"])+ gateway_path = tmp_path / "gateway.json"+ gateway_path.write_text(+ json.dumps(+ {+ "origin": "https://switch.example.test",+ "token": "SYNTHETIC-CONTROLLER-CREDENTIAL-0123456789",+ "instance_type": "r7i.2xlarge",+ }+ )+ )+ with (+ patch("switch_hosted_controller.cli.boto3"),+ patch("switch_hosted_controller.cli.Ec2Cloud") as cloud,+ ):+ cloud.return_value.get_instance.return_value = {"State": {"Name": "terminated"}}+ cloud.return_value.get_volume.return_value = {"State": "available", "Attachments": []}+ code, upgraded = run(+ capsys,+ config_path,+ "--gateway-config",+ str(gateway_path),+ "upgrade",+ created["machine_id"],+ "--confirm-instance-id",+ "i-0123456789abcdef0",+ )+ assert code == 0+ assert upgraded["instance_type"] == "r7i.2xlarge"+ assert upgraded["instance_id"] is None+++def test_a_database_bound_to_the_superseded_fingerprint_still_opens(capsys, config_path):+ config = ControllerConfig.load(config_path)+ store = MachineStore(config.state_db_path, config.superseded_fingerprint())+ machine_id = str(uuid.uuid4())+ store.insert(+ machine_id=machine_id,+ core_revision=0,+ instance_type="m6i.large",+ image_id=config.image_id,+ max_machines=config.max_machines,+ )+ store.close()++ code, status = run(capsys, config_path, "status", machine_id)++ assert code == 0+ assert status["machine_id"] == machine_id++class OneIteration:def __init__(self):self.checks = 0diff --git a/deploy/hosted/controller/tests/test_controller.py b/deploy/hosted/controller/tests/test_controller.pyindex 9379df3b5..586f50966 100644--- a/deploy/hosted/controller/tests/test_controller.py+++ b/deploy/hosted/controller/tests/test_controller.py@@ -19,6 +19,7 @@ from switch_hosted_controller.store import (CapacityError,MachineStore,StoreError,+ rebind_superseded_fingerprint,)MACHINE_ID = "3f1c2b4a-0000-4000-8000-000000000001"@@ -292,7 +293,7 @@ def test_uncertain_launch_then_stop_waits_for_late_instance(tmp_path: Path):{"InstanceIds": [worker["InstanceId"]]},)- reconciler = Reconciler(store, cloud)+ reconciler = Reconciler(store, cloud, instance_type=None)assert (reconciler.reconcile(machine.machine_id).observed_state is ObservedState.NEEDS_ATTENTION)@@ -411,7 +412,9 @@ def test_an_instance_with_a_stale_bundle_makes_way_for_one_launched_with_the_new{"TerminatingInstances": []},{"InstanceIds": [machine.instance_id]},)- result = Reconciler(store, Ec2Cloud(client, cfg)).reconcile(machine.machine_id)+ result = Reconciler(store, Ec2Cloud(client, cfg), instance_type=None).reconcile(+ machine.machine_id+ )stubber.assert_no_pending_responses()assert result.observed_state is ObservedState.PROVISIONINGassert result.instance_terminate_issued@@ -438,7 +441,9 @@ def test_starting_an_instance_with_the_current_bundle_only_starts_it(tmp_path: Pstubber.add_response("start_instances", {"StartingInstances": []}, {"InstanceIds": [machine.instance_id]})- result = Reconciler(store, Ec2Cloud(client, cfg)).reconcile(machine.machine_id)+ result = Reconciler(store, Ec2Cloud(client, cfg), instance_type=None).reconcile(+ machine.machine_id+ )stubber.assert_no_pending_responses()assert result.observed_state is ObservedState.PROVISIONINGassert result.instance_bundle == REVISION_2_BUNDLE@@ -607,7 +612,9 @@ def test_starting_an_enrolled_instance_keeps_the_user_data_it_enrolled_with(tmp_stubber.add_response("start_instances", {"StartingInstances": []}, {"InstanceIds": [machine.instance_id]})- result = Reconciler(store, Ec2Cloud(client, cfg)).reconcile(machine.machine_id)+ result = Reconciler(store, Ec2Cloud(client, cfg), instance_type=None).reconcile(+ machine.machine_id+ )stubber.assert_no_pending_responses()assert result.instance_bundle == REVISION_1_BUNDLEstore.close()@@ -807,7 +814,7 @@ def test_slot_era_database_with_a_live_machine_is_refused(tmp_path, row):connection.close()-def test_capacity_and_image_can_change_but_placement_cannot(tmp_path):+def test_capacity_image_and_instance_types_can_change_but_placement_cannot(tmp_path):cfg = config(tmp_path)MachineStore(cfg.state_db_path, cfg.fingerprint()).close()expanded = config(tmp_path, max_machines=2)@@ -815,23 +822,72 @@ def test_capacity_and_image_can_change_but_placement_cannot(tmp_path):current.close()new_image = replace(expanded, image_id="ami-11111111111111111")MachineStore(new_image.state_db_path, new_image.fingerprint()).close()+ new_types = replace(new_image, allowed_instance_types=frozenset({"r7i.2xlarge"}))+ MachineStore(new_types.state_db_path, new_types.fingerprint()).close()incompatible = replace(expanded, subnet_id="subnet-11111111111111111")with pytest.raises(StoreError, match="immutable"):MachineStore(incompatible.state_db_path, incompatible.fingerprint())+def test_a_database_bound_to_the_superseded_fingerprint_is_rebound(tmp_path):+ cfg = config(tmp_path)+ MachineStore(cfg.state_db_path, cfg.superseded_fingerprint()).close()+ with pytest.raises(StoreError, match="immutable"):+ MachineStore(cfg.state_db_path, cfg.fingerprint())+ rebind_superseded_fingerprint(+ cfg.state_db_path, cfg.superseded_fingerprint(), cfg.fingerprint()+ )+ MachineStore(cfg.state_db_path, cfg.fingerprint()).close()+++def test_rebinding_leaves_another_placement_refused(tmp_path):+ cfg = config(tmp_path)+ other = replace(cfg, subnet_id="subnet-11111111111111111")+ MachineStore(cfg.state_db_path, other.superseded_fingerprint()).close()+ rebind_superseded_fingerprint(+ cfg.state_db_path, cfg.superseded_fingerprint(), cfg.fingerprint()+ )+ with pytest.raises(StoreError, match="immutable"):+ MachineStore(cfg.state_db_path, cfg.fingerprint())+++def test_rebinding_a_missing_database_creates_nothing(tmp_path):+ cfg = config(tmp_path)+ rebind_superseded_fingerprint(+ cfg.state_db_path, cfg.superseded_fingerprint(), cfg.fingerprint()+ )+ assert not cfg.state_db_path.exists()+++def test_an_upgrade_can_change_only_the_instance_type(tmp_path):+ cfg = config(tmp_path)+ store, machine = store_and_machine(cfg)+ store.record_volume(machine.machine_id, "vol-0123456789abcdef0", cfg.availability_zone)+ machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0", "bundle-1")+ machine = store.set_desired(machine.machine_id, DesiredState.STOPPED, None)+ machine = store.mark_instance_terminal_observed(machine.machine_id, machine.instance_id)+ with pytest.raises(StoreError, match="already uses this image and instance type"):+ store.upgrade_terminated(machine, machine.image_id, machine.instance_type)+ upgraded = store.upgrade_terminated(machine, machine.image_id, "r7i.2xlarge")+ assert upgraded.instance_type == "r7i.2xlarge"+ assert upgraded.image_id == machine.image_id+ assert upgraded.instance_id is None+ assert upgraded.data_volume_id == machine.data_volume_id+ store.close()++def test_image_upgrade_requires_stopped_terminal_claim_and_preserves_disk(tmp_path):cfg = config(tmp_path)store, machine = store_and_machine(cfg)store.record_volume(machine.machine_id, "vol-0123456789abcdef0", cfg.availability_zone)machine = store.record_instance(machine.machine_id, "i-0123456789abcdef0", "bundle-1")with pytest.raises(StoreError, match="stopped"):- store.upgrade_terminated(machine, "ami-11111111111111111")+ store.upgrade_terminated(machine, "ami-11111111111111111", machine.instance_type)machine = store.set_desired(machine.machine_id, DesiredState.STOPPED, None)with pytest.raises(StoreError, match="terminated"):- store.upgrade_terminated(machine, "ami-11111111111111111")+ store.upgrade_terminated(machine, "ami-11111111111111111", machine.instance_type)machine = store.mark_instance_terminal_observed(machine.machine_id, machine.instance_id)- upgraded = store.upgrade_terminated(machine, "ami-11111111111111111")+ upgraded = store.upgrade_terminated(machine, "ami-11111111111111111", machine.instance_type)assert upgraded.instance_id is Noneassert upgraded.previous_instance_id == machine.instance_idassert upgraded.data_volume_id == machine.data_volume_id@@ -839,7 +895,7 @@ def test_image_upgrade_requires_stopped_terminal_claim_and_preserves_disk(tmp_paassert upgraded.image_id != machine.image_idassert upgraded.instance_bundle is Nonewith pytest.raises(StoreError, match="changed"):- store.upgrade_terminated(machine, "ami-11111111111111111")+ store.upgrade_terminated(machine, "ami-11111111111111111", machine.instance_type)store.close()@@ -868,7 +924,9 @@ def test_terminating_worker_waits_without_profile_or_replacement(tmp_path, desir{"Reservations": [{"Instances": [worker]}]},{"InstanceIds": [machine.instance_id]},)- result = Reconciler(store, Ec2Cloud(client, cfg)).reconcile(machine.machine_id)+ result = Reconciler(store, Ec2Cloud(client, cfg), instance_type=None).reconcile(+ machine.machine_id+ )expected = (ObservedState.PROVISIONINGif desired is DesiredState.RUNNINGdiff --git a/deploy/hosted/controller/tests/test_lifecycle_fencing.py b/deploy/hosted/controller/tests/test_lifecycle_fencing.pyindex 57ad661b6..c2f56e32d 100644--- a/deploy/hosted/controller/tests/test_lifecycle_fencing.py+++ b/deploy/hosted/controller/tests/test_lifecycle_fencing.py@@ -105,7 +105,7 @@ def test_terminated_then_not_found_remains_safely_stopped_and_deletable({"Reservations": [{"Instances": [terminal_instance(cloud, claim)]}]},{"InstanceIds": [machine.instance_id]},)- first = Reconciler(store, cloud).reconcile(machine.machine_id)+ first = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)assert first.instance_terminal_observedassert first.observed_state is ObservedState.STOPPED@@ -115,7 +115,7 @@ def test_terminated_then_not_found_remains_safely_stopped_and_deletable(service_error_code="InvalidInstanceID.NotFound",expected_params={"InstanceIds": [machine.instance_id]},)- second = Reconciler(store, cloud).reconcile(machine.machine_id)+ second = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)assert second.observed_state is ObservedState.STOPPEDassert second.observed_revision == second.desired_revisionassert second.observed_operation_id == second.operation_id@@ -175,7 +175,7 @@ def test_start_callback_cannot_publish_stale_success_or_error(tmp_path: Path, ststore.set_observed(machine, ObservedState.STOPPED)cloud = StartRaceCloud(cfg.state_db_path, cfg.fingerprint(), fail=start_fails)- Reconciler(store, cloud).reconcile_all()+ Reconciler(store, cloud, instance_type=None).reconcile_all()current = store.get(machine.machine_id)assert cloud.start_calls == 1@@ -243,7 +243,7 @@ def test_retained_stops_running_before_termination_and_keeps_volume(machine = record_compute(store, machine, cfg.availability_zone)store.set_desired(machine.machine_id, DesiredState.RETAINED, None)cloud = DeletionCloud(["running", "stopped", "terminated"])- reconciler = Reconciler(store, cloud)+ reconciler = Reconciler(store, cloud, instance_type=None)assert reconciler.reconcile(machine.machine_id).observed_state is ObservedState.STOPPINGassert cloud.calls == ["stop-instance"]@@ -269,7 +269,7 @@ def test_retained_with_a_missing_volume_needs_attention(tmp_path: Path):store.set_desired(machine.machine_id, DesiredState.RETAINED, None)cloud = DeletionCloud(["terminated"])cloud.volume_exists = False- result = Reconciler(store, cloud).reconcile(machine.machine_id)+ result = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)assert result.observed_state is ObservedState.NEEDS_ATTENTIONassert result.error == "recorded data volume cannot be found"store.close()@@ -302,12 +302,14 @@ def test_running_after_retained_launches_the_next_instance_on_the_kept_volume(tmstore, machine = store_and_machine(cfg)first = record_compute(store, machine, cfg.availability_zone)store.set_desired(machine.machine_id, DesiredState.RETAINED, None)- Reconciler(store, DeletionCloud(["terminated"])).reconcile(machine.machine_id)+ Reconciler(store, DeletionCloud(["terminated"]), instance_type=None).reconcile(+ machine.machine_id+ )bundled = with_bundle(store, machine.machine_id, 2)store.set_desired(machine.machine_id, DesiredState.RUNNING, None)cloud = LaunchCloud()- relaunched = Reconciler(store, cloud).reconcile(machine.machine_id)+ relaunched = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)assert relaunched.instance_id == "i-0000000000000000b"assert relaunched.data_volume_id == first.data_volume_id@@ -329,12 +331,14 @@ def test_a_new_instance_moves_onto_the_configured_image(tmp_path: Path):store, machine = store_and_machine(cfg)first = record_compute(store, machine, cfg.availability_zone)store.set_desired(machine.machine_id, DesiredState.RETAINED, None)- Reconciler(store, DeletionCloud(["terminated"])).reconcile(machine.machine_id)+ Reconciler(store, DeletionCloud(["terminated"]), instance_type=None).reconcile(+ machine.machine_id+ )with_bundle(store, machine.machine_id, 2)store.set_desired(machine.machine_id, DesiredState.RUNNING, None)cloud = LaunchCloud(image_id="ami-22222222222222222")- relaunched = Reconciler(store, cloud).reconcile(machine.machine_id)+ relaunched = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)[launched] = cloud.launchedassert first.image_id == cfg.image_id@@ -343,13 +347,40 @@ def test_a_new_instance_moves_onto_the_configured_image(tmp_path: Path):store.close()-def test_the_image_of_an_issued_launch_is_not_changed(tmp_path: Path):+ "configured, expected", [("r7i.2xlarge", "r7i.2xlarge"), (None, "m6i.large")]+)+def test_a_new_instance_moves_onto_the_configured_instance_type(+ tmp_path: Path, configured, expected+):+ cfg = config(tmp_path)+ store, machine = store_and_machine(cfg)+ first = record_compute(store, machine, cfg.availability_zone)+ store.set_desired(machine.machine_id, DesiredState.RETAINED, None)+ Reconciler(store, DeletionCloud(["terminated"]), instance_type=None).reconcile(+ machine.machine_id+ )+ with_bundle(store, machine.machine_id, 2)+ store.set_desired(machine.machine_id, DesiredState.RUNNING, None)+ cloud = LaunchCloud()++ relaunched = Reconciler(store, cloud, instance_type=configured).reconcile(machine.machine_id)++ [launched] = cloud.launched+ assert first.instance_type == "m6i.large"+ assert launched.instance_type == relaunched.instance_type == expected+ assert relaunched.data_volume_id == first.data_volume_id+ store.close()+++def test_the_image_and_type_of_an_issued_launch_are_not_changed(tmp_path: Path):cfg = config(tmp_path)store, machine = store_and_machine(cfg)machine = store.record_volume(machine.machine_id, "vol-0123456789abcdef0", "us-east-1a")machine = store.mark_instance_launch_intent(machine)machine = store.mark_instance_launch_issued(machine, datetime.now(UTC))- assert store.use_image(machine, "ami-22222222222222222").image_id == cfg.image_id+ unchanged = store.use_launch_spec(machine, "ami-22222222222222222", "r7i.2xlarge")+ assert (unchanged.image_id, unchanged.instance_type) == (cfg.image_id, "m6i.large")store.close()@@ -360,7 +391,7 @@ def test_deleted_keeps_the_volume_until_retain_until(tmp_path: Path, monkeypatchretain_until = datetime(2026, 1, 8, tzinfo=UTC)accept_delete(store, machine, retain_until)cloud = DeletionCloud(["terminated", None, None, None])- reconciler = Reconciler(store, cloud)+ reconciler = Reconciler(store, cloud, instance_type=None)monkeypatch.setattr(reconciler_module, "utcnow", lambda: retain_until - timedelta(seconds=1))assert reconciler.reconcile(machine.machine_id).observed_state is ObservedState.DELETING@@ -384,7 +415,7 @@ def test_accepted_delete_waits_for_pending_and_deletes_volume_only_after_terminamachine = record_compute(store, machine, cfg.availability_zone)accept_delete(store, machine, None)cloud = DeletionCloud(["pending", "stopped", "terminated", None])- reconciler = Reconciler(store, cloud)+ reconciler = Reconciler(store, cloud, instance_type=None)assert reconciler.reconcile(machine.machine_id).observed_state is ObservedState.DELETINGassert cloud.calls == []@@ -453,7 +484,7 @@ def test_deletion_completes_after_terminated_instance_is_purged(tmp_path: Path):{"VolumeIds": [machine.data_volume_id]},)stubber.add_response("delete_volume", {}, {"VolumeId": machine.data_volume_id})- deleting = Reconciler(store, cloud).reconcile(machine.machine_id)+ deleting = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)stubber.add_response("describe_instances", {"Reservations": []}, {"InstanceIds": [machine.instance_id]})@@ -462,14 +493,14 @@ def test_deletion_completes_after_terminated_instance_is_purged(tmp_path: Path):service_error_code="InvalidVolume.NotFound",expected_params={"VolumeIds": [machine.data_volume_id]},)- deleted = Reconciler(store, cloud).reconcile(machine.machine_id)+ deleted = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)stubber.assert_no_pending_responses()assert deleting.observed_state is ObservedState.DELETINGassert deleted.observed_state is ObservedState.DELETEDassert deleted.error is Nonewith Stubber(client):- again = Reconciler(store, cloud).reconcile(machine.machine_id)+ again = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)assert again.observed_state is ObservedState.DELETEDstore.close()@@ -483,7 +514,7 @@ def test_purged_instance_without_terminal_evidence_needs_attention(tmp_path: Patstubber.add_response("describe_instances", {"Reservations": []}, {"InstanceIds": [machine.instance_id]})- result = Reconciler(store, cloud).reconcile(machine.machine_id)+ result = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)assert result.observed_state is ObservedState.NEEDS_ATTENTIONassert result.error == "recorded instance state is unknown during deletion"store.close()@@ -496,7 +527,7 @@ def test_completed_deletion_is_not_reconciled_again(tmp_path: Path):accept_delete(store, machine, None)cloud = DeletionCloud(["terminated"])cloud.volume_exists = False- reconciler = Reconciler(store, cloud)+ reconciler = Reconciler(store, cloud, instance_type=None)assert reconciler.reconcile(machine.machine_id).observed_state is ObservedState.DELETEDcalls = list(cloud.calls)diff --git a/deploy/hosted/controller/tests/test_retries.py b/deploy/hosted/controller/tests/test_retries.pyindex 2963f6870..853d811ec 100644--- a/deploy/hosted/controller/tests/test_retries.py+++ b/deploy/hosted/controller/tests/test_retries.py@@ -29,7 +29,7 @@ def test_create_volume_error_is_recovered_by_tag_discovery(tmp_path: Path):with Stubber(client) as stubber:stubber.add_response("describe_volumes", {"Volumes": []}, {"Filters": filters})stubber.add_client_error("create_volume", service_error_code="RequestTimeout")- Reconciler(store, cloud).reconcile_all()+ Reconciler(store, cloud, instance_type=None).reconcile_all()uncertain = store.get(machine.machine_id)assert uncertain.volume_create_intent@@ -39,7 +39,7 @@ def test_create_volume_error_is_recovered_by_tag_discovery(tmp_path: Path):owned_volume = volume(cfg, uncertain)with Stubber(client) as stubber:stubber.add_response("describe_volumes", {"Volumes": [owned_volume]}, {"Filters": filters})- recovered = Reconciler(store, cloud).reconcile(machine.machine_id)+ recovered = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)assert recovered.data_volume_id == owned_volume["VolumeId"]store.close()@@ -58,7 +58,7 @@ def test_stop_cancels_queued_launch_before_any_api_write(tmp_path: Path):{"Reservations": []},{"Filters": cloud._resource_filters(machine, "worker")},)- stopped = Reconciler(store, cloud).reconcile(machine.machine_id)+ stopped = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)assert stopped.observed_state is ObservedState.STOPPEDassert not stopped.instance_launch_intentassert not stopped.instance_launch_issued@@ -101,7 +101,7 @@ def ready_to_launch(tmp_path: Path):def test_definitely_rejected_launch_is_forgotten(tmp_path: Path, code, status, cleared):_, store, machine, cloud = ready_to_launch(tmp_path)cloud.run_instance.side_effect = client_error(code, status, "RunInstances")- Reconciler(store, cloud).reconcile_all()+ Reconciler(store, cloud, instance_type=None).reconcile_all()failed = store.get(machine.machine_id)assert failed.observed_state is ObservedState.NEEDS_ATTENTION@@ -110,7 +110,7 @@ def test_definitely_rejected_launch_is_forgotten(tmp_path: Path, code, status, cstore.set_desired(machine.machine_id, DesiredState.RETAINED, None)cloud.run_instance.reset_mock()- retained = Reconciler(store, cloud).reconcile(machine.machine_id)+ retained = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)expected = ObservedState.RETAINED if cleared else ObservedState.STOPPINGassert retained.observed_state is expectedassert retained.instance_launch_issued is not cleared@@ -121,12 +121,12 @@ def test_definitely_rejected_launch_is_forgotten(tmp_path: Path, code, status, cdef test_ambiguous_launch_is_replayed_with_the_same_sequence(tmp_path: Path):_, store, machine, cloud = ready_to_launch(tmp_path)cloud.run_instance.side_effect = client_error("InternalError", 500, "RunInstances")- Reconciler(store, cloud).reconcile_all()+ Reconciler(store, cloud, instance_type=None).reconcile_all()assert store.get(machine.machine_id).instance_launch_issuedcloud.run_instance.side_effect = Nonecloud.run_instance.return_value = "i-0123456789abcdef0"- launched = Reconciler(store, cloud).reconcile(machine.machine_id)+ launched = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)assert launched.instance_id == "i-0123456789abcdef0"assert launched.instance_seq == 0cloud.validate_image.assert_called_once()@@ -149,7 +149,7 @@ def test_recent_unresolved_launch_is_not_forgotten(tmp_path: Path, desired):store, machine, cloud = unresolved_launch(tmp_path, desired, timedelta(minutes=1))busy = ObservedState.STOPPING if desired is DesiredState.RETAINED else ObservedState.DELETING- waiting = Reconciler(store, cloud).reconcile(machine.machine_id)+ waiting = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)assert waiting.observed_state is busyassert waiting.instance_launch_issuedassert waiting.instance_launch_issued_at is not None@@ -161,7 +161,7 @@ def test_recent_unresolved_launch_is_not_forgotten(tmp_path: Path, desired):def test_old_unresolved_launch_is_forgotten(tmp_path: Path, desired):store, machine, cloud = unresolved_launch(tmp_path, desired, timedelta(minutes=20))- forgotten = Reconciler(store, cloud).reconcile(machine.machine_id)+ forgotten = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)assert not forgotten.instance_launch_issuedassert forgotten.instance_launch_issued_at is Noneif desired is DesiredState.RETAINED:@@ -179,7 +179,7 @@ def test_retention_adopts_an_instance_found_by_launch_token(tmp_path: Path):worker = instance(cfg, machine, "terminated")cloud.find_launched_instance.return_value = worker- adopted = Reconciler(store, cloud).reconcile(machine.machine_id)+ adopted = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)assert adopted.instance_id == worker["InstanceId"]assert adopted.instance_launch_issuedstore.close()@@ -192,14 +192,14 @@ def test_definitely_rejected_volume_create_lets_retention_finish(tmp_path: Path)cloud.image_id = cfg.image_idcloud.get_volume.return_value = Nonecloud.create_volume.side_effect = client_error("VolumeLimitExceeded", 400, "CreateVolume")- Reconciler(store, cloud).reconcile_all()+ Reconciler(store, cloud, instance_type=None).reconcile_all()failed = store.get(machine.machine_id)assert failed.observed_state is ObservedState.NEEDS_ATTENTIONassert not failed.volume_create_issuedstore.set_desired(machine.machine_id, DesiredState.RETAINED, None)cloud.get_instance.return_value = None- retained = Reconciler(store, cloud).reconcile(machine.machine_id)+ retained = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)assert retained.observed_state is ObservedState.RETAINEDassert retained.error is Nonestore.close()@@ -237,13 +237,13 @@ def test_controller_termination_is_not_a_recovery(tmp_path: Path):cfg, store, machine, cloud = launched_instance(tmp_path)store.set_desired(machine.machine_id, DesiredState.RETAINED, None)cloud.get_instance.return_value = instance(cfg, machine, "stopped")- terminating = Reconciler(store, cloud).reconcile(machine.machine_id)+ terminating = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)assert terminating.observed_state is ObservedState.STOPPINGcloud.terminate_instance.assert_called_once()store.set_desired(machine.machine_id, DesiredState.RUNNING, None)cloud.get_instance.return_value = instance(cfg, machine, "terminated")- replaced = Reconciler(store, cloud).reconcile(machine.machine_id)+ replaced = Reconciler(store, cloud, instance_type=None).reconcile(machine.machine_id)assert replaced.instance_id is Noneassert replaced.instance_seq == 1assert replaced.recovery_count == 0@@ -252,7 +252,7 @@ def test_controller_termination_is_not_a_recovery(tmp_path: Path):def test_recovery_limit_applies_per_operation(tmp_path: Path):cfg, store, machine, cloud = launched_instance(tmp_path)- reconciler = Reconciler(store, cloud)+ reconciler = Reconciler(store, cloud, instance_type=None)cloud.get_instance.return_value = instance(cfg, machine, "terminated")for index in range(3):replaced = reconciler.reconcile(machine.machine_id)