Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

Say how a cloud machine moves to a new instance type, and flag the ones left

Petr Bauche7248a145490
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e676b2e38..751c51bdf 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -115,8 +115,11 @@ version of their own to them without also giving them a release of their own.
The hosted controller now launches every new instance of a machine as the
instance type in its `gateway.json`, at the same moment it already moves the
machine onto the configured image, and `upgrade --gateway-config` moves a
- stopped machine onto both. `allowed_instance_types` is no longer bound to the
- controller's state database, so it can change while machines exist; a
+ stopped machine onto both. Changing `instance_type` therefore resizes
+ existing machines as well as new ones; a machine that only stops and starts
+ keeps its instance and type. `allowed_instance_types` is no longer bound to
+ the controller's state database, so it can change while machines exist, and
+ `status` and `list` report `instance_type_allowed` for each machine. A
database bound before this release is accepted and rebound on first start,
as long as `allowed_instance_types` has not changed in the same deploy.
- **Following the conversation is described the same way everywhere, and the
diff --git a/deploy/hosted/README.md b/deploy/hosted/README.md
index 4720e913a..720068ff3 100644
--- a/deploy/hosted/README.md
+++ b/deploy/hosted/README.md
@@ -74,10 +74,14 @@ means the server cannot run them. Keep it no higher than the hosted controller's
Mount a secret with `gateway.json` for the hosted controller, holding `origin`,
the matching `token` and `instance_type`, and set the chart's
-`gatewaySecretName` to it. `instance_type` is used for every new machine, and
-an existing machine moves onto it the next time it launches an instance (see
-"Operator commands"). It must be one of `allowed_instance_types` in both
-Terraform and the controller configuration; `c7i.2xlarge` is recommended. The token authorizes only the
+`gatewaySecretName` to it. `instance_type` is the type of every machine, not
+only of the machines created after it is set: changing it resizes existing
+machines too, each the next time it gets a new instance (a recovery, new user
+data, a start after being retained, or `upgrade`), so it changes the cost and
+vCPU quota of the whole fleet. A machine that only stops and starts keeps its
+instance and its type. See "Operator commands" for moving a machine on purpose.
+It must be one of `allowed_instance_types` in both Terraform and the controller
+configuration; `c7i.2xlarge` is recommended. The token authorizes only the
hosted controller routes, for every machine on the server: their lifecycle
and the controllers they run, never a workspace's agents, rooms or logins. It
is not a user or agent API key.
@@ -148,10 +152,26 @@ switch-hosted-controller --config /etc/switch-hosted/controller.json --gateway-c
Every new instance of a machine launches from the configured image, as
`gateway.json`'s `instance_type`: a relaunch for new user data, a recovery, or
a retained machine starting again moves it onto a new `image_id` and instance
-type, keeping its disk. `upgrade` does it for a stopped machine whose instance
-is terminated; without `--gateway-config` it keeps the machine's instance type.
+type, keeping its disk. Stopping and starting a machine does not: it starts the
+same instance, which keeps its image and type for as long as it exists.
+
+To move a machine onto the configured image and type now:
+
+1. `stop` it, and wait for `status` to report `observed_state` `stopped`.
+2. Terminate its instance (`instance_id` in `status`) with the EC2 API or
+ console, and wait until it is terminated and the data volume is detached.
+3. `upgrade` it with `--gateway-config` and the terminated `instance_id`.
+ Without `--gateway-config`, `upgrade` keeps the machine's instance type.
+4. `start` it. Its next instance launches on the same data volume.
+
+On a server whose Core runs the machine, Core may ask for it to run again
+while it is stopped; move it when no agent needs it.
+
`allowed_instance_types` may change while machines exist: add the new type,
-move the machines, then drop the old one. The other placement settings
+set it in `gateway.json`, move every machine as above, and drop the old type
+once `list` reports `instance_type_allowed` true for each. IAM refuses to
+launch a dropped type but not to start an existing instance of one, so a
+machine left unmoved keeps running it. The other placement settings
(installation, region, zone, subnet, security groups, root device and volume
sizes) are bound to the state database and cannot change. A queued command is not confirmation that
AWS has done it. Deletion needs `--confirm-machine-id` and exactly one of
diff --git a/deploy/hosted/controller/src/switch_hosted_controller/cli.py b/deploy/hosted/controller/src/switch_hosted_controller/cli.py
index 590879efe..117b92b4f 100644
--- a/deploy/hosted/controller/src/switch_hosted_controller/cli.py
+++ b/deploy/hosted/controller/src/switch_hosted_controller/cli.py
@@ -89,11 +89,13 @@ def _state_command(config: ControllerConfig, args: argparse.Namespace) -> int:
image_id=config.image_id,
max_machines=config.max_machines,
)
- _print_machine(machine)
+ _print_machine(machine, config)
return 0
if args.command == "list":
print(
- json.dumps([_machine_output(machine) for machine in store.list()], sort_keys=True)
+ json.dumps(
+ [_machine_output(machine, config) for machine in store.list()], sort_keys=True
+ )
)
return 0
try:
@@ -102,19 +104,23 @@ def _state_command(config: ControllerConfig, args: argparse.Namespace) -> int:
raise ConfigError("machine_id must be a machine's UUID") from None
machine = store.get(machine_id)
if args.command == "status":
- _print_machine(machine)
+ _print_machine(machine, config)
return 0
if args.command == "start":
- _print_machine(store.set_desired(machine.machine_id, DesiredState.RUNNING, None))
+ _print_machine(
+ store.set_desired(machine.machine_id, DesiredState.RUNNING, None), config
+ )
return 0
if args.command == "stop":
- _print_machine(store.set_desired(machine.machine_id, DesiredState.STOPPED, None))
+ _print_machine(
+ store.set_desired(machine.machine_id, DesiredState.STOPPED, None), config
+ )
return 0
if args.command == "delete":
if args.confirm_machine_id != machine.machine_id:
raise StoreError("--confirm-machine-id must exactly match machine_id")
desired = DesiredState.DELETED if args.delete_volume else DesiredState.RETAINED
- _print_machine(store.set_desired(machine.machine_id, desired, None))
+ _print_machine(store.set_desired(machine.machine_id, desired, None), config)
return 0
if args.command == "upgrade":
if (
@@ -147,7 +153,7 @@ def _state_command(config: ControllerConfig, args: argparse.Namespace) -> int:
raise ConfigError("instance type is not in allowed_instance_types")
cloud.validate_image(replace(machine, image_id=config.image_id))
claim = store.mark_instance_terminal_observed(machine.machine_id, machine.instance_id)
- _print_machine(store.upgrade_terminated(claim, config.image_id, instance_type))
+ _print_machine(store.upgrade_terminated(claim, config.image_id, instance_type), config)
return 0
raise AssertionError(f"unhandled command {args.command}")
finally:
@@ -229,14 +235,17 @@ def _health(max_age: float) -> int:
raise ConfigError(str(error)) from error
-def _print_machine(machine: Machine) -> None:
- print(json.dumps(_machine_output(machine), sort_keys=True))
+def _print_machine(machine: Machine, config: ControllerConfig) -> None:
+ print(json.dumps(_machine_output(machine, config), sort_keys=True))
-def _machine_output(machine: Machine) -> dict[str, Any]:
+def _machine_output(machine: Machine, config: ControllerConfig) -> dict[str, Any]:
return {
"machine_id": machine.machine_id,
"instance_type": machine.instance_type,
+ # False while the machine still runs a type dropped from the list: its
+ # instance keeps it through stops and starts until it is replaced.
+ "instance_type_allowed": machine.instance_type in config.allowed_instance_types,
"desired_state": machine.desired_state.value,
"desired_revision": machine.desired_revision,
"observed_state": machine.observed_state.value,
diff --git a/deploy/hosted/controller/tests/test_cli.py b/deploy/hosted/controller/tests/test_cli.py
index 8033db54d..6910087c2 100644
--- a/deploy/hosted/controller/tests/test_cli.py
+++ b/deploy/hosted/controller/tests/test_cli.py
@@ -49,6 +49,7 @@ def test_create_status_and_delete_are_keyed_by_machine_id(capsys, config_path):
assert set(created) == {
"machine_id",
"instance_type",
+ "instance_type_allowed",
"desired_state",
"desired_revision",
"observed_state",
@@ -232,6 +233,22 @@ def test_a_database_bound_to_the_superseded_fingerprint_still_opens(capsys, conf
assert status["machine_id"] == machine_id
+def test_status_flags_a_machine_on_a_type_no_longer_allowed(capsys, config_path):
+ _, created = run(capsys, config_path, "create", "--instance-type", "m6i.large")
+ assert created["instance_type_allowed"] is True
+ raw = json.loads(config_path.read_text())
+ raw["allowed_instance_types"] = ["r7i.2xlarge"]
+ config_path.write_text(json.dumps(raw))
+
+ code, status = run(capsys, config_path, "status", created["machine_id"])
+
+ assert code == 0
+ assert status["instance_type"] == "m6i.large"
+ assert status["instance_type_allowed"] is False
+ _, listed = run(capsys, config_path, "list")
+ assert [m["instance_type_allowed"] for m in listed] == [False]
+
+
class OneIteration:
def __init__(self):
self.checks = 0