100% open-source IoT Platform - Integrate your devices, add intelligence, build custom apps
Deploying EKS load-test stacks
test/or-eks-load is a Bash helper for deploying a load1 or load2 profile. Save
your settings in one file, then use up to configure DNS access, create or
reconcile the cluster, and deploy the stack. The default deployment provides
HTTPS and MQTTS through HAProxy, with a separate namespace and storage per stack.
Run the commands below from the repository root.
Deploy and run a test
1. Prepare credentials and the Manager image
Install the tools required by the EKS deployment guide: AWS CLI, eksctl, kubectl, Helm, jq, envsubst, dig, curl, and Python 3. The helper itself uses Bash, including macOS's built-in Bash; the underlying EKS CLI uses Python for its MQTTS TLS check.
Authenticate to the cluster account using your normal AWS credential chain.
For pasted temporary credentials, export AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN, and unset AWS_PROFILE
and AWS_DEFAULT_PROFILE if previously set. Do not set a profile to an empty
string. A named cluster profile can instead be selected in the config below.
Build and push the custom Manager image using the load1 or load2 instructions. Ensure the cluster nodes can pull it, including ECR repository permissions for cross-account pulls. The supplied clusters use ARM nodes; the image examples publish both ARM64 and AMD64 images. The helper does not build images or change registry permissions.
2. Save your deployment settings
For load2:
mkdir -p .local
cp test/load2-eks/deployment.env.example .local/loadtest.env
For load1, copy test/load1-eks/deployment.env.example instead.
Edit .local/loadtest.env for your deployment. For example:
LOAD_TEST=load2
LOAD_PROFILE=xlarge
CLUSTER_NAME=loadtest
STACK_NAME=load2
AWS_REGION=eu-west-1
LOAD_HOSTNAME=load2.openremote.app
LOAD_MANAGER_REPOSITORY=134517981306.dkr.ecr.eu-west-1.amazonaws.com/openremote/manager
DNS_AWS_PROFILE=134517981306_AWSAdministratorAccess
LOAD_MANAGER_REPOSITORY includes /openremote/manager, without an image tag;
the profile selects load1 or load2. LOAD_HOSTNAME is a bare hostname,
without https:// or a path, and must be unused by another stack or cluster.
Use DNS_AWS_PROFILE for the account containing your public Route 53 zone.
Leave it empty to use the cluster credentials. If using an SSO profile,
authenticate it before deployment. Set CLUSTER_AWS_PROFILE only if you want
a specific named profile for cluster operations.
You do not normally need a zone ID, role ARN, external ID, or ownership ID.
The helper discovers the zone and configures a dedicated role for this cluster.
If multiple public zones match the same domain, it lists their IDs and stops;
add DNS_ZONE_ID=... to choose one explicitly.
The config is sourced as Bash code, so use only a file you trust. It needs
no export statements and must not contain access keys. .local/ is ignored
by Git. Helper settings come from this file, replacing the old environment
variable and -- forwarding interface.
3. Deploy
test/or-eks-load up --config .local/loadtest.env
This checks your credentials, configures DNS IAM, creates the cluster if absent or reconciles its shared add-ons if present, and applies the selected profile. It tests the ExternalDNS service account's actual IAM credential chain and Route 53 read access before starting the stack deployment. It then waits for DNS, trusted HTTPS, and MQTTS, and checks Keycloak's advertised login URL.
Initial dataset creation can take time; the default deployment timeout is 90 minutes. If credentials expire or a later deployment step fails, refresh your credentials, fix the reported issue, and rerun the same command. Reapplying retains stack data and credentials. Changing the selected profile does not resize existing cluster nodes or upgrade Kubernetes.
4. Retrieve credentials and run the test
test/or-eks-load credentials --config .local/loadtest.env
test/or-eks-load status --config .local/loadtest.env
Open https://load2.openremote.app/manager/ (substitute your configured hostname)
and sign in with the returned administrator credentials. Before running the
load scenario:
- Check that the page is styled and login redirects stay on the correct host.
- Verify the expected test users and assets, and asset creation/editing.
- Verify an authenticated MQTTS publish using a provisioned test account.
Configure the test client with MANAGER_HOSTNAME set to LOAD_HOSTNAME and
MQTTS port 8883. Plaintext MQTT is not exposed. Use the clients documented
in load1; for load2, use its
scenarios and parameters.
Manage the deployment
All helper commands use the same --config file.
| Command | What it does |
|---|---|
up |
Reconcile DNS IAM and cluster add-ons, then deploy and verify the stack |
cluster-up |
Configure DNS IAM and the cluster only, then check DNS access |
deploy |
Deploy and verify a stack on an already configured cluster, without changing DNS IAM or shared add-ons |
status |
Show stack status |
credentials |
Retrieve stack credentials |
uninstall |
Stop the stack while retaining data, credentials, and certificate state |
destroy-stack --confirm <stack-name> |
Delete one stack, its data, and its owned external resources |
destroy-cluster --confirm <cluster-name> |
Delete a cluster after all its stacks have been destroyed |
prepare --values-dir <new-directory> |
Write editable profile values locally, without contacting AWS |
Status, credentials, uninstall, and destruction use only cluster credentials.
deploy also uses only cluster credentials: it reads the installed ExternalDNS
configuration and tests its role access. up and cluster-up need permissions
to manage the dedicated IAM CloudFormation stack in the DNS account as well.
Reuse a cluster or deploy independently
To deploy another stack in an already configured cluster, copy the deployment
config and choose a new STACK_NAME and LOAD_HOSTNAME. Keep the same
CLUSTER_NAME and region, check available capacity, then run:
test/or-eks-load deploy --config .local/another-stack.env
The existing cluster must have the OpenRemote add-ons and managed ExternalDNS.
For an independently sized cluster, choose a new CLUSTER_NAME too and use
up; the helper derives a separate DNS role and ownership ID.
Customise component values
The standard profile requires no YAML copying. To change dataset parameters, resource limits, or image tags, prepare an editable copy:
test/or-eks-load prepare --config .local/loadtest.env --values-dir .local/loadtest-values
This refuses to overwrite an existing directory. Edit the generated component
files, then add this line to .local/loadtest.env:
LOAD_VALUES_DIR=loadtest-values
LOAD_VALUES_DIR is relative to the config file, whereas prepare's
--values-dir is relative to the current directory. Absolute paths also work.
With this setting, subsequent up and deploy calls use your files unchanged.
The image repository has already been rendered; edit manager.yaml to change
it. Remove the setting to return to the checked-in profile.
test/or-eks-load deploy --config .local/loadtest.env
An optional or-setup.yaml can supply further overrides. Prepared values can
also be passed directly to or-eks-stack or or-stack with --values-dir.
Update the Manager image or inspect logs
Prefer a unique image tag per build and update image.tag in prepared
manager.yaml, then run deploy. If you replace the image under the same tag,
reapplying unchanged values does not restart Pods. Load the trusted config into
your shell and explicitly restart Manager:
source .local/loadtest.env
kubectl --context "$CLUSTER_NAME@$AWS_REGION" --namespace "$STACK_NAME" rollout restart deployment/manager
kubectl --context "$CLUSTER_NAME@$AWS_REGION" --namespace "$STACK_NAME" rollout status deployment/manager --timeout=90m
Use the same context and namespace to inspect setup logs:
kubectl --context "$CLUSTER_NAME@$AWS_REGION" --namespace "$STACK_NAME" logs deployment/manager --tail=100
On clean initialization, logs should identify org.openremote.setup.load1.SetupTasks
or org.openremote.setup.load2.SetupTasks. Load2 also provides internal JMX:
kubectl --context "$CLUSTER_NAME@$AWS_REGION" --namespace "$STACK_NAME" port-forward service/manager-jmx 8085:8085
Reset the test dataset
Use prepared values and set or.setupRunOnRestart: true in manager.yaml.
Run deploy and wait for setup to finish, then set it back to false and run
deploy again before testing. While enabled, this setting wipes the stack's
database on every Manager startup. A normal reapply does not reset data.
Delete resources
To stop the stack while retaining its data:
test/or-eks-load uninstall --config .local/loadtest.env
To permanently delete the example stack, then its now-empty cluster:
test/or-eks-load destroy-stack --config .local/loadtest.env --confirm load2
test/or-eks-load destroy-cluster --config .local/loadtest.env --confirm loadtest
Use the exact names from your config. Destroy every stack before deleting a cluster; uninstall alone retains namespaces and is not enough. Cluster destruction refuses remaining stacks/endpoints.
The DNS-account IAM CloudFormation stack is retained for reuse. Once the
cluster is gone and its DNS cleanup has completed, you can separately delete
or-load-<cluster-account-id>-<cluster-name>-<region>-dns in the DNS account.
Configuration and deployment reference
Optional settings
| Setting | Default / purpose |
|---|---|
CLUSTER_AWS_PROFILE |
Empty: use the normal AWS credential chain |
DNS_AWS_PROFILE |
Empty: use cluster credentials |
DNS_ZONE_ID |
Discover the most specific matching public hosted zone; set to disambiguate |
DNS_DOMAIN |
Discovered zone name; set to constrain discovery |
LOAD_VALUES_DIR |
Render the selected profile into a temporary directory |
LOAD_TIMEOUT |
90m; increase for longer initialization |
LOAD_EXPOSURE |
haproxy; ingress also supported |
LOAD_MQTTS_HOSTNAME |
Required for ingress; a separate hostname in the same managed domain |
CERTIFICATE_ARN |
Ingress only: use an existing web ACM certificate |
MQTTS_CERTIFICATE_ARN |
Ingress only: use an existing MQTTS ACM certificate |
Load1 has one fixed profile; omit LOAD_PROFILE. Load2 supports large
(the fallback when omitted), xlarge-minimal, xlarge, 2xlarge, 4xlarge,
and 8xlarge. The example config selects xlarge.
DNS configuration and retries
The DNS role grants access to the selected zone/domain and trusts the cluster's exact ExternalDNS IRSA role. The helper uses a stable, account-qualified external ID identifying that relationship; it is not an access key. No manually copied role ARN or generated local state file is needed to resume on another machine.
On an existing cluster, up preserves ExternalDNS's TXT ownership ID and
refuses to change its managed zone/domain. It configures ExternalDNS to use the
helper's dedicated role, leaving any previously used DNS-account role intact.
Use deploy when the cluster's DNS IAM and add-ons should remain as configured.
The preflight requests a short-lived token for the external-dns service
account and tests both STS role assumptions and Route 53 reads. Your Kubernetes
identity needs permission to create that token. This catches trust/external-ID
mismatches early; DNS record writes and propagation are checked by deployment
readiness. Tokens and rendered temporary values are removed on exit, including
failure. Configuration and prepared values remain available for retries.
Ingress with ACM
For a new stack, add these settings before running up or deploy:
LOAD_EXPOSURE=ingress
LOAD_MQTTS_HOSTNAME=mqtt-load2.openremote.app
The web endpoint uses a shared ALB; MQTTS uses a dedicated NLB and the separate
hostname. Set the MQTT scenarios' MANAGER_HOSTNAME to LOAD_MQTTS_HOSTNAME;
browser access still uses LOAD_HOSTNAME. The CLI uses a configured shared ACM
certificate if available, otherwise manages certificates for the stack. The
optional ARN settings select existing certificates instead.
Exposure and hostname changes require a new stack. This topology differs from the legacy experiment, which puts both protocols behind one ACM-backed NLB. For deployment modes outside this helper, use the EKS CLI or or-stack directly.
Startup and capacity
Profiles contain the cluster template and component resource budgets. up
uses the template when creating a cluster; it does not resize existing nodes.
Manager and Keycloak use Recreate updates to avoid overlapping Pods on a
single-node cluster, so upgrades interrupt service.
If dataset initialization exceeds the Manager startup probe allowance, increase
that allowance in prepared manager.yaml as well as LOAD_TIMEOUT. A shared
cluster needs capacity for all stacks plus Kubernetes and shared controllers.
Namespace isolation does not remove CPU, disk, or network contention; use
separate clusters for independent benchmarks.
Legacy scripts
The old load1 scripts and load2's standard eks-setup-load.sh remain in the
repository but stop with guidance directing users to the CLIs.
The load2 ACM/NLB experiment
remains deployable using its existing setup, redeployment, and cleanup scripts.
Use a dedicated cluster for that workflow; do not mix it with CLI-managed stacks.
Offline validation
python3 kubernetes/test/load-helper-test
python3 kubernetes/test/keycloak-url-test
python3 kubernetes/test/load-eks-test
The helper tests require Python 3, Bash, jq, and envsubst. They stub AWS,
Kubernetes, Helm, and the EKS CLIs to check all profiles, DNS discovery,
role-access failures, retries, config handling, and separate cleanup operations.
The Keycloak URL tests and profile tests also require Helm. The profile tests
render all seven profiles through the real or-stack and cover the retained
legacy ACM/NLB scripts. No AWS credentials or cluster are used.
Real EKS acceptance still requires login, dataset initialization, MQTTS traffic, persistence, and independent stack cleanup against a deployed environment.