Skip to content

armgit / core

Public Imported from GitHub
0

100% open-source IoT Platform - Integrate your devices, add intelligence, build custom apps

Support multiple isolated OpenRemote stacks on Kubernetes (#3467)

Eric Bariauxebf4c2d12331
diff --git a/kubernetes/README-AWS.md b/kubernetes/README-AWS.md
index 3dad644f7..ab38819fb 100644
--- a/kubernetes/README-AWS.md
+++ b/kubernetes/README-AWS.md
@@ -2,7 +2,45 @@
## TL;DR
-Requirements: you need to have kubectl, helm, [aws cli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and [eksctl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html#eksctl-install-update) installed beforehand.
+Requirements: you need to have kubectl, helm, jq, curl, dig, Python 3 (for MQTTS readiness),
+[aws cli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html),
+and [eksctl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html#eksctl-install-update)
+installed beforehand.
+
+Cluster-only lifecycle management is available through
+[`or-eks-cluster`](or-eks-cluster). It creates, inspects, reconciles shared
+add-ons, and destroys the shared EKS cluster without deploying or deleting an
+OpenRemote stack. See [`cluster/eks/README.md`](cluster/eks/README.md) for its
+configuration and safety model.
+
+[`or-eks-stack`](or-eks-stack) is the normal stack command for an EKS cluster
+with OpenRemote-managed ExternalDNS. Its default HAProxy workflow coordinates
+the stack NLB and ACME certificate. Its explicit Ingress workflow validates an
+existing user-owned or cluster-configured shared ACM certificate, or creates
+and retains a stack-owned managed ACM certificate, and coordinates the shared
+ALB. Both delegate the Kubernetes installation to `or-stack`, wait for DNS to
+point at the current load balancer, and verify the trusted Manager HTTPS
+endpoint.
+
+Both EKS CLIs use the normal AWS credential chain by default and do not pass a
+profile argument to AWS or eksctl. For exported credentials, leave `AWS_PROFILE`
+and `AWS_DEFAULT_PROFILE` unset and omit `--profile`. Select a named profile
+explicitly with `--profile <name>`. Standard AWS environment configuration is
+still inherited; an empty `AWS_PROFILE` is not the same as an unset variable.
+The CLI-specific `OR_EKS_AWS_PROFILE` and `OR_EKS_STACK_AWS_PROFILE` environment
+overrides are no longer used. See [credential configuration](cluster/eks/README.md#configuration)
+for details, including refreshing existing kubeconfig contexts.
+
+HAProxy stacks can additionally expose MQTTS on the same NLB, hostname, and
+certificate with `--mqtts`. Ingress stacks use `--mqtts` with a separate
+hostname and an existing user-owned, cluster-shared, or stack-managed ACM
+certificate; `or-eks-stack` coordinates the certificate, dedicated MQTTS NLB,
+ExternalDNS records, trusted endpoint readiness, and destruction. Plaintext
+MQTT is not supported.
+
+The `eks-setup*.sh` and `eks-cleanup*.sh` scripts below are the legacy combined
+cluster-and-stack workflow. They will be split further as part of multi-stack
+support and must not be used to remove one stack from a shared cluster.
The scripts have been tested within the openremote account.
You must get proper credentials and set the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN variable before running them.
@@ -16,17 +54,807 @@ See comments within the script for more information.
## Setup
-We have two different options for deployment: using a HAProxy container or not.
-There are pros and cons to both approaches, see [Networking](#Networking) section for more details.
+We support two public routing options: HAProxy exposed directly through a
+per-stack NLB, or a private per-stack HAProxy gateway behind the shared ALB.
+There are pros and cons to both approaches; see the [Networking](#networking)
+section for more details.
We're creating an EKS cluster with a managed node group.
By default, the group uses 2 VMs of type t2.large. The script places them in AZ eu-west-1a and eu-west-1b
+### Hostname ownership
+
+Each public hostname belongs exclusively to one stack in the cluster, regardless
+of whether it uses HAProxy or shared ingress. The facade reserves it through
+`or-stack prepare --hostname ...` before workloads or ACM requests, preventing
+concurrent deployments from installing competing routes. Reapply preserves the
+reservation; changing a stack's hostname is rejected. Uninstall and failed
+operations retain ownership.
+
+An AWS-aware orchestrator can reserve and verify namespace ownership before it
+creates external resources:
+
+```bash
+./or-stack prepare \
+ --name stack-a \
+ --kube-context cluster@eu-west-1 \
+ --target eks \
+ --exposure ingress \
+ --dns external-dns \
+ --hostname stack-a.example.com
+```
+
+`or-eks-stack` performs this preparation automatically; normal deployments do not
+need to run it separately.
+
+`or-eks-stack destroy` releases a reservation after namespace and DNS cleanup
+(and managed certificate cleanup where applicable). An interrupted cleanup can
+be retried even when only the reservation remains. `or-eks-cluster destroy`
+refuses outstanding reservations so their ownership and recovery information
+are not lost.
+
+`or-eks-stack destroy` requires a hostname reservation in every certificate
+mode. Missing reservations stop destruction before resources are changed;
+inspect the stack and registry rather than bypassing DNS cleanup. A namespace
+left by a failed preparation that never acquired a reservation can be inspected
+and removed with the portable `or-stack destroy` command.
+
+For EKS stacks, the portable `or-stack destroy` command retains hostname
+reservations after Kubernetes cleanup. Complete external cleanup using
+`or-eks-stack destroy`, or verify that old public endpoints, DNS records, and
+stack-managed certificates have been removed before explicitly releasing the
+reservation with `or-stack release-hostname`.
+
+See [exclusive public hostnames](README.md#exclusive-public-hostnames) for the
+reason, registry permissions, required stack metadata, and explicit recovery
+when external cleanup is performed manually. The registry is cluster-local;
+coordination with other clusters and manually managed DNS remains external.
+
### Persistence
-For persistence, the PVs are backed by [EBS volumes](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html).
-Those must reside in the same AZ as the node that will be using them.
-This means that pods must be deployed to specific nodes using node affinity.
+For persistence, the component charts request dynamically provisioned
+[EBS volumes](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html).
+`or-eks-cluster` reconciles the `openremote-ebs` StorageClass, which uses the
+EBS CSI driver, encrypted `gp3` volumes, and `WaitForFirstConsumer` binding.
+The EBS volume is therefore created in the availability zone selected for its
+pod; no fixed volume IDs or pod node-affinity rules are required.
+
+The EKS values files select `openremote-ebs`. Manager and PostgreSQL PVCs are
+retained by an ordinary Helm uninstall. Explicitly deleting a retained PVC
+deletes its dynamically provisioned EBS volume because this StorageClass uses
+the `Delete` reclaim policy.
+
+### Namespaced stacks
+
+Use `or-eks-stack` for deployment with OpenRemote-managed DNS and HTTPS
+readiness checks. The examples below run from the `kubernetes` directory.
+Deploy a stack, retrieve its credentials, and use the lifecycle commands to
+inspect, uninstall, or destroy it. For custom DNS or orchestration, see
+[the advanced workflow](#advanced-deploy-with-or-stack-directly).
+
+#### Before deploying
+
+- Create and configure the shared cluster with `or-eks-cluster`. See the
+ [cluster setup instructions](cluster/eks/README.md) for storage, networking,
+ ExternalDNS, and cross-account DNS permissions.
+- Configure AWS credentials with access to the EKS account and cluster. Both
+ EKS CLIs use the normal AWS credential chain; add `--profile <name>` to select
+ a named profile explicitly.
+- Enable the OpenRemote-managed ExternalDNS release with dry-run disabled.
+ Its domain filter must cover the hostname you intend to deploy.
+- Choose a hostname strictly below the managed DNS domain. For example, with
+ `openremote.app` configured, `test.openremote.app` and
+ `staging.test.openremote.app` are supported, but `openremote.app` itself is not.
+ Any certificate you supply must cover the chosen hostname.
+
+`or-eks-stack` checks that the cluster is active, the selected kubeconfig
+context points to it, and the managed ExternalDNS configuration is suitable.
+Ingress deployments additionally require the shared `openremote-alb`
+IngressClass and its enforced HTTPS redirect.
+
+#### Deploy with HAProxy (default)
+
+After creating and configuring the shared cluster, the normal managed-DNS
+HAProxy deployment is one command. HAProxy is the default exposure and creates
+one internet-facing NLB per stack:
+
+```bash
+./or-eks-stack apply \
+ --name stack-a \
+ --cluster <cluster-name> \
+ --region eu-west-1 \
+ --hostname stack-a.example.com
+```
+
+The command installs the stack, waits for DNS to point to its NLB, obtains or
+reuses the HAProxy certificate, and waits for a trusted HTTPS response from
+Manager. After it succeeds, open `https://stack-a.example.com/manager/` and
+[retrieve the Manager credentials](#inspect-a-stack-and-retrieve-credentials).
+
+Enable MQTTS on the same hostname and NLB with:
+
+```bash
+./or-eks-stack apply \
+ --name stack-a \
+ --cluster <cluster-name> \
+ --region eu-west-1 \
+ --hostname stack-a.example.com \
+ --mqtts
+```
+
+This adds public TCP port 8883 to the existing proxy Service. Use
+`--mqtts-port <port>` only when a different public port is required. Repeat
+`--mqtts` on later applies that should retain the MQTTS exposure.
+
+The automated HTTP-01 certificate flow requires public HTTP port 80. The
+default timeout is 20 minutes for each readiness phase. Reapply the same command
+to update the stack; retained certificates are reused. To deploy another stack,
+choose a different stack name and hostname.
+
+#### Shared HTTPS routing with Ingress
+
+Select `--exposure ingress` to route web traffic through the shared OpenRemote
+ALB and a private HAProxy gateway in each stack namespace. TLS terminates at the
+ALB using an ACM certificate. `or-eks-stack apply` handles certificate validation,
+DNS readiness, and the trusted Manager HTTPS check.
+
+Choose the certificate mode before deploying:
+
+| Mode | Certificate source | Effect of stack destruction |
+| ---------- | -------------------------------------------------------------------- | --------------------------------------------------- |
+| `managed` | The stack requests and owns an ACM certificate. | Deletes the certificate and its validation records. |
+| `existing` | You supply an externally managed certificate ARN. | Leaves the certificate unchanged. |
+| `shared` | The stack uses the externally managed ARN configured on the cluster. | Leaves the certificate unchanged. |
+
+On the first apply, omitting `--certificate-mode` selects `shared` if the
+cluster has a shared certificate configured, otherwise `managed`. On later
+applies, the recorded mode is reused. The examples below select each mode
+explicitly.
+
+##### Create a stack-owned certificate
+
+With `managed`, certificate issuance completes before workloads are installed.
+ACM DNS validation can use the configured DNS account even when it is separate
+from the EKS account.
+
+Managed mode assumes that both the certificate and its ACM DNS validation
+records are exclusive to the stack. Other resources must not use the certificate,
+and other certificates must not depend on its validation records. ACM can reuse
+a validation CNAME for separate certificates, including certificates in other
+regions of the same AWS account. This exclusivity is a deployment requirement;
+the script's ownership checks do not establish it. Destroying the stack removes
+both its certificate and validation records.
+
+The resolved AWS identity needs `acm:RequestCertificate`,
+`acm:ListCertificates`, `acm:DescribeCertificate`, and
+`acm:ListTagsForCertificate` in the EKS account. It also needs
+`acm:DeleteCertificate` when destroying a stack with either managed endpoint, in
+addition to the EKS access already required by the facade. Route 53 write
+permission remains confined to the ExternalDNS roles; `or-eks-stack` does not
+use DNS-account credentials.
+
+```bash
+./or-eks-stack apply \
+ --name stack-a \
+ --cluster <cluster-name> \
+ --region eu-west-1 \
+ --exposure ingress \
+ --hostname stack-a.example.com \
+ --certificate-mode managed
+```
+
+##### Use an existing certificate
+
+Use `existing` with an ACM certificate in the EKS account and region. It must
+have status `ISSUED`, cover the hostname with an exact name or one-level
+wildcard, and be externally managed rather than owned by `or-eks-stack`.
+The command checks these requirements before changing the stack and rejects
+certificates carrying the `openremote.io/managed-by=or-eks-stack` ownership tag.
+
+```bash
+./or-eks-stack apply \
+ --name stack-a \
+ --cluster <cluster-name> \
+ --region eu-west-1 \
+ --exposure ingress \
+ --hostname stack-a.example.com \
+ --certificate-mode existing \
+ --certificate-arn arn:aws:acm:eu-west-1:123456789012:certificate/example
+```
+
+The certificate remains user-owned: apply only references it, while uninstall
+and destroy never modify or delete it. The selected mode and ARN are shown by
+`or-eks-stack status`.
+
+##### Use a cluster-configured shared certificate
+
+An externally managed certificate intended for multiple stacks can instead be
+recorded once as cluster configuration:
+
+```bash
+./or-eks-cluster apply \
+ --name <cluster-name> \
+ --region eu-west-1 \
+ --shared-certificate-arn arn:aws:acm:eu-west-1:123456789012:certificate/example
+```
+
+This option records an externally managed ARN on the `openremote-alb`
+IngressClass for stacks to use; it does not create, tag, renew, or delete the
+certificate. Omitting it on a later cluster apply preserves the reference.
+Use `--clear-shared-certificate` to remove the reference without changing ACM.
+
+Stacks then select it without repeating the ARN:
+
+```bash
+./or-eks-stack apply \
+ --name stack-a \
+ --cluster <cluster-name> \
+ --region eu-west-1 \
+ --exposure ingress \
+ --hostname stack-a.example.com \
+ --certificate-mode shared
+```
+
+Each stack validates the shared certificate's account, region, `ISSUED` status,
+hostname coverage, and ownership before changing resources. Uninstall and
+destroy never modify or delete the certificate.
+
+##### Reapply or rotate certificates
+
+Certificate mode is a persistent stack setting. When omitted on later applies,
+the recorded mode is reused, even if the cluster's shared certificate configuration
+has changed. An explicit different mode is rejected before modifying resources.
+A stack recorded as `shared` fails clearly if the cluster's shared certificate
+configuration is removed; it does not fall back to `managed`.
+
+In-place certificate mode migration is not supported. To change modes, back up
+any data you need to retain, destroy the stack and finish its cleanup, then
+recreate it in the desired mode. This avoids abandoning a managed certificate
+and its validation records while the Ingress switches to another certificate.
+
+Certificate rotation within a mode remains supported. Mode `existing` reuses
+its recorded ARN when omitted; supply `--certificate-mode existing` and a new
+`--certificate-arn` to replace it. Mode `shared` uses the cluster's current shared
+ARN on each apply. Mode `managed` reuses its owned certificate.
+
+#### MQTTS with Ingress
+
+Enable MQTTS for an Ingress stack with a second hostname. Its certificate mode
+is independent of the web certificate mode: either endpoint can use
+`existing`, `shared`, or `managed`. To select a dedicated stack-managed MQTTS
+certificate explicitly:
+
+```bash
+./or-eks-stack apply \
+ --name stack-a \
+ --cluster <cluster-name> \
+ --region eu-west-1 \
+ --exposure ingress \
+ --hostname stack-a.example.com \
+ --certificate-mode existing \
+ --certificate-arn arn:aws:acm:eu-west-1:123456789012:certificate/web \
+ --mqtts \
+ --mqtts-hostname mqtt-stack-a.example.com \
+ --mqtts-certificate-mode managed
+```
+
+The MQTTS hostname must differ from the web hostname because it points to a
+dedicated per-stack NLB rather than the shared web ALB. Managed mode records its
+intent before requesting ACM, requests or recovers one exactly tagged
+certificate for the MQTTS endpoint, publishes ACM's validation CNAME through
+the namespace's `acm-validation-mqtts` `DNSEndpoint`, and installs workloads
+only after ACM reports `ISSUED`.
+
+If neither an MQTTS mode nor ARN is supplied on first apply, the facade uses
+the cluster certificate configured by `or-eks-cluster
+--shared-certificate-arn`. It validates that the shared certificate also covers
+the MQTTS hostname, attaches the same ARN to the stack's dedicated NLB, and
+never deletes it. If the cluster has no shared ARN, selection falls back to
+managed mode. Shared mode can also be requested explicitly:
+
+```bash
+./or-eks-stack apply \
+ --name stack-a \
+ --cluster <cluster-name> \
+ --region eu-west-1 \
+ --exposure ingress \
+ --hostname stack-a.example.com \
+ --certificate-mode shared \
+ --mqtts \
+ --mqtts-hostname mqtt-stack-a.example.com \
+ --mqtts-certificate-mode shared
+```
+
+To supply a user-owned certificate instead, use:
+
+```bash
+./or-eks-stack apply \
+ --name stack-a \
+ --cluster <cluster-name> \
+ --region eu-west-1 \
+ --exposure ingress \
+ --hostname stack-a.example.com \
+ --certificate-mode existing \
+ --certificate-arn arn:aws:acm:eu-west-1:123456789012:certificate/web \
+ --mqtts \
+ --mqtts-hostname mqtt-stack-a.example.com \
+ --mqtts-certificate-mode existing \
+ --mqtts-certificate-arn arn:aws:acm:eu-west-1:123456789012:certificate/mqtts
+```
+
+An explicit MQTTS ARN also implies `existing`, so its mode option may be
+omitted. The certificate must be in the EKS account and region, have status
+`ISSUED`, and cover the MQTTS hostname. It must be externally owned:
+certificates tagged as managed by `or-eks-stack` are rejected, and the facade
+never renews or deletes it. A single wildcard or multi-name existing
+certificate may be supplied for both web and MQTTS when it meets the ownership
+and hostname checks.
+
+Repeat `--mqtts` on every reapply. After first selection, the MQTTS mode is
+persistent and the hostname and certificate ARN may be omitted because they are
+read from namespace annotations. An explicit different mode is rejected.
+Existing mode can rotate its certificate reference by supplying a new ARN;
+shared mode follows the cluster's current shared ARN; managed mode reuses its
+owned certificate. Changing or disabling the MQTTS hostname in place is not
+supported. To change mode or hostname, back up any data to retain, destroy the
+stack, and recreate it. `or-eks-stack status` reports the recorded MQTTS
+hostname, certificate mode, ARN, and current ACM status.
+
+Apply waits for both the shared ALB and dedicated MQTTS NLB, verifies each DNS
+name points to its current load balancer, and verifies trusted HTTPS and MQTTS.
+Uninstall retains both endpoint reservations and the recorded certificate
+state. Destroy validates the MQTTS Service and its lifecycle metadata, removes
+the namespace, confirms the NLB and both public DNS names are gone, and then
+releases both reservations. Existing and shared MQTTS certificates are always
+retained. NLB deletion is checked using the controller's AWS identity tags:
+`elbv2.k8s.aws/cluster=<cluster>` and `service.k8s.aws/stack=<stack>/proxy-mqtts`.
+This check does not depend on the Service's last reported address and also
+runs when the Service or namespace is already gone, so an interrupted destroy
+or an earlier uninstall cannot bypass it. AWS lookup failures retain both
+reservations; retry destroy once the lookup succeeds and the NLB is deleted.
+For managed mode it also waits for NLB detachment and
+validation-CNAME removal, then deletes only the certificate carrying the exact
+cluster, stack, hostname, and `openremote.io/endpoint=mqtts` ownership tags.
+Apply records certificate mode before ACM requests or Helm installation. If it
+stops earlier, between hostname reservation and recording mode, destroy also
+accepts the incomplete preparation when no MQTTS Service, recorded certificate
+ARN, or validation DNSEndpoint exists. The hostname, if already recorded, must
+match its reservation. Normal certificate ownership, NLB deletion, and DNS
+cleanup checks still apply before releasing either hostname. A low-level
+`or-stack` MQTTS endpoint with deployed resources is not adopted automatically
+while its namespace still exists.
+
+Destroying an Ingress stack with MQTTS also uses
+`elasticloadbalancing:DescribeLoadBalancers` and
+`elasticloadbalancing:DescribeTags` to confirm deletion of the dedicated NLB,
+including on retries after namespace deletion.
+
+#### Inspect a stack and retrieve credentials
+
+These commands apply to both HAProxy and Ingress stacks. Inspect the stack's
+resources and configuration with:
+
+```bash
+./or-eks-stack status \
+ --name stack-a \
+ --cluster <cluster-name> \
+ --region eu-west-1
+```
+
+New stacks receive independently generated Manager and PostgreSQL passwords.
+Retrieve a stack's Manager administrator login without exposing its database
+credentials with:
+
+```bash
+./or-eks-stack credentials \
+ --name stack-a \
+ --cluster <cluster-name> \
+ --region eu-west-1
+```
+
+#### Uninstall, restore, or destroy a stack
+
+##### Uninstall and restore
+
+Remove workloads while retaining their credentials and EBS-backed data with:
+
+```bash
+./or-eks-stack uninstall \
+ --name stack-a \
+ --cluster <cluster-name> \
+ --region eu-west-1
+```
+
+Reapplying the stack with the same target and exposure reuses its retained
+Secret and data PVCs. HAProxy exposure also reuses its retained certificate
+PVC; Ingress gateways have no certificate PVC because TLS terminates at the
+ALB. The selected target and exposure are stored as namespace labels; changing
+either is rejected until a deliberate migration workflow is implemented.
+
+For managed Ingress certificates, uninstall also retains the certificate and
+its DNS validation records so ACM can renew it and a later apply can reuse it.
+Restore workloads by rerunning the original `apply` command.
+
+##### Destroy the stack and its data
+
+Use `destroy` to delete the namespace, credentials, PVCs, and dynamically
+provisioned EBS volumes. This command applies to HAProxy and all Ingress
+certificate modes. Managed Ingress certificates and their validation records
+are also deleted; existing and shared certificates are preserved.
+
+```bash
+./or-eks-stack destroy \
+ --name stack-a \
+ --cluster <cluster-name> \
+ --region eu-west-1 \
+ --confirm stack-a
+```
+
+Destruction requires exact-name confirmation, matching stack ownership, and a
+hostname reservation. It stops if certificate or DNS ownership is absent,
+mismatched, or ambiguous. It waits for external DNS cleanup and for the
+`Delete` reclaim policy to remove the stack's PersistentVolumes. See
+[hostname ownership](#hostname-ownership) for recovery from incomplete preparation.
+
+An interrupted destroy can be retried, including after the namespace has gone.
+For managed certificates, the command recovers ownership from the retained
+reservation and certificate tags. The
+[managed-certificate lifecycle](#managed-certificate-lifecycle) explains the
+cleanup order.
+
+Complete stack cleanup before running `or-eks-cluster destroy`. Cluster
+destruction refuses remaining OpenRemote namespaces, non-system workload
+controllers (even those with no Pods), retained hostname reservations, and
+PersistentVolumes whose claims have already gone. See the
+[cluster lifecycle documentation](cluster/eks/README.md#lifecycle).
+
+#### Advanced: deploy with `or-stack` directly
+
+Use the portable `or-stack` command for custom DNS ownership, internal-only
+stacks, or custom orchestration. It installs Kubernetes resources but makes no
+AWS calls and does not perform the end-to-end DNS, certificate, and HTTPS
+readiness workflow of `or-eks-stack`.
+
+##### HAProxy with externally managed DNS
+
+Use this procedure when you manage DNS records yourself. `or-stack` deploys the
+Kubernetes resources; you then configure DNS, complete certificate setup, and
+verify HTTPS access.
+
+1. **Deploy the stack without DNS automation.**
+
+ ```bash
+ ./or-stack apply \
+ --name stack-a \
+ --kube-context <cluster-name>@eu-west-1 \
+ --target eks \
+ --hostname stack-a.example.com \
+ --dns none
+ ```
+
+2. **Point the hostname at the stack's load balancer.** Retrieve the NLB hostname:
+
+ ```bash
+ kubectl --context <cluster-name>@eu-west-1 \
+ --namespace stack-a \
+ get service proxy \
+ -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{"\n"}'
+ ```
+
+ If the output is empty, wait for the load balancer to be provisioned and
+ retry. Use your DNS provider to point `stack-a.example.com` at that hostname.
+ Wait until public DNS resolves to the current load balancer before continuing.
+
+3. **Complete certificate setup.** The HTTP-01 challenge requires public HTTP
+ port 80 to reach HAProxy. If the hostname already has retained Certbot
+ certificate state, skip the following command and reuse it. Otherwise,
+ request the certificate after DNS is ready:
+
+ ```bash
+ kubectl --context <cluster-name>@eu-west-1 \
+ --namespace stack-a \
+ exec deployment/proxy \
+ --container proxy \
+ -- /entrypoint.sh add stack-a.example.com
+ ```
+
+ If you intentionally use a custom public HTTP port, arrange a suitable
+ certificate separately instead of relying on this HTTP-01 procedure.
+
+4. **Verify certificate status and Manager access.** Inspect the certificate:
+
+ ```bash
+ ./or-stack status \
+ --name stack-a \
+ --kube-context <cluster-name>@eu-west-1
+ ```
+
+ Then open `https://stack-a.example.com/manager/` and confirm that Manager is
+ reachable over trusted HTTPS. Certificate files alone do not establish that
+ the public endpoint works.
+
+##### Alternative: let ExternalDNS manage the HAProxy record
+
+If your cluster has a compatible ExternalDNS controller, use
+`--dns external-dns` instead of `--dns none` in the apply command above. The
+proxy Service declares the hostname, and the controller creates the DNS record
+pointing to the NLB. Wait for DNS to resolve to the current load balancer, then
+complete the certificate and HTTPS checks in steps 3 and 4 yourself.
+
+##### Ingress with externally managed readiness
+
+Use `or-stack --exposure ingress` directly when DNS or endpoint readiness is
+managed elsewhere. The portable command accepts the ARN but deliberately does
+not inspect ACM or call Route 53.
+
+With this workflow, you are responsible for supplying a suitable issued ACM
+certificate, arranging DNS, and checking the public HTTPS endpoint. Inspect the
+ALB address once the controller has published it:
+
+```bash
+kubectl --context <cluster-name>@eu-west-1 \
+ --namespace stack-a \
+ get ingress proxy \
+ -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{"\n"}'
+```
+
+##### Ingress MQTTS with externally managed readiness
+
+The HTTP Ingress does not carry MQTT traffic. On EKS, `or-stack` can create a
+separate per-stack NLB for MQTTS while retaining the same private proxy gateway:
+
+```bash
+./or-stack apply \
+ --name stack-a \
+ --kube-context <cluster>@eu-west-1 \
+ --target eks \
+ --exposure ingress \
+ --dns external-dns \
+ --hostname stack-a.example.com \
+ --certificate-arn <web-acm-certificate-arn> \
+ --mqtts \
+ --mqtts-hostname mqtt-stack-a.example.com \
+ --mqtts-certificate-arn <mqtts-acm-certificate-arn>
+```
+
+The dedicated `proxy-mqtts` LoadBalancer Service creates an IP-target NLB. ACM
+terminates TLS on public port 8883, then the NLB forwards plaintext TCP to port
+1883 on the namespace-local proxy, which forwards it to Manager. The MQTTS
+hostname must differ from the web hostname because their DNS records target
+different load balancers. `--mqtts-port` changes the public NLB listener port.
+
+This low-level workflow only declares the provided certificate ARN; it does not
+inspect, create, renew, or delete the certificate. When `--dns external-dns` is
+selected, it declares the MQTTS hostname on the NLB Service. Without it, DNS is
+also the caller's responsibility.
+
+For managed DNS, certificate lifecycle, and endpoint readiness, use
+[MQTTS with Ingress](#mqtts-with-ingress) through `or-eks-stack`.
+
+##### Internal-only stacks
+
+Use `--exposure none` with `or-stack` for an internal-only EKS stack. It creates
+no proxy, Ingress, ALB, or NLB:
+
+```bash
+./or-stack apply \
+ --name stack-internal \
+ --kube-context <cluster-name>@eu-west-1 \
+ --target eks \
+ --exposure none
+```
+
+##### Use a separate ALB group
+
+A stack can opt out of the default group through its `proxy.yaml` values. Set
+`ingress.className: alb` and give the proxy Ingress a unique
+`alb.ingress.kubernetes.io/group.name`, together with the desired `scheme` and
+`target-type` annotations. Also add the `ssl-redirect` annotation because the
+dedicated class does not inherit the shared class's redirect setting. That
+creates a separate ALB group for that stack; the shared class remains the
+default.
+
+#### How deployment works
+
+The following details explain the validation, routing, and recovery behavior
+behind the commands above. They do not add manual steps to the normal
+`or-eks-stack` workflow.
+
+##### Cluster checks, storage, and network isolation
+
+The EKS target verifies that the EBS CSI driver and `openremote-ebs`
+StorageClass exist. It also requires the Amazon VPC CNI NetworkPolicy
+`PolicyEndpoint` CRD and node agent, confirming that native policy support is
+configured. Public HAProxy and Ingress exposure also require the AWS Load
+Balancer Controller, while `--exposure none` does not. The target selects
+encrypted dynamically provisioned EBS storage for Manager and PostgreSQL.
+HAProxy exposure additionally persists its certificate state; the private
+Ingress gateway does not store certificates.
+
+Each stack applies an ingress NetworkPolicy that allows traffic from its own
+namespace and rejects traffic originating in other stack namespaces. Egress is
+not restricted. In HAProxy mode, a second policy permits external traffic only
+to the HAProxy Pod's HTTP and HTTPS ports, plus its MQTTS port when enabled;
+Manager, Keycloak, PostgreSQL, and Manager's plaintext MQTT listener are not
+directly public. In Ingress mode, the second policy permits the ALB to reach
+HTTP port 8080 on a private per-stack HAProxy gateway. When low-level Ingress
+MQTTS is enabled, it also permits the dedicated NLB to reach the gateway's
+plaintext port 1883. That gateway then reaches Manager and Keycloak through the
+same-namespace rule, so neither application Pod is directly reachable from
+another stack namespace.
+`or-eks-cluster create` enables NetworkPolicy on the managed VPC CNI add-on,
+and `or-eks-cluster apply` reconciles that setting without changing the
+installed add-on version. New clusters give the VPC CNI add-on a dedicated IAM
+role with `AmazonEKS_CNI_Policy`; for existing add-ons, cluster reconciliation
+aligns the `aws-node` ServiceAccount annotation with the role already recorded
+by EKS.
+
+##### DNS domain restrictions
+
+The hostname must be below the managed domain because ExternalDNS prefixes
+its ownership TXT names with a record type. At the managed domain itself, these
+records would fall outside the IAM domain boundary. Subdomain depth is
+unrestricted by this check, but the certificate must still cover the hostname.
+
+##### HAProxy certificate lifecycle and readiness
+
+In HAProxy exposure mode, the proxy manages its own TLS certificate and keeps
+Certbot account and certificate data on its retained `proxy` PVC.
+
+`or-eks-stack` automates the necessary ordering. It waits until the public
+hostname and the current NLB hostname resolve to at least one common address,
+so a stale record from an older deployment is not considered ready. It then
+checks the proxy's persisted Certbot state and triggers `/entrypoint.sh add`
+only when the hostname has no certificate lineage. Existing private-key and
+full-chain files are reused; normal renewal remains the proxy's responsibility.
+Incomplete state or a failed inspection stops apply for investigation. If proxy
+startup creates the certificate concurrently and `add` fails, the facade
+rechecks the files before continuing.
+
+For manual certificate operations, `add` rejects an existing certificate
+lineage, while `renew` forces issuance rather than checking readiness.
+
+`or-eks-stack` independently waits for a trusted response from the canonical
+`/manager/` URL: persisted files alone do not prove that TLS or Manager is ready.
+A Manager error or connection timeout therefore does not trigger certificate
+issuance or forced renewal.
+
+When `--mqtts` is enabled, it also completes a trusted TLS handshake with the
+MQTTS listener before reporting the stack ready. HTTPS and MQTTS use the same
+hostname and HAProxy certificate.
+
+The proxy readiness probe and certificate status are deliberately separate.
+The Pod must be ready and reachable for the HTTP ACME challenge before a
+production certificate can be issued; `or-stack status` shows the certificate
+issuer and expiry once managed or custom certificate material is present.
+
+##### MQTTS readiness
+
+MQTTS readiness requires Python 3 with its standard `ssl` module and a working
+CA trust store (using Python's default CA paths or `SSL_CERT_FILE`/`SSL_CERT_DIR`).
+The probe verifies the certificate chain and requested hostname
+through the TLS API: OpenSSL/LibreSSL diagnostic output alone can report a zero
+verification code even when no TLS session was established. Each probe is
+limited to ten seconds or the remaining readiness timeout, whichever is shorter.
+This verifies TLS readiness; MQTT authentication and publish/subscribe behavior
+still require an MQTT client test.
+
+##### Managed-certificate lifecycle
+
+This lifecycle applies independently to managed web and MQTTS certificates
+on Ingress stacks. Namespace annotations record each endpoint's selected mode
+and certificate ARN. ACM tags identify ownership by deployment tool, cluster,
+stack, exact hostname, and endpoint, allowing the CLI to verify ownership and
+recover interrupted operations. Certificates created before endpoint tagging
+remain recognized as web certificates for upgrade compatibility.
+
+###### Apply
+
+`or-eks-stack apply` performs these steps automatically:
+
+1. Prepare the namespace and record the certificate mode before making ACM
+ requests or installing workloads.
+2. Reuse the recorded certificate ARN, discover a uniquely tagged certificate,
+ or request a new DNS-validated ACM certificate if none exists.
+3. Store validation CNAME records in the namespace's `acm-validation`
+ `DNSEndpoint` for web certificates and `acm-validation-mqtts` for MQTTS.
+ The latter certificate carries the `openremote.io/endpoint=mqtts` tag.
+ ExternalDNS publishes the records through its restricted cross-account role.
+4. Wait for every selected managed certificate to report `ISSUED` before
+ installing workloads.
+
+Both endpoints verify ownership before updating an existing validation
+`DNSEndpoint`. Only a confirmed missing resource permits creation; permission
+errors or API failures stop apply before writing validation records.
+
+###### Retry after interruption
+
+Rerun the same `apply` command. The recorded mode survives failed applies; if
+the certificate ARN was not recorded before the interruption, the command
+discovers the certificate through its ownership tags or requests one if none
+exists.
+
+A conditional namespace update prevents concurrent first applies from choosing
+different modes. If that update conflicts, retry `apply`.
+
+###### Uninstall
+
+`uninstall` retains the recorded mode, certificate, and validation records.
+ACM can continue renewing the certificate, and a later `apply` can reuse it.
+
+###### Destroy
+
+Before deleting anything, `destroy` validates the namespace, Ingress, MQTTS
+Service when present, each validation `DNSEndpoint`, and exact ACM ownership
+tags. It then:
+
+1. Removes the Ingress and waits for the web hostname to disappear from
+ authoritative DNS and for the managed web certificate to be detached.
+2. Destroys the namespace, including the MQTTS Service and validation resources.
+3. For MQTTS, waits for NLB deletion and managed-certificate detachment.
+4. Waits for public and validation DNS records to disappear from authoritative
+ DNS.
+5. Deletes the owned managed certificates.
+
+An interrupted `destroy` can be retried. The command can recover the certificate
+from its cluster, stack, and endpoint tags even after the namespace is gone. Existing and
+shared certificates are never deleted by stack destruction.
+
+##### Shared ALB routing and gateway design
+
+`or-eks-cluster` creates an `openremote-alb` IngressClass backed by an AWS Load
+Balancer Controller `IngressClassParams` resource. It fixes the scheme to
+`internet-facing`, uses VPC CNI Pod IPs as targets, and puts every matching
+Ingress in the `openremote-stacks` group. It also enforces redirection from
+HTTP to HTTPS. The controller therefore merges one proxy Ingress from each
+stack namespace onto one ALB by default.
+
+The resulting web path is:
+
+```text
+Client -> shared ALB (TLS/ACM) -> stack HAProxy gateway (HTTP) -> Manager/Keycloak
+```
+
+TLS terminates at the ALB. HAProxy receives plain HTTP and routes /auth to Keycloak and other paths to Manager.
+It does not request certificates or expose an HTTPS Service port in this mode.
+
+The gateway is an intentional NetworkPolicy boundary. The external ALB has no
+Kubernetes Pod or namespace labels, so a portable NetworkPolicy rule that lets
+it target Manager and Keycloak directly also admits Pods from other namespaces
+on those ports. Opening only the proxy port preserves direct cross-stack
+isolation.
+
+This design adds one internal HTTP hop and one small proxy Pod per stack. The
+reason to select it is ALB-managed web TLS and layer-7 integration: ACM, SNI,
+central HTTP redirects, and optional AWS features such as WAF. It does not
+reduce the final load-balancer count. MQTTS requires one additional NLB per
+Ingress stack, while HAProxy exposure reuses its existing per-stack NLB.
+
+Only namespaces carrying `app.kubernetes.io/part-of=openremote` may use this
+class. `or-stack` applies that label when it creates or reapplies a valid stack
+namespace. This cluster-side restriction prevents unrelated namespaces from
+joining the shared ALB merely by naming its group.
+
+The stack Ingress declares its hostname and the cluster controller creates the
+record that points to the ALB. The facade waits for the Ingress to publish its
+ALB hostname, confirms that public DNS resolves to that current ALB, and then
+verifies the trusted `/manager/` endpoint. The EKS target configures listeners
+on ports 80 and 443, while `or-stack` adds the validated certificate supplied
+through `--certificate-arn` to the stack's proxy Ingress.
+
+Certificate annotations are merged across the shared IngressGroup, allowing
+the ALB to use SNI when stacks use different certificates. A wildcard
+certificate can instead be passed to every stack. The shared class owns the
+HTTPS redirect so stacks cannot configure conflicting redirect behavior. See
+the AWS Load Balancer Controller documentation for
+[IngressClassParams](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/ingress/ingress_class/)
+and [IngressGroup annotation behavior](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/ingress/annotations/#ingressgroup).
+
+##### Cluster destruction and node drain
+
+Once the cluster-level cleanup checks succeed, `or-eks-cluster` bypasses system
+PodDisruptionBudgets during the final node drain so EKS add-ons cannot leave
+cluster deletion waiting indefinitely.
#### PosgreSQL data directory
@@ -78,20 +906,23 @@ HAProxy exposes a LoadBalancer service for communication from the outside world.
We're using [AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.7/) to automatically create a Network Load Balancer (NLB) based on that service.
One NLB is automatically created when a (LoadBalancer) Service object exists and destroyed when the Service is deleted.
-In this configuration, only MQTTS connectivity is available, not MQTT.
+The namespaced HAProxy profile exposes HTTP and HTTPS by default. `--mqtts`
+adds port 8883 to the same per-stack NLB and HAProxy terminates MQTTS with the
+same hostname and certificate used for HTTPS before forwarding to Manager's
+namespace-local port 1883. `--mqtts-port` can change the public Service port;
+the HAProxy container listener remains 8883. Plaintext MQTT is never exposed.
-#### Using Ingresses and individual LoadBalancer services
+#### Using Ingress with a per-stack gateway
-TL;DR: Use scripts `eks-setup.sh` and `eks-cleanup.sh` for this configuration.
+In this configuration, each stack creates one proxy Ingress. The AWS Load
+Balancer Controller merges those Ingresses into the shared ALB and targets the
+corresponding private HAProxy gateway Pods by IP. Manager and Keycloak expose
+only ClusterIP Services and are never direct ALB targets.
-In this configuration, standard kubernetes Ingress and Load Balancer services are used
-by each pod to expose their APIs to the outside world.
-
-We're using [AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.7/) to automatically create an Application Load Balancer based on the Ingress manifests.
-An Application Load Balancer (ALB) is automatically created when an Ingress object exists in the cluster and destroyed when there are none.
-
-By default, each ingress creates its own ALB, using the [IngressGroup](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/ingress/annotations/#ingressgroup) feature
-and the corresponding `alb.ingress.kubernetes.io/group.name` annotation on the Ingress allows to use a single ALB for multiple ingresses.
+The shared `IngressClassParams` supplies the common
+[IngressGroup](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/ingress/annotations/#ingressgroup)
+name, public scheme, IP target type, and HTTPS redirect. Deleting the last
+Ingress removes the shared ALB.
The ALB will automatically assign a public "Amazon" DNS name for access.
We want to use our own DNS names (for ease of use, consistency but also to associate a TLS certificate with it).
@@ -100,13 +931,15 @@ For that, we use [Route 53](https://aws.amazon.com/route53/) and an alias record
To create the certificate, we use [AWS Certificate Manager](https://docs.aws.amazon.com/acm/latest/userguide/acm-overview.html).
Domain ownership validation is performed via DNS record.
-For MQTT(S), the manager uses two Load Balancer services (one for MQTT, one for MQTTS).
-This creates one Network Load Balancer per service.
-We need to use different domain names to route traffic to the appropriate NLB.
-We use subdomains for that, with prefix mqtt and mqtts respectively.
-
-For MQTTS, we use the same certificate as for the ingress.
-During certificate creation, we add a SAN for the mqtts subdomain.
+The old combined scripts exposed MQTT(S) with additional Manager LoadBalancer
+services. The namespaced HAProxy workflow instead keeps Manager private and
+reuses the existing proxy NLB when `--mqtts` is selected. Ingress stacks still
+use the proxy as their isolation gateway. `or-eks-stack` creates one dedicated
+MQTTS NLB per stack, with an existing, cluster-shared, or stack-managed ACM
+certificate terminating TLS before plaintext MQTT is forwarded to proxy port
+1883 and then Manager. The facade coordinates the endpoint's separate hostname,
+certificate lifecycle, readiness, and teardown. Plaintext MQTT is not publicly
+exposed.
#### Annotations
@@ -182,12 +1015,8 @@ It is best to always check the CloudFormation stacks using the AWS Console and m
We are not using EKS Auto Mode for now. When we want to move forward with EKS, we should investigate this option.
-As indicated above, we're using EBS volumes and need to make sure the pod using them is on a node in the same AZ.
-This limits the possibility of distributing pods on the different worker nodes and negates
-some of the automatic orchestration offered by k8s.
-At this stage, it's not critical as anyway the pods using volumes are the DB (which is a stateful set with a single pod)
-and the manager (which is not architected yet to support multiple replicas).
-We could explore using storage class and not explicit binding and/or using other volume types.
-We should anyway review the manager usage of persistent storage and limit the use of the file system for this purpose.
+The database is a single-pod StatefulSet, and the manager is not currently
+architected to support multiple replicas. The manager's use of persistent file
+storage should still be reviewed and reduced before production usage.
When moving to production usage, the shell script might be replaced with CloudFormation templates or using Terraform or similar tools.
diff --git a/kubernetes/README.md b/kubernetes/README.md
index 4bd3d32f0..d8f418c26 100644
--- a/kubernetes/README.md
+++ b/kubernetes/README.md
@@ -6,46 +6,509 @@ It supposes you have already installed the required tools on your machine:
- a Kubernetes installation, tests were performed with Docker Desktop on macOS.
- kubectl
- helm
+- jq (hostname reservations and stack destruction)
-There are two options on how the OR components can be accessed:
+There are three exposure modes: public HAProxy, Kubernetes Ingress, and
+internal-only. Both public modes use a HAProxy Pod in each stack namespace.
+With HAProxy exposure it is the public TLS endpoint; with Ingress exposure it
+is a private gateway behind the Ingress controller. An Ingress controller must
+be installed for the latter mode
+([Nginx Controller](https://kubernetes.github.io/ingress-nginx/deploy/#quick-start)
+was used during local testing).
-- using a HAProxy pod to manage all connections
-- using the standard kubernetes network objects: Ingress and Service
+This README covers local deployment and the portable `or-stack` CLI.
+For AWS EKS deployment using `or-eks-cluster` and `or-eks-stack`, see [README-AWS.md](README-AWS.md).
-In the later case, an Ingress controller must be installed in the cluster ([Nginx Controller](https://kubernetes.github.io/ingress-nginx/deploy/#quick-start) was used during testing).
-In the former case, an Ingress controller should NOT be installed as this causes conflicts.
+## TL;DR: deploy a local stack
-This README file covers deployment on a local machine, for information on deploying into an EKS cluster on AWS, see README-AWS.md
+With Kubernetes enabled in Docker Desktop and the tools listed above installed,
+run the following from the repository root:
-## TL;DR
+```bash
+cd kubernetes
+./or-stack apply --name stack-a --kube-context docker-desktop --target local
+./or-stack credentials --name stack-a --kube-context docker-desktop
+```
+
+Open <https://localhost/manager> and sign in with the credentials printed by the
+last command. This uses the defaults: HAProxy exposure, hostname `localhost`,
+HTTP port 80, HTTPS port 443, and no DNS management. A browser warning is
+expected for the locally generated certificate.
+
+For another local cluster or different ports, see
+[Local deployment with HAProxy](#local-deployment-with-haproxy).
+
+## Namespaced stack management
+
+`or-stack` installs each OpenRemote stack into a namespace with the same name.
+The Helm release names remain `or-setup`, `postgresql`, `keycloak`, and
+`manager`, plus `proxy` when exposure is haproxy or ingress (for both local and EKS), inside every namespace.
+Existing component service names therefore continue to work while namespaced resources
+and persistent data remain independent.
+
+The stack command provides namespace preparation, apply, inspection, credential
+retrieval, uninstall, and explicit destruction operations. Target platform and
+public exposure are separate choices. `--target` selects local or EKS storage
+and cluster requirements; `--exposure` selects `haproxy`, `ingress`, or `none`.
+HAProxy is the default exposure for both targets. DNS ownership is a separate,
+portable choice: `--dns external-dns` declares the hostname to a compatible
+cluster controller, while the default `--dns none` leaves it externally
+managed.
+
+The selected target and exposure are stored on the namespace. Reapplying with a
+different choice is rejected because changing public routing in place needs a
+deliberate migration.
+
+### Local deployment with HAProxy
+
+The local target uses the cluster's default StorageClass and enables the
+PostgreSQL volume permission init container required by Docker Desktop's
+dynamically provisioned hostpath volumes.
+
+#### With LoadBalancer support
+
+To install or upgrade a local stack on Docker Desktop, or another cluster with
+LoadBalancer support:
+
+```bash
+./or-stack apply \
+ --name stack-a \
+ --kube-context docker-desktop \
+ --target local \
+ --http-port 8080 \
+ --https-port 8443
+```
+
+The local HAProxy Service uses the cluster's `LoadBalancer` support and reports
+the configured HTTPS port to Manager and Keycloak. On Docker Desktop, deploy a
+second independent stack with another name and different host-facing ports:
+
+```bash
+./or-stack apply \
+ --name stack-b \
+ --kube-context docker-desktop \
+ --target local \
+ --http-port 9080 \
+ --https-port 9443
+```
+
+The Managers are then available at `https://localhost:8443/manager` and
+`https://localhost:9443/manager`, using the proxy's locally generated
+certificates. Browser warnings for those local certificates are expected.
+
+MQTTS is disabled by default. To expose it through the same HAProxy and
+certificate, add `--mqtts`; for multiple local stacks, also give each Service a
+different public port, for example `--mqtts-port 18883` for stack A and
+`--mqtts-port 28883` for stack B. The HAProxy container continues to listen on
+8883 inside each namespace.
+
+#### Without LoadBalancer support
+
+For kind, kubeadm, or another cluster without a LoadBalancer implementation,
+put this in `./stacks/stack-a/proxy.yaml`:
+
+```yaml
+service:
+ http:
+ type: ClusterIP
+```
+
+Apply with explicit local ports, then keep a port-forward running:
+
+```bash
+./or-stack apply \
+ --name stack-a \
+ --kube-context kind-openremote \
+ --target local \
+ --hostname localhost \
+ --http-port 8080 \
+ --https-port 8443 \
+ --values-dir ./stacks/stack-a
+
+kubectl --context kind-openremote --namespace stack-a \
+ port-forward service/proxy 8080:8080 8443:8443
+```
+
+### Local deployment with Ingress
+
+Select Kubernetes Ingress explicitly with `--exposure ingress` when calling
+`or-stack apply --target local`. Ingress traffic terminates TLS at
+the configured controller and then passes through a private per-stack HAProxy
+gateway to Manager or Keycloak. A local Ingress installation and its TLS Secret
+are user-managed. `or-stack` declares the TLS hostname; a controller may use its
+default certificate when no `secretName` is provided.
+
+### Internal-only deployment
+
+Use `--exposure none` when calling `or-stack apply` for an internal-only stack.
+This mode deploys the backend components without HAProxy or an Ingress.
+
+### Customizing a stack
+
+`apply` accepts an optional `--values-dir`. Files named `or-setup.yaml`,
+`postgresql.yaml`, `keycloak.yaml`, `manager.yaml`, and `proxy.yaml` in that
+directory are applied after the selected target and exposure values.
+Command-line hostname configuration is also available:
+
+```bash
+./or-stack apply \
+ --name stack-a \
+ --kube-context docker-desktop \
+ --target local \
+ --hostname stack-a.localhost \
+ --http-port 8080 \
+ --https-port 8443 \
+ --values-dir ./stacks/stack-a
+```
+
+To choose initial credentials instead of generating them, put the following in
+`or-setup.yaml` in that values directory:
+
+```yaml
+credentials:
+ adminPassword: "replace-me"
+ postgresqlUsername: "postgres"
+ postgresqlPassword: "replace-me-too"
+```
+
+Avoid committing credential values to source control. These settings apply only
+when the stack Secret is first created. A later `apply` preserves the existing
+values and rejects a conflicting override because changing a Kubernetes Secret
+alone does not rotate credentials inside an initialized PostgreSQL or Keycloak
+database.
+
+### Inspecting a stack and retrieving credentials
+
+#### Stack status
+
+Show resources belonging to one stack:
+
+```bash
+./or-stack status --name stack-a --kube-context docker-desktop
+```
+
+For HAProxy exposure, `status` also reports the external address and the issuer
+and expiry of a readable managed or custom certificate. If no such certificate
+is found, it warns that the fallback self-signed certificate may be in use.
+
+The HAProxy Pod readiness probe checks that the proxy can serve traffic;
+it does not wait for certificate issuance. An HTTP ACME challenge needs the
+Service to send traffic to that Pod before issuance can complete. Pod readiness
+therefore does not by itself confirm that a trusted certificate is available.
+
+#### Manager credentials
+
+Each new stack receives a generated Manager administrator password and a
+separate generated PostgreSQL password. Retrieve the Manager login at any time
+with:
+
+```bash
+./or-stack credentials --name stack-a --kube-context docker-desktop
+```
+
+The Manager username is `admin`. The command deliberately does not display the
+PostgreSQL credentials.
+
+### Authentication and CORS configuration
+
+The following settings are applied automatically by `or-stack`; they are useful
+to check when troubleshooting login or browser request failures.
+
+For public stacks, `or-stack` sets Keycloak's hostname to the full public HTTPS
+URL, including `/auth` and any non-default HTTPS port. Manager discovers the
+token issuer through Keycloak's internal HTTP endpoint; a bare hostname would
+let discovery and browser requests produce different issuers, causing login
+to fail with `Invalid token issuer`. After correcting this setting on an
+existing stack, restart Manager so it refreshes its cached issuer.
+
+`or-stack` also sets `or.webserver.allowedOrigins`, which the Manager chart maps
+to `OR_WEBSERVER_ALLOWED_ORIGINS`, with the stack's exact public HTTPS origin,
+including its port. Manager's default CORS origins use the hostname without a
+custom port; browser writes such as asset
+creation would otherwise fail with HTTP 403 `CORS origin denied`, even when
+login and page loading work. Each stack allows its own public origin without
+enabling wildcard CORS access.
+
+The Manager chart groups all `OR_WEBSERVER_*` settings under `or.webserver` in
+[`manager/values.yaml`](manager/values.yaml): listener host and port, CORS origins,
+methods and headers, request logging, thread limits, and request size limits.
+Null or empty values omit the corresponding environment variable so Manager
+uses its application default. `listenPort` also sets the container's named HTTP
+port, which the Service target and default health probes use; `service.http.port`
+continues to control the Service's exposed port independently.
+
+### Namespace preparation for deployment tools
+
+Higher-level deployment tools can use `or-stack prepare` to reserve the namespace
+and hostname before provisioning external resources. Normal deployments can call
+`apply` directly, which also performs this preparation.
+
+`prepare` creates the namespace and its OpenRemote ownership/configuration
+labels, and atomically reserves the requested public hostname. It installs no
+Helm release, workload, Service, Ingress, Secret, or PVC, and it performs no AWS
+or cluster-capability checks. Repeating it is safe
+when the namespace has the matching stack label, target, and exposure. It
+refuses an unrelated namespace or an attempt to change the stored target or
+exposure or reserved hostname.
+
+### Network isolation
+
+Every stack installs a `NetworkPolicy` that selects all Pods in its namespace.
+Inbound traffic is allowed from Pods in the same namespace and denied from
+other namespaces. Outbound traffic remains unrestricted so Manager and
+Keycloak can still use DNS and external integrations.
+
+NetworkPolicy enforcement is provided by the cluster's CNI plugin, not by the
+Kubernetes API. A cluster can accept and display the policy while ignoring it.
+Docker Desktop, kind, and kubeadm installations must therefore use a
+NetworkPolicy-capable CNI and be verified with a connectivity test.
+
+The Docker Desktop cluster used during development accepted the policies but
+did not enforce them: cross-namespace connections succeeded in both directions.
+Consequently, deploying with `--target local` on Docker Desktop isolates names
+and storage, but does not by itself guarantee network isolation. Treat that
+environment as non-enforcing unless the connectivity test below proves
+otherwise. For local enforcement testing, use a cluster configured with a
+policy-capable CNI; for example, create a kind cluster with its default CNI
+disabled and install Calico or Cilium.
+
+With two running stacks, create a controller-managed test Pod in stack A:
+
+```bash
+kubectl create deployment network-policy-test \
+ --namespace stack-a \
+ --image busybox:1.36 \
+ -- sleep 3600
+
+kubectl rollout status deployment/network-policy-test \
+ --namespace stack-a \
+ --timeout 2m
+```
+
+Its own PostgreSQL service must be reachable:
+
+```bash
+kubectl exec --namespace stack-a deployment/network-policy-test -- \
+ nc -z -w 3 postgresql.stack-a.svc.cluster.local 5432
+```
+
+The PostgreSQL service in stack B must time out or be rejected:
+
+```bash
+kubectl exec --namespace stack-a deployment/network-policy-test -- \
+ nc -z -w 3 postgresql.stack-b.svc.cluster.local 5432
+```
+
+The same-stack command must succeed and the cross-stack command must fail. If
+both commands succeed, as observed with the development Docker Desktop cluster,
+the policies are installed but the cluster is not enforcing them. Remove the
+test workload afterwards:
+
+```bash
+kubectl delete deployment network-policy-test --namespace stack-a
+```
+
+Set `networkPolicy.enabled: false` in a stack's `or-setup.yaml` only when
+isolation is deliberately not required. `networkPolicy.additionalIngressFrom`
+accepts additional Kubernetes `NetworkPolicyPeer` entries for trusted sources.
+Ingress exposure permits external traffic to port 8080 on the stack's proxy
+Pod and, for EKS Ingress MQTTS, to its edge-terminated port 1883. HAProxy
+exposure permits external traffic only to ports 8080 and 8443 on that Pod, plus
+port 8883 when MQTTS is explicitly enabled. In both modes, the proxy reaches
+Manager and Keycloak through the same-namespace rule, while direct access from
+other namespaces to Manager, Keycloak, PostgreSQL, and Manager's MQTT listener
+remains denied. `none` adds no public ingress rule.
+
+### Exclusive public hostnames
+
+Namespaces isolate Kubernetes names and storage, but hostnames used for external
+access are shared routing identities. Two stacks declaring the same hostname
+could create conflicting routes or DNS records, directing clients to the wrong
+stack.
+
+With the OpenRemote-managed ExternalDNS configuration, DNS ownership identifies
+the cluster, not an individual stack.
+
+For local deployments, prepare and apply reserve hostnames for Ingress stacks and ExternalDNS-managed endpoints before installing workloads.
+Local HAProxy with unmanaged DNS remains exempt, allowing the localhost:8443 / localhost:9443 workflow. New internal-only stacks reserve nothing.
+Hostnames are normalized to lowercase without a trailing dot. For EKS-specific behavior, see Hostname ownership (README-AWS.md#hostname-ownership)
+
+The reservation registry is ConfigMap `kube-system/openremote-hostnames`. Each
+entry records the owning stack namespace, its Kubernetes UID, and the endpoint
+role (`web` or `mqtts`). Conditional Kubernetes updates reserve each hostname
+and the stack's fixed hostname for that role together: two concurrent applies
+cannot both acquire the same hostname, and one stack cannot concurrently
+acquire two different hostnames for the same endpoint. An Ingress stack may
+therefore own one web hostname and one distinct MQTTS hostname. Retrying with
+the same namespace UID and hostname is safe. Changing either hostname or
+recreating a namespace under the same name does not transfer ownership.
+
+Stack operators need `jq`, permission to get/create/update that ConfigMap,
+and cluster-wide read access to namespaces, Ingresses and Services, in addition
+to the usual stack lifecycle permissions. Kubernetes RBAC cannot restrict
+`create` by resource name; an
+administrator can pre-create the empty registry with label
+`app.kubernetes.io/managed-by: or-stack` and `data: {}` so operators need only
+`get` and `update` on this named ConfigMap. The scripts refuse an unlabelled or
+malformed registry.
+Only trusted deployment operators and automation should have write access to this registry,
+because modifying it can bypass hostname ownership checks.
+
+The lifecycle scripts support stacks created with the current metadata and
+reservation format. Target and exposure come from namespace labels, and hostname ownership
+comes from the reservation and its namespace annotation. Missing target or
+exposure labels are treated as inconsistent state, not inferred from workloads.
+The scripts still inspect other namespaces' reservations and existing Ingress
+rules and ExternalDNS annotations to avoid conflicting with other workloads.
+These checks coordinate these scripts; they are not an admission policy against
+manually created conflicting resources or arbitrary additional hostnames in
+custom chart overrides. Reservations are scoped to one Kubernetes cluster.
+
+Reservations survive uninstall and failed applies. They have no automatic expiry
+or namespace ownerReference: deleting a namespace does not prove that external
+DNS or certificate cleanup finished. Disabling ExternalDNS also retains the
+reservation's DNS cleanup requirement.
+
+For local deployments, `or-stack destroy` releases Ingress reservations with
+unmanaged DNS after Kubernetes cleanup. When ExternalDNS is used, it retains
+the reservation until external cleanup is confirmed. Verify that old public
+endpoints, DNS records, and stack-managed certificates have been removed before
+explicitly releasing it. For example, after cleaning up a local stack that used
+ExternalDNS:
+
+```bash
+./or-stack release-hostname \
+ --name stack-a \
+ --kube-context kind-openremote \
+ --hostname stack-a.example.com \
+ --confirm stack-a
+```
+
+This confirmation asserts that external cleanup is complete; the portable
+command does not contact a DNS provider. It additionally requires the namespace
+to be absent and no conflicting Kubernetes routes to remain, and only removes
+the named stack's reservation. It does not delete DNS or certificates. There is
+no force-takeover flag. For hostname changes, destroy and clean up the old stack,
+release its reservation, then recreate it with the new hostname; back up any data
+you intend to retain before destruction. In-place hostname migration is not supported.
+
+For EKS cleanup and recovery, see
+[Hostname ownership](README-AWS.md#hostname-ownership).
+
+### Uninstalling or destroying a stack
+
+Uninstall a stack's Helm releases while preserving its namespace, credentials,
+PVCs, and data:
+
+```bash
+./or-stack uninstall \
+ --name stack-a \
+ --kube-context docker-desktop
+```
+
+The releases are removed in reverse dependency order, beginning with HAProxy
+when present. Before uninstalling, `or-stack` refuses to continue if the stack
+Secret or a chart-managed PVC does not carry Helm's `keep` resource policy.
+Running `apply` again with the same stack name, context, target, exposure, and
+ports restores the workloads with the retained credentials, data volumes, and
+HAProxy ACME/certificate state.
-The following steps use the first option, managing connections through HAProxy.
+Destroying a stack is the explicit data-purge operation:
-### Create the Persistent Volumes and the required secrets
+```bash
+./or-stack destroy \
+ --name stack-a \
+ --kube-context docker-desktop \
+ --confirm stack-a
+```
+
+Destroy requires both exact-name confirmation and a matching
+`openremote.io/stack` label on the namespace. It uninstalls remaining releases
+and deletes the entire namespace, including every PVC and other namespaced
+resource in it. Backing volume deletion then follows the PV's StorageClass
+reclaim policy; Docker Desktop's default StorageClass currently use `Delete`.
+
+For the same reason, `apply` refuses to adopt a pre-existing namespace that
+does not already have the matching stack label. This prevents an unrelated
+namespace from later becoming eligible for stack destruction.
+
+### DNS ownership
-This is performed via the `or-setup` helm chart.
-Review the `values.yaml` file under the `or-setup` folder and create one with values appropriate for your environment.
+`or-stack` does not call any DNS provider. With the default
+`--dns none`, it adds no DNS metadata. With `--dns external-dns`, it places the
+GA `external-dns.kubernetes.io/hostname` annotation and the
+`openremote.io/managed-dns=true` selection label on exactly one resource:
-You should certainly edit the `basePath` value to point to a folder on your local machine.
-Under macOS, it needs to be located under your home folder (/Users/xxx/...).
+- the proxy `Ingress` for ingress exposure;
+- the proxy `LoadBalancer` Service for HAProxy exposure.
+
+The cluster's ExternalDNS installation must watch `Ingress` and `Service`
+sources and select that label. A higher-level orchestrator may also use
+namespaced `DNSEndpoint` resources for records that do not derive their target
+from an Ingress or Service, such as certificate-validation CNAME records.
+These records are managed by higher-level deployment tools; or-stack itself does not create them.
+For the OpenRemote-managed ExternalDNS installation on EKS, see the EKS cluster documentation (cluster/eks/README.md).
+A bring-your-own controller on another Kubernetes platform can use the same contract. ExternalDNS `0.22` or
+later understands the GA annotation prefix by default; a bring-your-own controller must support that prefix.
+
+DNS ownership requires public exposure and a fully qualified, non-local
+hostname. For example:
+
+```bash
+./or-stack apply \
+ --name stack-a \
+ --kube-context cluster@eu-west-1 \
+ --target eks \
+ --hostname stack-a.example.com \
+ --dns external-dns
+```
+
+The DNS mode is recorded in the stack namespace and shown by
+`or-stack status`. Unlike target or exposure, it may be changed deliberately: reapply
+with `--dns none` to remove the annotation and return the hostname to external
+management. A TXT-registry controller using `sync` policy then removes only
+records it owns. Wait for that reconciliation before creating a manual record
+with the same name or destroying the cluster.
+
+## Manual deployment with Helm (legacy workflow)
+
+The following steps install the component charts directly with Helm in the
+current namespace, using HAProxy for external access. This is the legacy manual
+workflow; use `or-stack` for namespaced stack deployment and lifecycle management,
+as shown in the [local quick start](#tldr-deploy-a-local-stack).
+
+### Create the required secrets
+
+The `or-setup` chart creates `openremote-secret` to hold the PostgreSQL and
+Manager administrator credentials. It generates passwords unless initial
+values are supplied through the chart's `credentials` values.
+
+```bash
+helm install or-setup or-setup
+```
-The `openremote-secrets.yaml` file under `or-setup/templates` creates a secret to hold sensitive configuration from OpenRemote
-(database username and password, Keycloak admin password).
-The values in this file are the default ones, update as required.
+For this legacy manual installation, retrieve the generated Manager password
+directly from the Secret:
```bash
-helm install or-setup or-setup -f your_values.yaml
+kubectl get secret openremote-secret \
+ -o 'go-template={{index .data "admin-password" | base64decode}}{{"\n"}}'
```
+The Manager and PostgreSQL charts create their own namespaced PersistentVolumeClaims.
+By default, they use the cluster's default StorageClass for dynamic provisioning.
+Set `persistence.storageClass` in each chart when the cluster has no default or
+when a specific provisioner must be used.
+
### Install the charts
Install the different charts in order
```bash
-helm install proxy proxy
helm install postgresql postgresql
helm install keycloak keycloak
helm install manager manager
+helm install proxy proxy
```
If running under linux, you must enable the requiresPermissionsFix flag when installing postgresql
@@ -56,17 +519,35 @@ helm install postgresql postgresql --set requiresPermissionsFix=true
## Caveats
-### Persistent Volumes lifecycle
+### Persistent storage lifecycle
-Uninstalling the charts will delete the Persistent Volume Claims but not the PVs.
-Those will go to the 'Released' state, at which point they can't be bound again.
-You need to manually transition them to 'Available' before you re-install the charts
+Manager, PostgreSQL, and HAProxy PVCs have the `helm.sh/resource-policy: keep`
+annotation by default. HAProxy mounts its claim at `/deployment`, the proxy
+image's location for Certbot account data and generated certificates. Replacing
+the Pod or uninstalling/reapplying the release therefore does not needlessly
+request a new certificate. Set `persistence.retain=false` if Helm should delete
+a chart-managed claim during uninstall.
-```bash
-kubectl patch pv manager-data-pv -p '{"spec":{"claimRef": null}}'
-kubectl patch pv postgresql-data-pv -p '{"spec":{"claimRef": null}}'
+Set `persistence.existingClaim` to use a PVC managed outside the chart. When it
+is set, the chart does not create, annotate, or delete that claim.
+
+The proxy can instead load full-chain PEM files (including their private keys)
+from a Secret by setting `certificate.existingSecret`. Secret keys are mounted
+as files under `/data/proxy/certs`; manage and rotate that Secret outside this
+chart. ACME is disabled when this setting is present. If no proxy-managed state
+is required, also disable its PVC:
+
+```yaml
+certificate:
+ existingSecret: stack-certificate
+persistence:
+ enabled: false
```
+Deleting a retained PVC or its namespace is an explicit data-purge operation.
+Whether the backing volume is also deleted then depends on its StorageClass
+reclaim policy.
+
### Release names
Several object names are constructed based on the release name.
@@ -199,39 +680,89 @@ and port forwarding, this is the hostname you need to use for the JMX configurat
#### Accessing MQTT
-MQTTS is accessible on port 8883 through HAProxy.
-Non-TLS access to MQTT requires using a manual port forward to the pod e.g.
-`kubectl port-forward manager-… 1883:1883`
+MQTTS is disabled by default. Enable it with `--mqtts` when applying an HAProxy
+stack:
+
+```bash
+./or-stack apply \
+ --name stack-a \
+ --kube-context docker-desktop \
+ --target local \
+ --hostname localhost \
+ --http-port 8080 \
+ --https-port 8443 \
+ --mqtts \
+ --mqtts-port 18883
+```
+
+The public endpoint is `mqtts://localhost:18883` in this example. HAProxy
+terminates TLS with the same certificate used for HTTPS and forwards the
+connection to Manager port 1883 inside the stack namespace. Repeat `--mqtts`
+on later applies that should retain this exposure. Plaintext MQTT is not
+publicly exposed. On clusters without LoadBalancer support, include the MQTTS
+mapping when port-forwarding, for example `18883:18883` when the Service port is 18883.
#### Using with IDE for development
-It is not possible to run the manager inside the IDE and have HAProxy deployed as a pod in the cluster.
-Use an ingress instead for that scenario.
+Both public exposure modes now use a HAProxy Pod and expect Manager to be
+available through its namespace-local Service. To run Manager in the IDE, use
+the direct development workflow and port-forward its Kubernetes dependencies;
+do not apply either managed public exposure profile.
#### Certificate management
HAProxy is responsible for certificate management (either using the default self-signed or using ACME).
Please refer to the proxy documentation for more information.
-### When using Ingress (no HAProxy pod)
+### When using Ingress
-For this scenario, make sure an Ingress controller is installed in the cluster.
-Review the values files for your deployment scenario, in particular make sure to configure and enable the ingress on keycloak and manager pods.
-Install the different charts in order
+Ingress mode still deploys a HAProxy Pod, but it has a narrower role than in
+HAProxy exposure mode:
+
+```text
+Client -> Ingress controller -> per-stack HAProxy gateway -> Manager/Keycloak
+```
+
+The Kubernetes Ingress targets the proxy Service, never the Manager or
+Keycloak Services. TLS terminates at the Ingress controller, ACME and
+certificate persistence are disabled in the proxy, and HAProxy receives plain
+HTTP before routing `/auth` to Keycloak and other paths to Manager.
+
+This gateway provides a consistent NetworkPolicy boundary across Ingress
+implementations. For web traffic, the default policy allows access from outside
+the stack namespace only to the proxy's HTTP port. HAProxy then reaches Manager and
+Keycloak within the same namespace, keeping their ports inaccessible directly
+from other namespaces. This also works when the Ingress traffic comes from an
+external load balancer without Kubernetes Pod or namespace labels.
+
+The tradeoff is one additional internal HTTP hop and one small proxy Pod per
+stack. TLS termination and external HTTP routing remain the responsibility of
+the Ingress controller; available features and certificate configuration depend
+on that controller. For AWS-specific routing and load-balancer tradeoffs, see
+[Shared HTTPS routing with Ingress](README-AWS.md#shared-https-routing-with-ingress).
+
+Apply this mode through `or-stack` so the proxy, its Ingress, and its
+NetworkPolicy are configured consistently:
```bash
-helm install postgresql postgresql
-helm install keycloak keycloak
-helm install manager manager
+./or-stack apply \
+ --name stack-a \
+ --kube-context <context> \
+ --target local \
+ --exposure ingress \
+ --hostname stack-a.example.com
```
#### Accessing MQTT
-If, in the manager values files, you enable the MQTT/MQTTS services, you can directly access them (by default on port 1883 and 8883).
-**In this configuration, the MQTTS service does not have a certificate and although named MQTTS will only accept MQTT connections !!!**
+The HTTP Ingress does not carry MQTT traffic. `--mqtts` with Ingress exposure
+is supported only on EKS; see [MQTTS with Ingress](README-AWS.md#mqtts-with-ingress)
+for deployment and lifecycle management, or the
+[portable workflow](README-AWS.md#ingress-mqtts-with-externally-managed-readiness)
+when DNS and certificates are managed separately.
-Alternatively, if you do not enable a service, you can use manual port forwarding to the pod e.g.
-`kubectl port-forward manager-… 1883:1883`
+For local Ingress development, a plaintext MQTT connection can use a manual
+port-forward to Manager port 1883. Plaintext MQTT is never publicly exposed.
#### Running a custom project
@@ -298,36 +829,37 @@ Make sure that other ports (e.g. MQTT 1883) are not used / forwarded from pods.
### Changing hostname / adding custom certificate
-The default values file creates the ingress on localhost and uses the default kubernetes fake certificate for its TLS termination.
-You can change your hostname by overriding the appropriate values, e.g. using a custom values files.
-Here is an example of a `values-openremote.yaml` file that you could use
+With `--exposure ingress`, the proxy chart owns the Ingress. Pass its hostname
+to `or-stack` with `--hostname`. For a local controller, configure TLS through
+the stack's `proxy.yaml` override. For example:
```yaml
ingress:
- hosts:
- - host: test.openremote.io
- paths:
- - path: /
- pathType: Prefix
+ hostname: test.openremote.io
tls:
- hosts:
- test.openremote.io
-
-or:
- hostname: "test.openremote.io"
```
-You can then deploy the manager using
+Apply it as part of the complete stack:
```bash
-helm install manager manager -f values-openremote.yaml
+./or-stack apply \
+ --name stack-a \
+ --kube-context <context> \
+ --target local \
+ --exposure ingress \
+ --hostname test.openremote.io \
+ --values-dir ./stacks/stack-a
```
If in addition to changing the hostname, you'd like to use a custom certificate for the TLS termination, you need to create a kubernetes secret with the certificate and reference it from the ingress definition.
Supposing you have the private/public keys available in .PEM encoded files, you create the secret using
```
-kubectl create secret tls or-manager-tls --key test.openremote.io.key --cert test.openremote.io.crt
+kubectl --namespace stack-a create secret tls openremote-web-tls \
+ --key test.openremote.io.key \
+ --cert test.openremote.io.crt
```
With that in place, you can now reference the secret in the ingress configuration.
@@ -335,18 +867,11 @@ The above example file now becomes
```yaml
ingress:
- hosts:
- - host: test.openremote.io
- paths:
- - path: /
- pathType: Prefix
+ hostname: test.openremote.io
tls:
- hosts:
- test.openremote.io
- secretName: or-manager-tls
-
-or:
- hostname: "test.openremote.io"
+ secretName: openremote-web-tls
```
### Connecting to the database
@@ -360,3 +885,42 @@ kubectl port-forward svc/postgresql 5432:5432
```
## Guidelines
+
+## Testing the deployment tools
+
+The regression tests in `kubernetes/test` check chart rendering and deployment
+command behavior without deploying a stack. They use stubs for Kubernetes and
+AWS operations; no running cluster or AWS credentials are required.
+`or-setup-test` uses the real Helm executable to lint and render the charts.
+
+To run the full set below, install Bash, Helm, jq, and Python 3.9 or newer,
+along with standard Unix command-line utilities. The MQTTS readiness tests
+also need OpenSSL or LibreSSL to generate temporary certificates and permission
+to listen on loopback sockets. Run these commands from the repository root:
+
+```bash
+bash kubernetes/test/or-setup-test
+bash kubernetes/test/legacy-haproxy-cleanup-test
+bash kubernetes/test/or-stack-test
+python3 kubernetes/test/hostname-reservations-test
+bash kubernetes/test/or-eks-cluster-test
+bash kubernetes/test/or-eks-stack-test
+bash kubernetes/test/aws-profile-test
+python3 kubernetes/test/mqtts-readiness-test
+```
+
+| Test | Coverage |
+| ----------------------------- | ----------------------------------------------------------------------------------------------------------------- |
+| `or-setup-test` | Helm chart linting and rendered resources for the supported targets and exposure modes. |
+| `legacy-haproxy-cleanup-test` | Legacy EKS HAProxy cleanup deletes retained claims and waits for their volumes before removing the EBS driver. |
+| `or-stack-test` | Portable stack commands, values selection, credentials, and lifecycle safeguards. |
+| `hostname-reservations-test` | Hostname ownership, concurrent reservations, and cleanup/recovery using a persistent, atomic Kubernetes API stub. |
+| `or-eks-cluster-test` | EKS cluster add-on configuration and destruction safeguards. |
+| `or-eks-stack-test` | EKS stack orchestration, DNS and certificate handling, and cleanup safeguards. |
+| `aws-profile-test` | Default credential behavior and explicit AWS profile selection in both EKS CLIs. |
+| `mqtts-readiness-test` | TLS handshake verification, hostname and trust checks, and timeout handling using local test servers. |
+
+These tests do not verify a live cluster's routing, certificate issuance, or
+NetworkPolicy enforcement. After deploying, use the connectivity checks in
+[Network isolation](#network-isolation) to verify that the cluster enforces
+cross-stack isolation.
diff --git a/kubernetes/cluster.yaml b/kubernetes/cluster.yaml
index c1aa80dbd..d337ce7b2 100644
--- a/kubernetes/cluster.yaml
+++ b/kubernetes/cluster.yaml
@@ -27,6 +27,12 @@ nodeGroups:
desiredCapacity: 2
addons:
+ - name: vpc-cni
+ version: latest
+ attachPolicyARNs:
+ - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
+ configurationValues: '{"enableNetworkPolicy":"true"}'
+ resolveConflicts: overwrite
- name: aws-ebs-csi-driver
version: latest
wellKnownPolicies:
diff --git a/kubernetes/cluster/eks/README.md b/kubernetes/cluster/eks/README.md
new file mode 100644
index 000000000..4481da64f
--- /dev/null
+++ b/kubernetes/cluster/eks/README.md
@@ -0,0 +1,396 @@
+# EKS cluster management
+
+`kubernetes/or-eks-cluster` manages the shared EKS cluster, native VPC CNI
+NetworkPolicy enforcement, its shared AWS Load Balancer Controller and ALB
+IngressClass, the `openremote-ebs` StorageClass, and optionally ExternalDNS. It
+does not deploy or delete OpenRemote stacks, stack namespaces, certificates, or
+stack data volumes. ExternalDNS changes records only for stack resources that
+explicitly opt in.
+
+After the shared cluster is ready, use `kubernetes/or-eks-stack` for the normal
+stack workflow.
+
+The current implementation uses the existing `kubernetes/cluster.yaml`
+`eksctl` configuration. This boundary is intended to remain stable when EKS
+infrastructure management moves from `eksctl` to CloudFormation.
+
+## Configuration
+
+The cluster name is deliberately required. The default AWS region is
+`eu-west-1`; no named AWS profile is selected by the CLI.
+
+```bash
+export OR_EKS_CLUSTER_NAME=openremote-test
+export OR_EKS_AWS_REGION=eu-west-1
+```
+
+Optional settings are:
+
+```bash
+export OR_EKS_CONFIG_FILE=/path/to/cluster.yaml
+export OR_EKS_KUBE_CONTEXT=openremote-test@eu-west-1
+export OR_EKS_LOAD_BALANCER_CONTROLLER_CHART_VERSION=1.14.0
+export OR_EKS_SHARED_CERTIFICATE_ARN=arn:aws:acm:eu-west-1:210987654321:certificate/example
+```
+
+ExternalDNS is disabled when no DNS configuration is supplied. Its complete
+managed configuration is:
+
+```bash
+export OR_EKS_EXTERNAL_DNS_HOSTED_ZONE_ID=Z0123456789
+export OR_EKS_EXTERNAL_DNS_DOMAIN=example.com
+export OR_EKS_EXTERNAL_DNS_ASSUME_ROLE_ARN=arn:aws:iam::123456789012:role/openremote-external-dns
+export OR_EKS_EXTERNAL_DNS_OWNER_ID=210987654321-openremote-test-eu-west-1
+
+# Optional
+export OR_EKS_EXTERNAL_DNS_ASSUME_ROLE_EXTERNAL_ID=replace-with-a-random-value
+export OR_EKS_EXTERNAL_DNS_IRSA_ROLE_NAME=openremote-openremote-test-eu-west-1-external-dns
+export OR_EKS_EXTERNAL_DNS_CHART_VERSION=1.21.1
+export OR_EKS_EXTERNAL_DNS_IMAGE_TAG=v0.22.0
+```
+
+The hosted-zone ID, domain, assumed role ARN, and owner ID form one atomic
+configuration: if any is supplied, all four are required. Keep the owner ID
+stable and unique among every ExternalDNS instance that can access the zone.
+Changing it after records exist abandons the old TXT ownership records.
+
+Command-line options override environment variables. Run
+`kubernetes/or-eks-cluster --help` for the complete command reference.
+
+The script does not write AWS credentials or profiles. Both EKS CLIs omit the
+AWS/eksctl `--profile` argument by default, allowing the normal credential chain
+to resolve credentials. Exported access keys and session tokens therefore work
+without a named profile. Standard AWS environment settings remain inherited;
+in particular, unset `AWS_PROFILE` and `AWS_DEFAULT_PROFILE` if you do not want
+environment-based profile selection. Do not export `AWS_PROFILE=''`, which
+AWS CLI can interpret as an empty profile name.
+
+To select a named profile explicitly, pass `--profile <name>` to each EKS CLI
+invocation. `--profile ''` is still accepted and has the same behavior as omitting
+the option; it does not clear inherited AWS environment variables.
+
+## Lifecycle
+
+Run the examples in this section from the repository root.
+
+### Create and inspect a cluster
+
+Create a cluster and a stable kubeconfig context:
+
+```bash
+kubernetes/or-eks-cluster create \
+ --name openremote-test \
+ --region eu-west-1
+```
+
+Inspect it:
+
+```bash
+kubernetes/or-eks-cluster status --name openremote-test
+```
+
+### Reconcile cluster configuration
+
+`apply` currently enables NetworkPolicy enforcement in the existing managed
+VPC CNI configuration, reconciles the AWS Load Balancer Controller and its
+CRDs, applies the shared `openremote-alb` IngressClass and `openremote-ebs`
+StorageClass, and reconciles ExternalDNS when configured. It intentionally does
+not attempt to mutate node-group infrastructure; that reconciliation will be
+implemented by the future CloudFormation cluster definition.
+
+```bash
+kubernetes/or-eks-cluster apply --name openremote-test
+```
+
+The VPC CNI configuration update retains other configuration keys and does not
+change the installed add-on version. If that version does not support native
+NetworkPolicy, `apply` stops and asks for an explicit add-on update rather than
+performing a version upgrade implicitly. New clusters select the latest
+compatible VPC CNI version, give its dedicated IAM role the
+`AmazonEKS_CNI_Policy`, and enable NetworkPolicy during creation.
+
+For an existing managed add-on with a `serviceAccountRoleArn`, `apply` also
+ensures that the `aws-node` ServiceAccount carries the matching IRSA annotation
+before starting an add-on update. This prevents a VPC CNI rollout from losing
+access to EC2 network-interface operations. If no dedicated add-on role exists,
+the script warns that those permissions must instead come from the node IAM
+role. Add-on waiter failures include the EKS health issue and current
+`aws-node` rollout state.
+
+See the AWS documentation for the current
+[VPC CNI NetworkPolicy prerequisites and behavior](https://docs.aws.amazon.com/eks/latest/userguide/cni-network-policy.html)
+and [managed add-on configuration](https://docs.aws.amazon.com/eks/latest/userguide/cni-network-policy-configure.html).
+
+### Destroy a cluster
+
+Remove OpenRemote stacks and decide which data to retain before destroying the
+cluster. Uninstalling workloads alone leaves namespaces, credentials, and PVCs,
+which continue to block cluster destruction. Follow the
+[stack cleanup instructions](../../README-AWS.md#uninstall-restore-or-destroy-a-stack)
+and wait for external DNS and backing-volume cleanup to finish.
+
+Cluster destruction requires an exact-name confirmation and is refused while
+OpenRemote namespaces, Ingresses, LoadBalancer Services, PVCs, PersistentVolumes,
+non-system Pods or workload controllers, namespaced `DNSEndpoint` resources,
+or hostname reservations remain:
+
+```bash
+kubernetes/or-eks-cluster destroy \
+ --name openremote-test \
+ --confirm openremote-test
+```
+
+The shared ConfigMap `kube-system/openremote-hostnames` must contain no remaining
+reservations. A reservation can outlive its namespace after interrupted external
+cleanup. Finish `or-eks-stack destroy` or the documented
+[manual hostname release](../../README.md#exclusive-public-hostnames) before
+retrying cluster destruction; do not discard the registry to bypass this check.
+
+The namespace check also protects internal stacks without public hostnames and
+interrupted preparations. Workload checks include Deployments, StatefulSets,
+DaemonSets, ReplicaSets, ReplicationControllers, Jobs, and CronJobs, even when
+they have no Pods. Controllers in `kube-system`, `kube-public`, and
+`kube-node-lease` are excluded, matching the Pod check. Empty unrelated
+namespaces do not block deletion.
+
+PersistentVolumes block deletion even after their PVCs have gone, so retained
+storage or incomplete cleanup cannot silently lose its Kubernetes metadata.
+The preflight lists blocking namespaces, controllers, and volumes. Inspect and
+resolve them before retrying; for retained storage, preserve the data and
+recovery information you need before removing its Kubernetes metadata. The
+preflight does not delete these resources or discover orphaned AWS resources
+that no longer have Kubernetes metadata.
+
+After this preflight succeeds, deletion bypasses PodDisruptionBudgets while
+draining the node group. This prevents replicated EKS system add-ons such as
+CoreDNS, the EBS CSI controller, and metrics-server from blocking deletion once
+all nodes have been cordoned. The bypass does not weaken the preflight for
+OpenRemote stacks or other non-system workloads.
+
+When the OpenRemote-managed ExternalDNS release exists, the preflight therefore
+requires its Ingress and LoadBalancer Service sources to be gone before the
+script uninstalls the controller. Pass `--skip-external-dns` only when its
+lifecycle belongs to another operator.
+
+### Kubernetes upgrades
+
+Kubernetes minor-version upgrades are intentionally outside this first phase.
+The current `cluster.yaml` uses a self-managed `nodeGroups` entry, so an upgrade
+strategy must be designed separately before the cluster command exposes that
+lifecycle operation.
+
+## Shared ALB ingress
+
+The `openremote-alb` IngressClass uses `IngressClassParams` to place all of its
+Ingresses in one `openremote-stacks` group. This produces one internet-facing
+ALB for the cluster by default and sends traffic directly to Pod IPs through
+the Amazon VPC CNI. Its namespace selector accepts only namespaces labeled
+`app.kubernetes.io/part-of=openremote`; `or-stack` applies that label when
+preparing a stack namespace. The class enforces HTTPS redirection to port 443
+for every group member.
+
+The class defines shared routing behavior and can hold an optional shared
+certificate ARN. Individual stacks define their hostnames, routes, certificate
+selection, and DNS records. For stack deployment and certificate modes, see
+[Shared HTTPS routing with Ingress](../../README-AWS.md#shared-https-routing-with-ingress).
+
+See the controller's
+[IngressClass documentation](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/ingress/ingress_class/)
+for the enforced group and namespace-selector behavior.
+
+### Configure a shared certificate
+
+Supplying `--shared-certificate-arn` during create or apply records an
+externally managed ACM certificate on `openremote-alb`. Later applies that omit
+the option preserve the reference. `--clear-shared-certificate` removes only
+the reference; neither command changes or deletes the certificate. Cluster
+status reports the currently deployed value.
+
+From the repository root, configure the shared certificate with:
+
+```bash
+kubernetes/or-eks-cluster apply \
+ --name openremote-test \
+ --region eu-west-1 \
+ --shared-certificate-arn <acm-certificate-arn>
+```
+
+This configuration supplies a certificate for stacks that select `shared`
+mode. Changing the reference does not change existing stacks' certificate
+ownership modes, and cluster destruction does not delete the certificate.
+For stack selection and certificate rotation, see
+[Use a cluster-configured shared certificate](../../README-AWS.md#use-a-cluster-configured-shared-certificate).
+
+## Automatic Route 53 records
+
+The optional managed ExternalDNS installation is cluster-scoped. It watches
+`Ingress`, `Service`, and namespaced `DNSEndpoint` resources, but filters all
+of them by `openremote.io/managed-dns=true`. It uses only public Route 53 zones
+and is restricted to the configured zone ID and domain. It uses TXT ownership
+with a cluster-unique owner ID and the `sync` policy, so deleting a source
+removes its owned record but not another controller's or an operator's records.
+The chart and image are pinned independently; the current defaults are chart
+`1.21.1` and ExternalDNS `v0.22.0`.
+
+Hostnames managed by this controller must be strict subdomains of the configured
+domain: for `example.com`, both `stack.example.com` and `staging.stack.example.com` qualify,
+but `example.com` does not. Otherwise ExternalDNS's record-type prefix places
+the ownership TXT name outside the domain allowed by the DNS role.
+
+Cluster reconciliation explicitly applies the pinned chart's `DNSEndpoint`
+CRD before installing ExternalDNS and waits for it to become established. The
+chart grants the controller read/watch access to `DNSEndpoint` objects and
+status-update access when its `crd` source is enabled. Cluster status reports
+whether both the CRD and source are active.
+
+For `Ingress` and `Service` sources, the controller supports the GA
+`external-dns.kubernetes.io/hostname` annotation alongside the required
+selection label. `DNSEndpoint` resources specify DNS records directly.
+
+For stack deployment and automated DNS and HTTPS readiness checks, see
+[Namespaced stacks](../../README-AWS.md#namespaced-stacks). For how stacks
+declare DNS ownership on their resources, see
+[DNS ownership](../../README.md#dns-ownership).
+
+### Cross-account IAM bootstrap
+
+Two short-lived credential chains remain separate:
+
+```text
+ExternalDNS Pod -> cluster-account IRSA role -> DNS-account Route 53 role
+```
+
+The cluster role may call only `sts:AssumeRole` on the configured DNS role. The
+DNS role may change only `A`, `AAAA`, `CNAME`, and `TXT` records in one hosted
+zone and only at or below one DNS suffix. No static AWS credentials are stored
+in Kubernetes.
+
+Create the DNS-account role with
+[`external-dns-route53-role.yaml`](external-dns-route53-role.yaml). The role's
+trust policy names the deterministic cluster role through an ARN condition, so
+this CloudFormation stack can be deployed before the EKS cluster or IRSA role
+exists. For example:
+
+```bash
+export CLUSTER_NAME=openremote-test
+export CLUSTER_REGION=eu-west-1
+export CLUSTER_AWS_PROFILE=or
+export DNS_AWS_PROFILE=dns
+export DNS_ZONE_ID=Z0123456789
+export DNS_DOMAIN=example.com
+export EXTERNAL_DNS_EXTERNAL_ID=replace-with-a-random-value
+export EXTERNAL_DNS_IAM_STACK=openremote-test-external-dns
+
+export CLUSTER_ACCOUNT_ID="$(aws sts get-caller-identity \
+ --profile "$CLUSTER_AWS_PROFILE" \
+ --query Account \
+ --output text)"
+export EXTERNAL_DNS_IRSA_ROLE_NAME="openremote-${CLUSTER_NAME}-${CLUSTER_REGION}-external-dns"
+
+aws cloudformation deploy \
+ --profile "$DNS_AWS_PROFILE" \
+ --region "$CLUSTER_REGION" \
+ --stack-name "$EXTERNAL_DNS_IAM_STACK" \
+ --template-file external-dns-route53-role.yaml \
+ --capabilities CAPABILITY_NAMED_IAM \
+ --parameter-overrides \
+ HostedZoneId="$DNS_ZONE_ID" \
+ DomainName="$DNS_DOMAIN" \
+ ClusterAccountId="$CLUSTER_ACCOUNT_ID" \
+ ClusterExternalDNSRoleName="$EXTERNAL_DNS_IRSA_ROLE_NAME" \
+ ExternalId="$EXTERNAL_DNS_EXTERNAL_ID"
+
+export EXTERNAL_DNS_ROLE_ARN="$(aws cloudformation describe-stacks \
+ --profile "$DNS_AWS_PROFILE" \
+ --region "$CLUSTER_REGION" \
+ --stack-name "$EXTERNAL_DNS_IAM_STACK" \
+ --query 'Stacks[0].Outputs[?OutputKey==`RoleArn`].OutputValue | [0]' \
+ --output text)"
+```
+
+Run those commands from `kubernetes/cluster/eks`, or pass an absolute template
+path. Supplying `RoleName` to the template is optional. If no external ID is
+wanted, omit `ExternalId` from the parameter overrides and from the cluster
+command.
+
+Configure a new or existing cluster with the same source role name:
+
+```bash
+../../or-eks-cluster apply \
+ --name "$CLUSTER_NAME" \
+ --region "$CLUSTER_REGION" \
+ --profile "$CLUSTER_AWS_PROFILE" \
+ --external-dns-zone-id "$DNS_ZONE_ID" \
+ --external-dns-domain "$DNS_DOMAIN" \
+ --external-dns-role-arn "$EXTERNAL_DNS_ROLE_ARN" \
+ --external-dns-external-id "$EXTERNAL_DNS_EXTERNAL_ID" \
+ --external-dns-owner-id "${CLUSTER_ACCOUNT_ID}-${CLUSTER_NAME}-${CLUSTER_REGION}" \
+ --external-dns-irsa-role-name "$EXTERNAL_DNS_IRSA_ROLE_NAME"
+```
+
+Use `create` instead of `apply` when the cluster does not exist. The DNS-account
+CloudFormation stack must already exist so ExternalDNS can assume its output
+role when Helm waits for the Deployment. `or-eks-cluster` creates or updates
+the cluster-side role and `kube-system/external-dns` ServiceAccount through
+eksctl, then installs the controller.
+
+For a controller managed outside this command, pass `--skip-external-dns`.
+`or-stack --dns external-dns` remains usable, provided that controller watches
+Ingresses and Services, selects `openremote.io/managed-dns=true`, uses the GA
+annotation prefix, and has a safe ownership registry. The skip option also
+prevents `or-eks-cluster destroy` from explicitly uninstalling that Helm
+release.
+
+### Migrating existing DNS records
+
+ExternalDNS does not silently adopt a manually created record. Migrate existing
+records with this dry-run-first sequence:
+
+1. Reconcile the cluster using the complete DNS configuration plus
+ `--external-dns-dry-run`.
+2. Reapply each intended stack using `or-stack apply` with its existing
+ configuration and `--dns external-dns`. Use the portable command here because
+ `or-eks-stack apply` rejects a controller running in dry-run mode.
+3. Inspect the proposed changes with `kubectl --namespace kube-system logs
+deployment/openremote-external-dns`.
+4. Capture the exact current record values, then delete only the manually
+ managed `A`, `AAAA`, or `CNAME` records for those stack hostnames. Do not
+ delete ACM validation records or unrelated TXT records.
+5. While still in dry-run, confirm that the logs propose creating the expected
+ alias and TXT ownership records.
+6. Run `or-eks-cluster apply` again with the same DNS configuration and without
+ `--external-dns-dry-run`.
+7. Verify the new alias and TXT records in Route 53 and test the endpoint.
+
+### Monitoring and removing ExternalDNS
+
+`or-eks-cluster status` reports the ExternalDNS Helm health, TXT owner ID,
+dry-run state, and IRSA role. The controller's AWS provider and TXT registry are
+documented in the upstream
+[AWS tutorial](https://kubernetes-sigs.github.io/external-dns/latest/docs/tutorials/aws/)
+and [registry documentation](https://kubernetes-sigs.github.io/external-dns/latest/docs/registry/registry/).
+
+ExternalDNS waits for the ALB or NLB hostname in a source resource's status,
+then creates the Route 53 alias and its TXT ownership record. When a source
+resource or its DNS declaration is removed, the controller's event-triggered
+sync removes only records carrying its owner ID.
+
+Keep the controller running until its managed DNS records have been removed.
+Wait for the public alias and TXT records to disappear before shutting down the
+cluster or creating manual replacements. For stack and certificate cleanup,
+follow the [stack lifecycle instructions](../../README-AWS.md#uninstall-restore-or-destroy-a-stack).
+The [cluster destruction checks](#destroy-a-cluster) prevent controller removal
+while managed source resources remain.
+
+The DNS-account CloudFormation stack is deliberately not deleted with the
+cluster. It can be retained for a recreated cluster using the same source role
+name, or deleted explicitly in the DNS account after all managed records are
+gone.
+
+## Persistent storage
+
+The `openremote-ebs` StorageClass uses the standard EBS CSI driver, encrypted
+`gp3` volumes, and `WaitForFirstConsumer` binding. Its `Delete` reclaim policy
+removes the EBS volume when its PVC is explicitly deleted. The component charts
+retain their PVCs during an ordinary Helm uninstall, and cluster destruction is
+blocked while any PVC or PersistentVolume remains.
diff --git a/kubernetes/cluster/eks/external-dns-route53-role.yaml b/kubernetes/cluster/eks/external-dns-route53-role.yaml
new file mode 100644
index 000000000..d09a3cffd
--- /dev/null
+++ b/kubernetes/cluster/eks/external-dns-route53-role.yaml
@@ -0,0 +1,108 @@
+AWSTemplateFormatVersion: "2010-09-09"
+Description: >-
+ Cross-account Route 53 role assumed by the OpenRemote ExternalDNS IRSA role.
+
+Parameters:
+ HostedZoneId:
+ Type: String
+ Description: Public Route 53 hosted-zone ID without the /hostedzone/ prefix.
+ AllowedPattern: ^Z[A-Z0-9]+$
+ DomainName:
+ Type: String
+ Description: >-
+ Lowercase DNS suffix that ExternalDNS may change, without a trailing dot.
+ AllowedPattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
+ ClusterAccountId:
+ Type: String
+ Description: AWS account containing the EKS cluster-side IRSA role.
+ AllowedPattern: ^[0-9]{12}$
+ ClusterExternalDNSRoleName:
+ Type: String
+ Description: >-
+ Exact IRSA role name used by or-eks-cluster in the EKS account.
+ MinLength: 1
+ MaxLength: 64
+ AllowedPattern: ^[A-Za-z0-9_+=,.@-]+$
+ ExternalId:
+ Type: String
+ Description: Optional external ID required when the cluster role assumes this role.
+ Default: ""
+ NoEcho: true
+ RoleName:
+ Type: String
+ Description: Optional fixed name for the role created in the DNS account.
+ Default: ""
+ MaxLength: 64
+ AllowedPattern: ^$|^[A-Za-z0-9_+=,.@-]+$
+
+Conditions:
+ HasExternalId: !Not [!Equals [!Ref ExternalId, ""]]
+ HasRoleName: !Not [!Equals [!Ref RoleName, ""]]
+
+Resources:
+ ExternalDNSRoute53Role:
+ Type: AWS::IAM::Role
+ Properties:
+ Description: !Sub OpenRemote ExternalDNS access to ${DomainName}
+ RoleName: !If [HasRoleName, !Ref RoleName, !Ref "AWS::NoValue"]
+ AssumeRolePolicyDocument:
+ Version: "2012-10-17"
+ Statement:
+ - Sid: AllowClusterExternalDNSRole
+ Effect: Allow
+ Principal:
+ AWS: !Sub arn:${AWS::Partition}:iam::${ClusterAccountId}:root
+ Action: sts:AssumeRole
+ Condition: !If
+ - HasExternalId
+ - ArnEquals:
+ aws:PrincipalArn: !Sub >-
+ arn:${AWS::Partition}:iam::${ClusterAccountId}:role/${ClusterExternalDNSRoleName}
+ StringEquals:
+ sts:ExternalId: !Ref ExternalId
+ - ArnEquals:
+ aws:PrincipalArn: !Sub >-
+ arn:${AWS::Partition}:iam::${ClusterAccountId}:role/${ClusterExternalDNSRoleName}
+ Policies:
+ - PolicyName: ManageExternalDNSRecords
+ PolicyDocument:
+ Version: "2012-10-17"
+ Statement:
+ - Sid: ChangeRecordsInSelectedZoneAndDomain
+ Effect: Allow
+ Action: route53:ChangeResourceRecordSets
+ Resource: !Sub >-
+ arn:${AWS::Partition}:route53:::hostedzone/${HostedZoneId}
+ Condition:
+ ForAllValues:StringLike:
+ route53:ChangeResourceRecordSetsNormalizedRecordNames:
+ - !Ref DomainName
+ - !Sub "*.${DomainName}"
+ ForAllValues:StringEquals:
+ route53:ChangeResourceRecordSetsRecordTypes:
+ - A
+ - AAAA
+ - CNAME
+ - TXT
+ route53:ChangeResourceRecordSetsActions:
+ - CREATE
+ - UPSERT
+ - DELETE
+ - Sid: ReadRoute53Metadata
+ Effect: Allow
+ Action:
+ - route53:GetHostedZone
+ - route53:ListHostedZones
+ - route53:ListHostedZonesByName
+ - route53:ListResourceRecordSets
+ - route53:ListTagsForResource
+ Resource: "*"
+
+Outputs:
+ RoleArn:
+ Description: Pass this ARN to or-eks-cluster --external-dns-role-arn.
+ Value: !GetAtt ExternalDNSRoute53Role.Arn
+ ClusterSourceRoleArn:
+ Description: Exact cluster-account role allowed to assume this role.
+ Value: !Sub >-
+ arn:${AWS::Partition}:iam::${ClusterAccountId}:role/${ClusterExternalDNSRoleName}
diff --git a/kubernetes/cluster/eks/ingress-class.yaml b/kubernetes/cluster/eks/ingress-class.yaml
new file mode 100644
index 000000000..613c8e610
--- /dev/null
+++ b/kubernetes/cluster/eks/ingress-class.yaml
@@ -0,0 +1,24 @@
+apiVersion: elbv2.k8s.aws/v1beta1
+kind: IngressClassParams
+metadata:
+ name: openremote-alb
+spec:
+ group:
+ name: openremote-stacks
+ namespaceSelector:
+ matchLabels:
+ app.kubernetes.io/part-of: openremote
+ scheme: internet-facing
+ sslRedirectPort: "443"
+ targetType: ip
+---
+apiVersion: networking.k8s.io/v1
+kind: IngressClass
+metadata:
+ name: openremote-alb
+spec:
+ controller: ingress.k8s.aws/alb
+ parameters:
+ apiGroup: elbv2.k8s.aws
+ kind: IngressClassParams
+ name: openremote-alb
diff --git a/kubernetes/cluster/eks/storage-class.yaml b/kubernetes/cluster/eks/storage-class.yaml
new file mode 100644
index 000000000..d750e4386
--- /dev/null
+++ b/kubernetes/cluster/eks/storage-class.yaml
@@ -0,0 +1,11 @@
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+ name: openremote-ebs
+provisioner: ebs.csi.aws.com
+parameters:
+ type: gp3
+ encrypted: "true"
+reclaimPolicy: Delete
+allowVolumeExpansion: true
+volumeBindingMode: WaitForFirstConsumer
diff --git a/kubernetes/eks-cleanup-haproxy.sh b/kubernetes/eks-cleanup-haproxy.sh
index fd11a8f7b..d19af6b79 100755
--- a/kubernetes/eks-cleanup-haproxy.sh
+++ b/kubernetes/eks-cleanup-haproxy.sh
@@ -8,6 +8,10 @@ if [ -z "$CLUSTER_VPC_ID" ] || [ "$CLUSTER_VPC_ID" = "None" ]; then
exit 1
fi
+MANAGER_PV_NAME=$(kubectl get pvc manager -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)
+PSQL_PV_NAME=$(kubectl get pvc postgresql -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)
+PROXY_PV_NAME=$(kubectl get pvc proxy -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)
+
DNS_NAME=$(aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?VpcId=='$CLUSTER_VPC_ID' && Type=='network' && Scheme=='internet-facing'].DNSName | [0]")
HOSTED_ZONE_ID=$(aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?VpcId=='$CLUSTER_VPC_ID' && Type=='network' && Scheme=='internet-facing'].CanonicalHostedZoneId | [0]")
@@ -44,12 +48,14 @@ while aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?Vp
done
helm uninstall aws-load-balancer-controller -n kube-system
-MANAGER_VOLUMEID=$(kubectl get pv manager-data-pv -o=jsonpath='{.spec.awsElasticBlockStore.volumeID}')
-PSQL_VOLUMEID=$(kubectl get pv postgresql-data-pv -o=jsonpath='{.spec.awsElasticBlockStore.volumeID}')
-kubectl delete pv manager-data-pv
-kubectl delete pv postgresql-data-pv
-aws ec2 delete-volume --volume-id $MANAGER_VOLUMEID
-aws ec2 delete-volume --volume-id $PSQL_VOLUMEID
+# The component charts retain their PVCs on Helm uninstall. This legacy full
+# cleanup explicitly purges them; the StorageClass then deletes their EBS volumes.
+kubectl delete pvc manager postgresql proxy --ignore-not-found --wait=true
+for pv_name in "$MANAGER_PV_NAME" "$PSQL_PV_NAME" "$PROXY_PV_NAME"; do
+ if [ -n "$pv_name" ] && kubectl get "pv/$pv_name" >/dev/null 2>&1; then
+ kubectl wait --for=delete "pv/$pv_name" --timeout=5m
+ fi
+done
helm uninstall or-setup
# Manually deleting all addons, this should not be required but otherwise the delete cluster fails
diff --git a/kubernetes/eks-cleanup.sh b/kubernetes/eks-cleanup.sh
index 491a28663..64757b5dd 100755
--- a/kubernetes/eks-cleanup.sh
+++ b/kubernetes/eks-cleanup.sh
@@ -9,6 +9,8 @@ if [ -z "$CLUSTER_VPC_ID" ] || [ "$CLUSTER_VPC_ID" = "None" ]; then
fi
CERTIFICATE_ARN=$(kubectl get ingress manager -o jsonpath='{.metadata.annotations.\alb\.ingress\.kubernetes\.io\/certificate-arn}')
+MANAGER_PV_NAME=$(kubectl get pvc manager -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)
+PSQL_PV_NAME=$(kubectl get pvc postgresql -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)
DNS_RECORD_NAME=$(aws acm describe-certificate --certificate-arn $CERTIFICATE_ARN --profile or --query "Certificate.DomainValidationOptions[0].ResourceRecord.Name")
DNS_RECORD_VALUE=$(aws acm describe-certificate --certificate-arn $CERTIFICATE_ARN --profile or --query "Certificate.DomainValidationOptions[0].ResourceRecord.Value")
@@ -100,12 +102,14 @@ if (( i > MAX_RETRIES )); then
echo "⚠️ Timed out trying to delete certificate $CERTIFICATE_ARN after $MAX_RETRIES attempts."
fi
-MANAGER_VOLUMEID=$(kubectl get pv manager-data-pv -o=jsonpath='{.spec.awsElasticBlockStore.volumeID}')
-PSQL_VOLUMEID=$(kubectl get pv postgresql-data-pv -o=jsonpath='{.spec.awsElasticBlockStore.volumeID}')
-kubectl delete pv manager-data-pv
-kubectl delete pv postgresql-data-pv
-aws ec2 delete-volume --volume-id $MANAGER_VOLUMEID
-aws ec2 delete-volume --volume-id $PSQL_VOLUMEID
+# The component charts retain their PVCs on Helm uninstall. This legacy full
+# cleanup explicitly purges them; the StorageClass then deletes their EBS volumes.
+kubectl delete pvc manager postgresql --ignore-not-found --wait=true
+for pv_name in "$MANAGER_PV_NAME" "$PSQL_PV_NAME"; do
+ if [ -n "$pv_name" ] && kubectl get "pv/$pv_name" >/dev/null 2>&1; then
+ kubectl wait --for=delete "pv/$pv_name" --timeout=5m
+ fi
+done
helm uninstall or-setup
# Manually deleting all addons, this should not be required but otherwise the delete cluster fails
diff --git a/kubernetes/eks-setup-haproxy.sh b/kubernetes/eks-setup-haproxy.sh
index 8d9399f63..6ffb76f02 100755
--- a/kubernetes/eks-setup-haproxy.sh
+++ b/kubernetes/eks-setup-haproxy.sh
@@ -15,15 +15,14 @@ helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
--set serviceAccount.create=false \
--set serviceAccount.name=aws-load-balancer-controller
-PSQL_VOLUMEID=$(aws ec2 create-volume --size 1 --availability-zone eu-west-1a --tag-specifications "ResourceType=volume,Tags=[{Key=Name,Value=psql-data}]" --query VolumeId)
-MANAGER_VOLUMEID=$(aws ec2 create-volume --size 1 --availability-zone eu-west-1a --tag-specifications "ResourceType=volume,Tags=[{Key=Name,Value=manager-data}]" --query VolumeId)
-
# Wait for AWS LB ctrl to be ready
kubectl rollout status deployment aws-load-balancer-controller -n kube-system --timeout=300s
CLUSTER_DNS=$(kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}:{.spec.ports[?(@.name=="dns")].port}')
-helm install or-setup or-setup --set aws.enabled=true --set aws.managerVolumeId=$MANAGER_VOLUMEID --set aws.psqlVolumeId=$PSQL_VOLUMEID
+kubectl apply -f cluster/eks/storage-class.yaml
+
+helm install or-setup or-setup
helm install proxy proxy -f proxy/values-eks.yaml --set or.nameserver=$CLUSTER_DNS --set or.hostname=$FQDN
diff --git a/kubernetes/eks-setup.sh b/kubernetes/eks-setup.sh
index a781fbe8b..9589ae823 100755
--- a/kubernetes/eks-setup.sh
+++ b/kubernetes/eks-setup.sh
@@ -15,13 +15,12 @@ helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
--set serviceAccount.create=false \
--set serviceAccount.name=aws-load-balancer-controller
-PSQL_VOLUMEID=$(aws ec2 create-volume --size 1 --availability-zone eu-west-1a --tag-specifications "ResourceType=volume,Tags=[{Key=Name,Value=psql-data}]" --query VolumeId)
-MANAGER_VOLUMEID=$(aws ec2 create-volume --size 1 --availability-zone eu-west-1a --tag-specifications "ResourceType=volume,Tags=[{Key=Name,Value=manager-data}]" --query VolumeId)
-
# Wait for AWS LB ctrl to be ready
kubectl rollout status deployment aws-load-balancer-controller -n kube-system --timeout=300s
-helm install or-setup or-setup --set aws.enabled=true --set aws.managerVolumeId=$MANAGER_VOLUMEID --set aws.psqlVolumeId=$PSQL_VOLUMEID
+kubectl apply -f cluster/eks/storage-class.yaml
+
+helm install or-setup or-setup
CERTIFICATE_ARN=$(aws acm request-certificate --domain-name $FQDN --subject-alternative-names $MQTTS_FQDN --validation-method DNS --profile or --query "CertificateArn" --output text)
diff --git a/kubernetes/keycloak/templates/NOTES.txt b/kubernetes/keycloak/templates/NOTES.txt
index 3f1d26c6f..26ff386be 100644
--- a/kubernetes/keycloak/templates/NOTES.txt
+++ b/kubernetes/keycloak/templates/NOTES.txt
@@ -1,2 +1,2 @@
OpenRemote Keycloak {{ .Chart.AppVersion }}
-Visit https://{{ .Values.or.hostname }}/auth
+Visit {{ include "keycloak.publicUrl" . }}
diff --git a/kubernetes/keycloak/templates/_helpers.tpl b/kubernetes/keycloak/templates/_helpers.tpl
index 5ae124b1b..1d2670433 100644
--- a/kubernetes/keycloak/templates/_helpers.tpl
+++ b/kubernetes/keycloak/templates/_helpers.tpl
@@ -5,6 +5,19 @@ Expand the name of the chart.
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
+{{/*
+The CLI supplies a complete public URL, including any custom port and /auth.
+Keep the existing HTTPS /auth default for direct Helm users and legacy scripts
+that supply only a hostname.
+*/}}
+{{- define "keycloak.publicUrl" -}}
+{{- if or (hasPrefix "https://" .Values.or.hostname) (hasPrefix "http://" .Values.or.hostname) -}}
+{{- .Values.or.hostname -}}
+{{- else -}}
+{{- printf "https://%s/auth" .Values.or.hostname -}}
+{{- end -}}
+{{- end }}
+
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
diff --git a/kubernetes/keycloak/templates/deployment.yaml b/kubernetes/keycloak/templates/deployment.yaml
index 693e59138..e7fa4483b 100644
--- a/kubernetes/keycloak/templates/deployment.yaml
+++ b/kubernetes/keycloak/templates/deployment.yaml
@@ -77,7 +77,7 @@ spec:
- name: KC_DB_POOL_MAX_SIZE
value: "{{ .Values.or.postgresql.poolMaxSize }}"
- name: KC_HOSTNAME
- value: "https://{{ .Values.or.hostname }}/auth"
+ value: {{ include "keycloak.publicUrl" . | quote }}
- name: KC_BOOTSTRAP_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
diff --git a/kubernetes/keycloak/templates/ingress.yaml b/kubernetes/keycloak/templates/ingress.yaml
index aa0781802..0ee56a4b6 100644
--- a/kubernetes/keycloak/templates/ingress.yaml
+++ b/kubernetes/keycloak/templates/ingress.yaml
@@ -27,7 +27,7 @@ spec:
{{- end }}
rules:
{{- range .Values.ingress.hosts }}
- - host: {{ .host | quote }}
+ - host: {{ default .host $.Values.ingress.hostname | quote }}
http:
paths:
{{- range .paths }}
diff --git a/kubernetes/keycloak/values.yaml b/kubernetes/keycloak/values.yaml
index f0f776ad7..882b21bff 100644
--- a/kubernetes/keycloak/values.yaml
+++ b/kubernetes/keycloak/values.yaml
@@ -64,6 +64,8 @@ service:
ingress:
enabled: false
className: "nginx"
+ # When set, overrides the host of every rule without replacing its paths.
+ hostname: ""
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
@@ -124,7 +126,9 @@ or:
# - name: foo
# value: bar
- # Configure web server
+ # Bare hostnames become https://<hostname>/auth; full URLs are used unchanged.
+ # Public stacks set a full HTTPS URL (including /auth and any custom port)
+ # so internal discovery and browser requests use the same token issuer.
hostname: "localhost"
port: "-1"
diff --git a/kubernetes/lib/hostname-reservations b/kubernetes/lib/hostname-reservations
new file mode 100644
index 000000000..61a77c22c
--- /dev/null
+++ b/kubernetes/lib/hostname-reservations
@@ -0,0 +1,234 @@
+#!/usr/bin/env bash
+
+# Shared by the portable and EKS lifecycle commands. A namespace isolates resource
+# names, but not ALB host rules or ExternalDNS records. In particular, ExternalDNS's
+# TXT owner identifies the cluster, not an individual stack. Two stacks claiming
+# the same hostname could therefore send users to the wrong Manager.
+#
+# A scan is necessary for pre-existing routes, but cannot arbitrate concurrent
+# applies. This registry uses Kubernetes create/resourceVersion conflict detection
+# to reserve both a hostname and a stack's immutable hostname in one transaction.
+# Keep it outside stack namespaces, without ownerReferences or expiry: namespace
+# deletion is not proof that external DNS/certificate cleanup has completed.
+# No CRD, controller, AWS credentials, or process-local lock is required.
+
+HOSTNAME_REGISTRY_NAME=openremote-hostnames
+HOSTNAME_REGISTRY_NAMESPACE=kube-system
+
+read_hostname_registry() {
+ local registry
+ require_command jq
+ registry="$(kubectl --context "$KUBE_CONTEXT" --namespace "$HOSTNAME_REGISTRY_NAMESPACE" \
+ get configmap "$HOSTNAME_REGISTRY_NAME" --ignore-not-found -o json)" || die \
+ "Unable to read hostname reservations; refusing to continue without ownership checks."
+ if [ -z "$registry" ]; then
+ printf '%s\n' '{"apiVersion":"v1","kind":"ConfigMap","metadata":{"name":"openremote-hostnames","namespace":"kube-system","labels":{"app.kubernetes.io/managed-by":"or-stack"}},"data":{}}'
+ return
+ fi
+ jq -e '.metadata.labels["app.kubernetes.io/managed-by"] == "or-stack" and
+ ((.data // {}) | type == "object" and all(.[];
+ fromjson | (.namespace | type == "string" and length > 0) and
+ (.namespaceUID | type == "string" and length > 0) and
+ (.target == "eks" or .target == "local") and
+ (.dns == "none" or .dns == "external-dns") and
+ (.exposure == "haproxy" or .exposure == "ingress") and
+ (.endpoint == null or .endpoint == "web" or .endpoint == "mqtts")))' \
+ <<< "$registry" >/dev/null || die \
+ "ConfigMap kube-system/openremote-hostnames is unowned or malformed; refusing to adopt it."
+ printf '%s\n' "$registry"
+}
+
+# Return failure only for a concurrent update. Other errors must not be mistaken
+# for contention, especially an RBAC denial or an unavailable API server.
+write_hostname_registry() {
+ local registry="$1"
+ local operation=create
+ local output
+ if jq -e '.metadata.resourceVersion != null' <<< "$registry" >/dev/null; then
+ operation=replace
+ fi
+ if output="$(kubectl --context "$KUBE_CONTEXT" --namespace "$HOSTNAME_REGISTRY_NAMESPACE" \
+ "$operation" -f - <<< "$registry" 2>&1)"; then
+ return 0
+ fi
+ case "$output" in
+ *'(AlreadyExists)'*|*'(Conflict)'*) return 1 ;;
+ *) die "Unable to write hostname reservations: $output" ;;
+ esac
+}
+
+stack_endpoint_hostname_reservation() {
+ local endpoint="$1"
+ local registry
+ registry="$(read_hostname_registry)" || return 1
+ jq -c --arg stack "$STACK_NAME" --arg endpoint "$endpoint" \
+ '[(.data // {} | to_entries[]) | (.value | fromjson) + {hostname: .key} |
+ select(.namespace == $stack and (.endpoint // "web") == $endpoint)] |
+ if length > 1 then error("multiple hostname reservations for one stack endpoint")
+ else .[0] // empty end' <<< "$registry"
+}
+
+stack_hostname_reservation() {
+ stack_endpoint_hostname_reservation web
+}
+
+stack_mqtts_hostname_reservation() {
+ stack_endpoint_hostname_reservation mqtts
+}
+
+stack_hostname_reservations() {
+ local registry
+ registry="$(read_hostname_registry)" || return 1
+ jq -c --arg stack "$STACK_NAME" \
+ '[(.data // {} | to_entries[]) | (.value | fromjson) +
+ {hostname: .key, endpoint: (.value | fromjson | .endpoint // "web")} |
+ select(.namespace == $stack)] | sort_by(.endpoint, .hostname)[]' <<< "$registry"
+}
+
+check_existing_hostname_routes() {
+ local hostname="$1"
+ local allowed_namespace="$2"
+ local routes
+ local conflict
+ # Inspect routes from any cluster workload, including external tools using
+ # either annotation prefix. This is conflict detection, not stack adoption.
+ # Ingress rules matter even when DNS is maintained outside Kubernetes.
+ routes="$(kubectl --context "$KUBE_CONTEXT" get ingress,services --all-namespaces -o json)" || die \
+ "Unable to inspect existing public routes before reserving '$hostname'."
+ conflict="$(jq -r --arg host "$hostname" --arg allowed "$allowed_namespace" '
+ def names:
+ (.spec.rules[]?.host // empty),
+ ((.metadata.annotations["external-dns.kubernetes.io/hostname"] // "") | split(",")[]),
+ ((.metadata.annotations["external-dns.alpha.kubernetes.io/hostname"] // "") | split(",")[]);
+ def normalized: ascii_downcase | gsub("^\\s+|\\s+$"; "") | rtrimstr(".");
+ .items[] | . as $resource | names | normalized | select(length > 0) |
+ . as $declared | select(. == $host or (startswith("*.") and ($host | endswith($declared[1:])))) |
+ select($resource.metadata.namespace != $allowed) |
+ "\($resource.kind) \($resource.metadata.namespace)/\($resource.metadata.name)"' \
+ <<< "$routes")" || die "Unable to parse existing hostname declarations."
+ [ -z "$conflict" ] || die \
+ "Hostname '$hostname' conflicts with existing routing: $conflict"
+}
+
+reserve_stack_endpoint_hostname() {
+ local hostname="$1"
+ local endpoint="$2"
+ local reservation_annotation="$3"
+ local namespace_json namespace_uid recorded_hostname namespaces conflict registry owner replacement attempt
+ local previously_reserved
+ require_command jq
+ namespace_json="$(kubectl --context "$KUBE_CONTEXT" get namespace "$STACK_NAME" -o json)" || die \
+ "Unable to inspect namespace identity before reserving its hostname."
+ # A newly created namespace can have no annotations map. Read optional values
+ # through jq, which handles missing/null maps, rather than indexing a nil Go
+ # template value. Reuse the same response for namespace identity below.
+ previously_reserved="$(jq -r --arg annotation "$reservation_annotation" \
+ '.metadata.annotations[$annotation] // empty' \
+ <<< "$namespace_json")" || die "Unable to read the stack's hostname reservation metadata."
+ if [ "$endpoint" = web ] && [ -z "$previously_reserved" ] && { [ "$EXPOSURE" = none ] || \
+ { [ "$TARGET" = local ] && [ "$EXPOSURE" = haproxy ] && [ "$DNS_MODE" = none ]; }; }; then
+ # The registry write may have succeeded before namespace annotation failed.
+ # Changing to internal/local exposure must not bypass that retained claim.
+ previously_reserved="$(stack_hostname_reservation | jq -r '.hostname // empty')" || die "Unable to inspect retained hostname ownership."
+ [ -n "$previously_reserved" ] || return 0
+ fi
+ namespace_uid="$(jq -er --arg stack "$STACK_NAME" '
+ select(.metadata.labels["openremote.io/stack"] == $stack) |
+ .metadata.uid | select(type == "string" and length > 0)' <<< "$namespace_json")" || die \
+ "The stack namespace has no verifiable identity."
+ recorded_hostname="$previously_reserved"
+ [ -z "$recorded_hostname" ] || [ "$recorded_hostname" = "$hostname" ] || die \
+ "Stack '$STACK_NAME' reserves '$recorded_hostname' for its $endpoint endpoint; hostname changes require explicit migration."
+
+ namespaces="$(kubectl --context "$KUBE_CONTEXT" get namespaces -o json)" || die \
+ "Unable to inspect existing stack hostname ownership."
+ conflict="$(jq -r --arg host "$hostname" --arg stack "$STACK_NAME" '
+ .items[] | select(.metadata.name != $stack) |
+ select([.metadata.annotations["openremote.io/reserved-hostname"],
+ .metadata.annotations["openremote.io/reserved-mqtts-hostname"]] |
+ any((. // "" | ascii_downcase | rtrimstr(".")) == $host)) |
+ .metadata.name' <<< "$namespaces")" || die "Unable to parse namespace hostname ownership."
+ [ -z "$conflict" ] || die "Hostname '$hostname' is already recorded by stack '$conflict'."
+ check_existing_hostname_routes "$hostname" "$STACK_NAME"
+
+ for attempt in {1..10}; do
+ registry="$(read_hostname_registry)" || die "Unable to read hostname reservations."
+ owner="$(jq -r --arg host "$hostname" --arg stack "$STACK_NAME" --arg uid "$namespace_uid" \
+ --arg endpoint "$endpoint" '
+ (.data // {} | to_entries[]) | (.value | fromjson) + {hostname: .key} |
+ . + {endpoint: (.endpoint // "web")} |
+ select((.hostname == $host or (.namespace == $stack and .endpoint == $endpoint)) and
+ (.hostname != $host or .namespace != $stack or .namespaceUID != $uid or .endpoint != $endpoint)) |
+ "\(.namespace) (namespace UID \(.namespaceUID), hostname \(.hostname))"' <<< "$registry")"
+ [ -z "$owner" ] || die \
+ "Hostname '$hostname' cannot be reserved by '$STACK_NAME'; conflicting reservation: $owner. Retained reservations require explicit cleanup."
+ # Removing exposure or ExternalDNS annotations does not prove external DNS
+ # has disappeared. Retain that cleanup obligation until explicit release.
+ replacement="$(jq -c --arg host "$hostname" --arg stack "$STACK_NAME" \
+ --arg uid "$namespace_uid" --arg target "$TARGET" --arg dns "$DNS_MODE" \
+ --arg exposure "$EXPOSURE" --arg endpoint "$endpoint" '
+ (.data[$host] // "{}" | fromjson) as $previous |
+ .data[$host] = ({namespace: $stack, namespaceUID: $uid, target: $target,
+ dns: (if $previous.dns == "external-dns" then $previous.dns else $dns end),
+ exposure: (if $exposure == "none" then $previous.exposure else $exposure end),
+ endpoint: $endpoint} | tojson)' <<< "$registry")"
+ if write_hostname_registry "$replacement"; then
+ kubectl --context "$KUBE_CONTEXT" annotate namespace "$STACK_NAME" \
+ "$reservation_annotation=$hostname" --overwrite >/dev/null
+ log "Reserved $endpoint hostname '$hostname' for stack '$STACK_NAME'."
+ return
+ fi
+ done
+ die "Hostname reservations changed repeatedly; retry apply. No workloads were installed."
+}
+
+reserve_stack_hostname() {
+ reserve_stack_endpoint_hostname "$HOSTNAME_VALUE" web "openremote.io/reserved-hostname"
+}
+
+reserve_stack_mqtts_hostname() {
+ reserve_stack_endpoint_hostname "$MQTTS_HOSTNAME_VALUE" mqtts "openremote.io/reserved-mqtts-hostname"
+}
+
+require_reservation_namespace_identity() {
+ local reservation="$1"
+ local namespace_json
+ [ -n "$reservation" ] || return 0
+ namespace_json="$(kubectl --context "$KUBE_CONTEXT" get namespace "$STACK_NAME" -o json)" || die \
+ "Unable to verify the reserved namespace identity before destroy."
+ [ "$(jq -r '.namespaceUID' <<< "$reservation")" = "$(jq -r '.metadata.uid' <<< "$namespace_json")" ] || die \
+ "The hostname reservation belongs to a different namespace UID; refusing to destroy a recreated namespace."
+}
+
+# The explicit release command confirms external cleanup on portable workflows.
+# EKS calls it only after its DNS/certificate cleanup has completed. Never release
+# on uninstall or with an EXIT trap: a failed apply may already expose a route.
+release_stack_hostname() {
+ local namespace_name registry reservation replacement attempt original_owner=""
+ check_existing_hostname_routes "$HOSTNAME_VALUE" ""
+ for attempt in {1..10}; do
+ registry="$(read_hostname_registry)" || die "Unable to read hostname reservations."
+ # Read identity before checking namespace absence, and repeat after a CAS
+ # conflict. Otherwise concurrent release/recreation could make a new claim
+ # look releasable based on an earlier observation of namespace absence.
+ namespace_name="$(kubectl --context "$KUBE_CONTEXT" get namespace "$STACK_NAME" \
+ --ignore-not-found -o name)" || die "Unable to verify namespace removal."
+ [ -z "$namespace_name" ] || die \
+ "Namespace '$STACK_NAME' still exists; destroy it before releasing its hostname."
+ reservation="$(jq -r --arg host "$HOSTNAME_VALUE" '.data[$host] // empty' <<< "$registry")"
+ [ -n "$reservation" ] || { log "Hostname '$HOSTNAME_VALUE' has no reservation."; return; }
+ [ "$(jq -r '.namespace' <<< "$reservation")" = "$STACK_NAME" ] || die \
+ "Hostname '$HOSTNAME_VALUE' belongs to another stack; refusing to release it."
+ if [ -z "$original_owner" ]; then
+ original_owner="$(jq -r '.namespaceUID' <<< "$reservation")"
+ fi
+ [ "$(jq -r '.namespaceUID' <<< "$reservation")" = "$original_owner" ] || die \
+ "Hostname ownership changed during release; refusing to remove the new reservation."
+ replacement="$(jq -c --arg host "$HOSTNAME_VALUE" 'del(.data[$host])' <<< "$registry")"
+ if write_hostname_registry "$replacement"; then
+ log "Released hostname '$HOSTNAME_VALUE'."
+ return
+ fi
+ done
+ die "Hostname reservations changed repeatedly; retry release-hostname."
+}
diff --git a/kubernetes/manager/Chart.yaml b/kubernetes/manager/Chart.yaml
index 1457f34cc..d79d54553 100644
--- a/kubernetes/manager/Chart.yaml
+++ b/kubernetes/manager/Chart.yaml
@@ -4,6 +4,6 @@ description: A Helm chart for deploying OpenRemote manager in Kubernetes
type: application
-version: 0.1.0
+version: 0.2.0
appVersion: "1.31.1"
diff --git a/kubernetes/manager/templates/_helpers.tpl b/kubernetes/manager/templates/_helpers.tpl
index 0b9927b4d..ae23f230b 100644
--- a/kubernetes/manager/templates/_helpers.tpl
+++ b/kubernetes/manager/templates/_helpers.tpl
@@ -61,6 +61,13 @@ Create the name of the service account to use
{{- end }}
{{- end }}
+{{/*
+Return the PVC used for manager data.
+*/}}
+{{- define "manager.persistence.claimName" -}}
+{{- default (include "manager.fullname" .) .Values.persistence.existingClaim -}}
+{{- end }}
+
{{/*
Create the name of the mqtt service to use
*/}}
diff --git a/kubernetes/manager/templates/deployment.yaml b/kubernetes/manager/templates/deployment.yaml
index bdec8e9f7..0acb81735 100644
--- a/kubernetes/manager/templates/deployment.yaml
+++ b/kubernetes/manager/templates/deployment.yaml
@@ -4,6 +4,7 @@
{{- $loggingExistingConfigMap := .Values.logging.existingConfigMap -}}
{{- $loggingEnabled := or $loggingInlineConfig $loggingExistingConfigMap -}}
{{- $loggingExternalConfig := $loggingExistingConfigMap -}}
+{{- $webserver := .Values.or.webserver | default dict -}}
apiVersion: apps/v1
kind: Deployment
metadata:
@@ -59,6 +60,10 @@ spec:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
+ ports:
+ - name: http
+ containerPort: {{ $webserver.listenPort | default 8080 }}
+ protocol: TCP
livenessProbe:
{{- toYaml .Values.livenessProbe | nindent 12 }}
readinessProbe:
@@ -80,6 +85,30 @@ spec:
- name: OR_SSL_PORT
value: "{{ .Values.or.port }}"
+ {{- range $name, $value := dict
+ "OR_WEBSERVER_LISTEN_HOST" $webserver.listenHost
+ "OR_WEBSERVER_LISTEN_PORT" $webserver.listenPort
+ "OR_WEBSERVER_ALLOWED_ORIGINS" $webserver.allowedOrigins
+ "OR_WEBSERVER_ALLOWED_METHODS" $webserver.allowedMethods
+ "OR_WEBSERVER_ALLOWED_HEADERS" $webserver.allowedHeaders
+ "OR_WEBSERVER_EXPOSED_HEADERS" $webserver.exposedHeaders
+ "OR_WEBSERVER_DUMP_REQUESTS" $webserver.dumpRequests
+ "OR_WEBSERVER_WORKER_THREADS_MAX" $webserver.workerThreadsMax
+ "OR_WEBSERVER_IO_THREADS_MAX" $webserver.ioThreadsMax
+ "OR_WEBSERVER_MAX_ENTITY_SIZE" $webserver.maxEntitySize
+ "OR_WEBSERVER_MULTIPART_MAX_ENTITY_SIZE" $webserver.multipartMaxEntitySize
+ }}
+ {{- if and (ne $value nil) (ne (toString $value) "") }}
+ - name: {{ $name }}
+ {{- /* YAML numbers may be floats; keep integer settings out of scientific notation. */}}
+ {{- if kindIs "float64" $value }}
+ value: {{ $value | toJson | quote }}
+ {{- else }}
+ value: {{ $value | quote }}
+ {{- end }}
+ {{- end }}
+ {{- end }}
+
- name: OR_FIREBASE_CONFIG_FILE
value: "{{ .Values.or.firebaseConfigFile }}"
@@ -99,12 +128,12 @@ spec:
value: "{{ .Values.or.postgresql.database }}"
- name: OR_DB_SCHEMA
value: "{{ .Values.or.postgresql.schema }}"
- - name: KC_DB_USERNAME
+ - name: OR_DB_USERNAME
valueFrom:
secretKeyRef:
name: openremote-secret
key: postgresql-username
- - name: KC_DB_PASSWORD
+ - name: OR_DB_PASSWORD
valueFrom:
secretKeyRef:
name: openremote-secret
@@ -197,7 +226,7 @@ spec:
{{- end }}
- name: manager-data
persistentVolumeClaim:
- claimName: {{ include "manager.fullname" . }}
+ claimName: {{ include "manager.persistence.claimName" . }}
{{- with .Values.volumes }}
{{- toYaml . | nindent 8 }}
{{- end }}
diff --git a/kubernetes/manager/templates/ingress.yaml b/kubernetes/manager/templates/ingress.yaml
index 10c46822f..15fd496d2 100644
--- a/kubernetes/manager/templates/ingress.yaml
+++ b/kubernetes/manager/templates/ingress.yaml
@@ -27,7 +27,7 @@ spec:
{{- end }}
rules:
{{- range .Values.ingress.hosts }}
- - host: {{ .host | quote }}
+ - host: {{ default .host $.Values.ingress.hostname | quote }}
http:
paths:
{{- range .paths }}
diff --git a/kubernetes/manager/templates/pvc.yaml b/kubernetes/manager/templates/pvc.yaml
index c93755c7f..66d347b46 100644
--- a/kubernetes/manager/templates/pvc.yaml
+++ b/kubernetes/manager/templates/pvc.yaml
@@ -1,12 +1,29 @@
+{{- if not .Values.persistence.existingClaim }}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: {{ include "manager.fullname" . }}
+ labels:
+ {{- include "manager.labels" . | nindent 4 }}
+ {{- if or .Values.persistence.retain .Values.persistence.annotations }}
+ annotations:
+ {{- if .Values.persistence.retain }}
+ helm.sh/resource-policy: keep
+ {{- end }}
+ {{- with .Values.persistence.annotations }}
+ {{- toYaml . | nindent 4 }}
+ {{- end }}
+ {{- end }}
spec:
accessModes:
- - ReadWriteOnce
+ {{- toYaml .Values.persistence.accessModes | nindent 4 }}
resources:
requests:
- storage: {{ .Values.volumeSize }}
- volumeName: manager-data-pv
- storageClassName: manual
+ storage: {{ .Values.persistence.size }}
+ {{- with .Values.persistence.storageClass }}
+ storageClassName: {{ . | quote }}
+ {{- end }}
+ {{- with .Values.persistence.volumeName }}
+ volumeName: {{ . | quote }}
+ {{- end }}
+{{- end }}
diff --git a/kubernetes/manager/templates/service.yaml b/kubernetes/manager/templates/service.yaml
index a828f80f1..59fa76dbc 100644
--- a/kubernetes/manager/templates/service.yaml
+++ b/kubernetes/manager/templates/service.yaml
@@ -8,7 +8,7 @@ spec:
type: {{ .Values.service.http.type }}
ports:
- port: {{ .Values.service.http.port }}
- targetPort: 8080
+ targetPort: http
protocol: TCP
name: http
{{- if and (not (.Values.service.mqtt.enabled)) (not (.Values.service.mqtts.enabled)) }}
@@ -20,7 +20,7 @@ spec:
selector:
{{- include "manager.selectorLabels" . | nindent 4 }}
---
-{{- if .Values.service.mqtt.enabled -}}
+{{- if .Values.service.mqtt.enabled }}
apiVersion: v1
kind: Service
metadata:
@@ -43,7 +43,7 @@ spec:
protocol: TCP
---
{{- end }}
-{{- if .Values.service.mqtts.enabled -}}
+{{- if .Values.service.mqtts.enabled }}
apiVersion: v1
kind: Service
metadata:
@@ -86,7 +86,7 @@ spec:
{{- include "manager.selectorLabels" . | nindent 4 }}
{{- end }}
---
-{{- if .Values.service.jmx.enabled -}}
+{{- if .Values.service.jmx.enabled }}
apiVersion: v1
kind: Service
metadata:
diff --git a/kubernetes/manager/values-eks.yaml b/kubernetes/manager/values-eks.yaml
index aa2722767..047c99624 100644
--- a/kubernetes/manager/values-eks.yaml
+++ b/kubernetes/manager/values-eks.yaml
@@ -49,15 +49,9 @@ service:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
-affinity:
- nodeAffinity:
- requiredDuringSchedulingIgnoredDuringExecution:
- nodeSelectorTerms:
- - matchExpressions:
- - key: topology.kubernetes.io/zone
- operator: In
- values:
- - eu-west-1a
+
+persistence:
+ storageClass: openremote-ebs
or:
hostname: k8stest.openremote.app
diff --git a/kubernetes/manager/values-haproxy-eks.yaml b/kubernetes/manager/values-haproxy-eks.yaml
index d7b3c19e9..77e9b8312 100644
--- a/kubernetes/manager/values-haproxy-eks.yaml
+++ b/kubernetes/manager/values-haproxy-eks.yaml
@@ -10,15 +10,8 @@ service:
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
-affinity:
- nodeAffinity:
- requiredDuringSchedulingIgnoredDuringExecution:
- nodeSelectorTerms:
- - matchExpressions:
- - key: topology.kubernetes.io/zone
- operator: In
- values:
- - eu-west-1a
+persistence:
+ storageClass: openremote-ebs
or:
hostname: k8stest.openremote.app
diff --git a/kubernetes/manager/values.yaml b/kubernetes/manager/values.yaml
index 7c925911d..ba76fea46 100644
--- a/kubernetes/manager/values.yaml
+++ b/kubernetes/manager/values.yaml
@@ -82,6 +82,8 @@ service:
ingress:
enabled: false
className: "nginx"
+ # When set, overrides the host of every rule without replacing its paths.
+ hostname: ""
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
@@ -111,22 +113,34 @@ resources: {}
livenessProbe:
httpGet:
path: /
- port: 8080
+ port: http
periodSeconds: 5
readinessProbe:
httpGet:
path: /
- port: 8080
+ port: http
periodSeconds: 5
startupProbe:
httpGet:
path: /
- port: 8080
+ port: http
failureThreshold: 30
periodSeconds: 5
-# Size of volume requested in PVC, must be aligned with available PV
-volumeSize: "1Gi"
+# Persistent manager data. When storageClass is empty, the cluster's default
+# StorageClass dynamically provisions the volume. Set existingClaim to use a
+# PVC managed outside this chart instead.
+persistence:
+ size: "1Gi"
+ storageClass: ""
+ existingClaim: ""
+ # Optionally bind the chart-managed claim to a pre-provisioned PV.
+ volumeName: ""
+ accessModes:
+ - ReadWriteOnce
+ # Preserve the chart-managed PVC when Helm uninstalls the release.
+ retain: true
+ annotations: {}
# Additional volumes on the output Deployment definition.
volumes: []
@@ -187,6 +201,31 @@ or:
hostname: "localhost"
port: "-1"
+ # Manager web server settings, mapped to OR_WEBSERVER_* environment variables.
+ # Null or empty values are omitted to retain the application's defaults.
+ webserver:
+ # Interface binding; application default: 0.0.0.0.
+ listenHost: "0.0.0.0"
+ # Container HTTP port. The Service target and default probes follow this port.
+ # service.http.port controls the port exposed by the Service independently.
+ listenPort: 8080
+ # Comma-separated CORS origins in addition to OR_HOSTNAME and OR_ADDITIONAL_HOSTNAMES.
+ allowedOrigins: ""
+ # Comma-separated CORS methods and request headers; application defaults: *.
+ allowedMethods: ""
+ allowedHeaders: ""
+ # Comma-separated response headers exposed through CORS; none by default.
+ exposedHeaders: ""
+ # Log web server requests; application default: false.
+ dumpRequests: false
+ # Application default: the greater of the processor core count and 10.
+ workerThreadsMax: null
+ # Application default: the greater of the processor core count and 2.
+ ioThreadsMax: null
+ # Maximum request entity sizes in bytes; application defaults: 30000000.
+ maxEntitySize: null
+ multipartMaxEntitySize: null
+
# It is important that all services have the same timezone and are time synchronized.
# Bearer tokens are only valid for minutes, and authentication fails if Keycloak drifts
timeZone: "Europe/Amsterdam"
diff --git a/kubernetes/openremote-manager-pv.yaml b/kubernetes/openremote-manager-pv.yaml
deleted file mode 100644
index 46bf783f7..000000000
--- a/kubernetes/openremote-manager-pv.yaml
+++ /dev/null
@@ -1,13 +0,0 @@
-apiVersion: v1
-kind: PersistentVolume
-metadata:
- name: manager-data-pv
-spec:
- capacity:
- storage: 1Gi
- accessModes:
- - ReadWriteOnce
- hostPath:
- path: /Users/xxx/Development/OpenRemote/pv/manager-data
- persistentVolumeReclaimPolicy: Retain
- storageClassName: manual
diff --git a/kubernetes/openremote-psql-pv.yaml b/kubernetes/openremote-psql-pv.yaml
deleted file mode 100644
index 4c4e19691..000000000
--- a/kubernetes/openremote-psql-pv.yaml
+++ /dev/null
@@ -1,13 +0,0 @@
-apiVersion: v1
-kind: PersistentVolume
-metadata:
- name: postgresql-data-pv
-spec:
- capacity:
- storage: 5Gi
- accessModes:
- - ReadWriteOnce
- hostPath:
- path: /Users/xxx/Development/OpenRemote/pv/postgresql-data
- persistentVolumeReclaimPolicy: Retain
- storageClassName: manual
diff --git a/kubernetes/or-eks-cluster b/kubernetes/or-eks-cluster
new file mode 100755
index 000000000..7d1c759a7
--- /dev/null
+++ b/kubernetes/or-eks-cluster
@@ -0,0 +1,1168 @@
+#!/usr/bin/env bash
+
+# Cluster lifecycle only; OpenRemote stack lifecycle is managed separately.
+
+set -Eeuo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+KUBERNETES_DIR="$SCRIPT_DIR"
+source "$SCRIPT_DIR/lib/hostname-reservations"
+
+DEFAULT_CONFIG_FILE="$KUBERNETES_DIR/cluster.yaml"
+DEFAULT_STORAGE_CLASS_FILE="$KUBERNETES_DIR/cluster/eks/storage-class.yaml"
+DEFAULT_INGRESS_CLASS_FILE="$KUBERNETES_DIR/cluster/eks/ingress-class.yaml"
+DEFAULT_AWS_REGION="eu-west-1"
+DEFAULT_LOAD_BALANCER_CONTROLLER_CHART_VERSION="1.14.0"
+DEFAULT_EXTERNAL_DNS_CHART_VERSION="1.21.1"
+DEFAULT_EXTERNAL_DNS_IMAGE_TAG="v0.22.0"
+EXTERNAL_DNS_RELEASE_NAME="openremote-external-dns"
+EXTERNAL_DNS_SERVICE_ACCOUNT_NAME="external-dns"
+EXTERNAL_DNS_ENDPOINT_CRD="dnsendpoints.externaldns.k8s.io"
+SHARED_CERTIFICATE_ARN_ANNOTATION="openremote.io/shared-certificate-arn"
+
+COMMAND=""
+CLUSTER_NAME="${OR_EKS_CLUSTER_NAME:-}"
+AWS_REGION_NAME="${OR_EKS_AWS_REGION:-${AWS_REGION:-$DEFAULT_AWS_REGION}}"
+# Only --profile selects an explicit profile; otherwise leave credential
+# resolution to AWS and eksctl, including their standard environment settings.
+AWS_PROFILE_NAME=""
+CONFIG_FILE="${OR_EKS_CONFIG_FILE:-$DEFAULT_CONFIG_FILE}"
+KUBE_CONTEXT="${OR_EKS_KUBE_CONTEXT:-}"
+LOAD_BALANCER_CONTROLLER_CHART_VERSION="${OR_EKS_LOAD_BALANCER_CONTROLLER_CHART_VERSION:-$DEFAULT_LOAD_BALANCER_CONTROLLER_CHART_VERSION}"
+EXTERNAL_DNS_CHART_VERSION="${OR_EKS_EXTERNAL_DNS_CHART_VERSION:-$DEFAULT_EXTERNAL_DNS_CHART_VERSION}"
+EXTERNAL_DNS_IMAGE_TAG="${OR_EKS_EXTERNAL_DNS_IMAGE_TAG:-$DEFAULT_EXTERNAL_DNS_IMAGE_TAG}"
+EXTERNAL_DNS_HOSTED_ZONE_ID="${OR_EKS_EXTERNAL_DNS_HOSTED_ZONE_ID:-}"
+EXTERNAL_DNS_DOMAIN="${OR_EKS_EXTERNAL_DNS_DOMAIN:-}"
+EXTERNAL_DNS_ASSUME_ROLE_ARN="${OR_EKS_EXTERNAL_DNS_ASSUME_ROLE_ARN:-}"
+EXTERNAL_DNS_ASSUME_ROLE_EXTERNAL_ID="${OR_EKS_EXTERNAL_DNS_ASSUME_ROLE_EXTERNAL_ID:-}"
+EXTERNAL_DNS_OWNER_ID="${OR_EKS_EXTERNAL_DNS_OWNER_ID:-}"
+EXTERNAL_DNS_IRSA_ROLE_NAME="${OR_EKS_EXTERNAL_DNS_IRSA_ROLE_NAME:-}"
+EXTERNAL_DNS_DRY_RUN="${OR_EKS_EXTERNAL_DNS_DRY_RUN:-false}"
+SHARED_CERTIFICATE_ARN="${OR_EKS_SHARED_CERTIFICATE_ARN:-}"
+CONFIRM_CLUSTER_NAME=""
+MANAGE_LOAD_BALANCER_CONTROLLER=true
+MANAGE_EXTERNAL_DNS=true
+CLEAR_SHARED_CERTIFICATE=false
+RENDERED_CONFIG_FILE=""
+EXTERNAL_DNS_IAM_CONFIG_FILE=""
+
+usage() {
+ cat <<'EOF'
+Manage the shared Amazon EKS cluster used to host OpenRemote stacks.
+
+Usage:
+ or-eks-cluster create [options]
+ or-eks-cluster apply [options]
+ or-eks-cluster status [options]
+ or-eks-cluster kubeconfig [options]
+ or-eks-cluster destroy --confirm <cluster-name> [options]
+
+Commands:
+ create Create the EKS cluster and reconcile shared resources.
+ apply Reconcile shared cluster resources. Does not change the
+ Kubernetes minor version or node-group infrastructure.
+ status Show EKS, node, add-on, and controller status.
+ kubeconfig Create or refresh the stable kubeconfig context.
+ destroy Delete an empty cluster after an exact-name confirmation.
+
+Options:
+ --name <name> EKS cluster name. Env: OR_EKS_CLUSTER_NAME.
+ --region <region> AWS region. Env: OR_EKS_AWS_REGION. Default: eu-west-1.
+ --profile <name> Explicit AWS profile. Default: no profile argument;
+ use the normal AWS credential chain. An empty value
+ also leaves profile selection to AWS.
+ --config <path> eksctl ClusterConfig template. Env: OR_EKS_CONFIG_FILE.
+ --kube-context <c> Stable kubeconfig context. Defaults to <name>@<region>.
+ --load-balancer-controller-version <version>
+ Helm chart version to reconcile. Env:
+ OR_EKS_LOAD_BALANCER_CONTROLLER_CHART_VERSION.
+ --skip-load-balancer-controller
+ Do not install or reconcile the controller.
+ --shared-certificate-arn <arn>
+ Record an externally managed ACM certificate as the
+ cluster's shared Ingress certificate. Env:
+ OR_EKS_SHARED_CERTIFICATE_ARN.
+ --clear-shared-certificate
+ Remove the configured shared certificate reference.
+ The ACM certificate itself is never modified or deleted.
+ --external-dns-zone-id <id>
+ Public Route 53 hosted-zone ID. Env:
+ OR_EKS_EXTERNAL_DNS_HOSTED_ZONE_ID.
+ --external-dns-domain <domain>
+ DNS suffix managed in that zone. Env:
+ OR_EKS_EXTERNAL_DNS_DOMAIN.
+ --external-dns-role-arn <arn>
+ DNS-account role for ExternalDNS to assume. Env:
+ OR_EKS_EXTERNAL_DNS_ASSUME_ROLE_ARN.
+ --external-dns-external-id <id>
+ Optional external ID required by the DNS-account role.
+ Env: OR_EKS_EXTERNAL_DNS_ASSUME_ROLE_EXTERNAL_ID.
+ --external-dns-owner-id <id>
+ Stable, cluster-unique TXT registry owner. Env:
+ OR_EKS_EXTERNAL_DNS_OWNER_ID.
+ --external-dns-irsa-role-name <name>
+ Cluster-account IRSA role name. Defaults to
+ openremote-<cluster>-<region>-external-dns. Env:
+ OR_EKS_EXTERNAL_DNS_IRSA_ROLE_NAME.
+ --external-dns-chart-version <version>
+ Helm chart version. Env:
+ OR_EKS_EXTERNAL_DNS_CHART_VERSION. Default: 1.21.1.
+ --external-dns-image-tag <tag>
+ ExternalDNS image tag. Env:
+ OR_EKS_EXTERNAL_DNS_IMAGE_TAG. Default: v0.22.0.
+ --external-dns-dry-run
+ Report desired DNS changes without applying them.
+ Env: OR_EKS_EXTERNAL_DNS_DRY_RUN.
+ --skip-external-dns
+ Do not install, reconcile, or uninstall ExternalDNS;
+ use this for a bring-your-own controller.
+ --confirm <name> Required exact cluster name for destroy.
+ -h, --help Show this help.
+
+The command manages only cluster-scoped infrastructure and add-ons. It never
+installs or removes an OpenRemote stack, stack namespace, certificate, or stack
+data volume. A shared certificate ARN is configuration only; the certificate
+remains externally managed. Its optional ExternalDNS add-on reconciles only
+explicitly labeled stack endpoints.
+EOF
+}
+
+log() {
+ printf '[or-eks-cluster] %s\n' "$*"
+}
+
+warn() {
+ printf '[or-eks-cluster] WARNING: %s\n' "$*" >&2
+}
+
+die() {
+ printf '[or-eks-cluster] ERROR: %s\n' "$*" >&2
+ exit 1
+}
+
+cleanup() {
+ if [ -n "$RENDERED_CONFIG_FILE" ] && [ -f "$RENDERED_CONFIG_FILE" ]; then
+ rm -f "$RENDERED_CONFIG_FILE"
+ fi
+ if [ -n "$EXTERNAL_DNS_IAM_CONFIG_FILE" ] && [ -f "$EXTERNAL_DNS_IAM_CONFIG_FILE" ]; then
+ rm -f "$EXTERNAL_DNS_IAM_CONFIG_FILE"
+ fi
+}
+
+trap cleanup EXIT
+
+require_command() {
+ command -v "$1" >/dev/null 2>&1 || die "Required command '$1' was not found in PATH."
+}
+
+aws_eks() {
+ if [ -n "$AWS_PROFILE_NAME" ]; then
+ aws eks "$@" --region "$AWS_REGION_NAME" --profile "$AWS_PROFILE_NAME"
+ else
+ aws eks "$@" --region "$AWS_REGION_NAME"
+ fi
+}
+
+eksctl_with_profile() {
+ if [ -n "$AWS_PROFILE_NAME" ]; then
+ eksctl "$@" --profile "$AWS_PROFILE_NAME"
+ else
+ eksctl "$@"
+ fi
+}
+
+parse_arguments() {
+ if [ "$#" -eq 0 ]; then
+ usage >&2
+ exit 1
+ fi
+
+ case "$1" in
+ -h|--help)
+ usage
+ exit 0
+ ;;
+ create|apply|status|kubeconfig|destroy)
+ COMMAND="$1"
+ shift
+ ;;
+ *)
+ die "Unknown command '$1'. Run with --help for usage."
+ ;;
+ esac
+
+ while [ "$#" -gt 0 ]; do
+ case "$1" in
+ --name)
+ [ "$#" -ge 2 ] || die "Option --name requires a value."
+ CLUSTER_NAME="$2"
+ shift 2
+ ;;
+ --region)
+ [ "$#" -ge 2 ] || die "Option --region requires a value."
+ AWS_REGION_NAME="$2"
+ shift 2
+ ;;
+ --profile)
+ [ "$#" -ge 2 ] || die "Option --profile requires a value."
+ AWS_PROFILE_NAME="$2"
+ shift 2
+ ;;
+ --config)
+ [ "$#" -ge 2 ] || die "Option --config requires a value."
+ CONFIG_FILE="$2"
+ shift 2
+ ;;
+ --kube-context)
+ [ "$#" -ge 2 ] || die "Option --kube-context requires a value."
+ KUBE_CONTEXT="$2"
+ shift 2
+ ;;
+ --load-balancer-controller-version)
+ [ "$#" -ge 2 ] || die "Option --load-balancer-controller-version requires a value."
+ LOAD_BALANCER_CONTROLLER_CHART_VERSION="$2"
+ shift 2
+ ;;
+ --skip-load-balancer-controller)
+ MANAGE_LOAD_BALANCER_CONTROLLER=false
+ shift
+ ;;
+ --shared-certificate-arn)
+ [ "$#" -ge 2 ] || die "Option --shared-certificate-arn requires a value."
+ SHARED_CERTIFICATE_ARN="$2"
+ shift 2
+ ;;
+ --clear-shared-certificate)
+ CLEAR_SHARED_CERTIFICATE=true
+ shift
+ ;;
+ --external-dns-zone-id)
+ [ "$#" -ge 2 ] || die "Option --external-dns-zone-id requires a value."
+ EXTERNAL_DNS_HOSTED_ZONE_ID="$2"
+ shift 2
+ ;;
+ --external-dns-domain)
+ [ "$#" -ge 2 ] || die "Option --external-dns-domain requires a value."
+ EXTERNAL_DNS_DOMAIN="$2"
+ shift 2
+ ;;
+ --external-dns-role-arn)
+ [ "$#" -ge 2 ] || die "Option --external-dns-role-arn requires a value."
+ EXTERNAL_DNS_ASSUME_ROLE_ARN="$2"
+ shift 2
+ ;;
+ --external-dns-external-id)
+ [ "$#" -ge 2 ] || die "Option --external-dns-external-id requires a value."
+ EXTERNAL_DNS_ASSUME_ROLE_EXTERNAL_ID="$2"
+ shift 2
+ ;;
+ --external-dns-owner-id)
+ [ "$#" -ge 2 ] || die "Option --external-dns-owner-id requires a value."
+ EXTERNAL_DNS_OWNER_ID="$2"
+ shift 2
+ ;;
+ --external-dns-irsa-role-name)
+ [ "$#" -ge 2 ] || die "Option --external-dns-irsa-role-name requires a value."
+ EXTERNAL_DNS_IRSA_ROLE_NAME="$2"
+ shift 2
+ ;;
+ --external-dns-chart-version)
+ [ "$#" -ge 2 ] || die "Option --external-dns-chart-version requires a value."
+ EXTERNAL_DNS_CHART_VERSION="$2"
+ shift 2
+ ;;
+ --external-dns-image-tag)
+ [ "$#" -ge 2 ] || die "Option --external-dns-image-tag requires a value."
+ EXTERNAL_DNS_IMAGE_TAG="$2"
+ shift 2
+ ;;
+ --external-dns-dry-run)
+ EXTERNAL_DNS_DRY_RUN=true
+ shift
+ ;;
+ --skip-external-dns)
+ MANAGE_EXTERNAL_DNS=false
+ shift
+ ;;
+ --confirm)
+ [ "$#" -ge 2 ] || die "Option --confirm requires a value."
+ CONFIRM_CLUSTER_NAME="$2"
+ shift 2
+ ;;
+ -h|--help)
+ usage
+ exit 0
+ ;;
+ *)
+ die "Unknown option '$1'. Run with --help for usage."
+ ;;
+ esac
+ done
+
+ [ -n "$CLUSTER_NAME" ] || die "Cluster name is required; use --name or OR_EKS_CLUSTER_NAME."
+
+ if [ -z "$KUBE_CONTEXT" ]; then
+ KUBE_CONTEXT="${CLUSTER_NAME}@${AWS_REGION_NAME}"
+ fi
+
+ EXTERNAL_DNS_HOSTED_ZONE_ID="${EXTERNAL_DNS_HOSTED_ZONE_ID#/hostedzone/}"
+ EXTERNAL_DNS_DOMAIN="${EXTERNAL_DNS_DOMAIN%.}"
+ if [ -z "$EXTERNAL_DNS_IRSA_ROLE_NAME" ]; then
+ EXTERNAL_DNS_IRSA_ROLE_NAME="openremote-${CLUSTER_NAME}-${AWS_REGION_NAME}-external-dns"
+ fi
+}
+
+validate_shared_certificate_configuration() {
+ if [ -n "$SHARED_CERTIFICATE_ARN" ] && [ "$CLEAR_SHARED_CERTIFICATE" = true ]; then
+ die "Options --shared-certificate-arn and --clear-shared-certificate cannot be used together."
+ fi
+ if { [ -n "$SHARED_CERTIFICATE_ARN" ] || [ "$CLEAR_SHARED_CERTIFICATE" = true ]; } && \
+ [ "$MANAGE_LOAD_BALANCER_CONTROLLER" != true ]; then
+ die "Shared certificate configuration requires management of the openremote-alb IngressClass."
+ fi
+ if [ -z "$SHARED_CERTIFICATE_ARN" ]; then
+ return
+ fi
+ if [[ ! "$SHARED_CERTIFICATE_ARN" =~ ^arn:[^:]+:acm:([a-z0-9-]+):[0-9]{12}:certificate/[A-Za-z0-9-]+$ ]]; then
+ die "Shared certificate ARN must be a valid ACM certificate ARN."
+ fi
+ [ "${BASH_REMATCH[1]}" = "$AWS_REGION_NAME" ] || die \
+ "Shared certificate '$SHARED_CERTIFICATE_ARN' is not in cluster region '$AWS_REGION_NAME'."
+}
+
+external_dns_configuration_present() {
+ [ -n "$EXTERNAL_DNS_HOSTED_ZONE_ID" ] || \
+ [ -n "$EXTERNAL_DNS_DOMAIN" ] || \
+ [ -n "$EXTERNAL_DNS_ASSUME_ROLE_ARN" ] || \
+ [ -n "$EXTERNAL_DNS_ASSUME_ROLE_EXTERNAL_ID" ] || \
+ [ -n "$EXTERNAL_DNS_OWNER_ID" ] || \
+ [ "$EXTERNAL_DNS_DRY_RUN" = true ]
+}
+
+validate_external_dns_configuration() {
+ case "$EXTERNAL_DNS_DRY_RUN" in
+ true|false)
+ ;;
+ *)
+ die "OR_EKS_EXTERNAL_DNS_DRY_RUN must be true or false."
+ ;;
+ esac
+
+ if [ "$MANAGE_EXTERNAL_DNS" != true ]; then
+ return
+ fi
+ if ! external_dns_configuration_present; then
+ return
+ fi
+
+ [ -n "$EXTERNAL_DNS_HOSTED_ZONE_ID" ] || die \
+ "ExternalDNS requires --external-dns-zone-id."
+ [ -n "$EXTERNAL_DNS_DOMAIN" ] || die \
+ "ExternalDNS requires --external-dns-domain."
+ [ -n "$EXTERNAL_DNS_ASSUME_ROLE_ARN" ] || die \
+ "ExternalDNS requires --external-dns-role-arn."
+ [ -n "$EXTERNAL_DNS_OWNER_ID" ] || die \
+ "ExternalDNS requires --external-dns-owner-id."
+ [ -n "$EXTERNAL_DNS_CHART_VERSION" ] || die \
+ "A pinned ExternalDNS chart version is required."
+ [ -n "$EXTERNAL_DNS_IMAGE_TAG" ] || die \
+ "A pinned ExternalDNS image tag is required."
+
+ [[ "$EXTERNAL_DNS_HOSTED_ZONE_ID" =~ ^Z[A-Z0-9]+$ ]] || die \
+ "ExternalDNS hosted-zone ID must be a Route 53 ID such as Z0123456789."
+ [[ "$EXTERNAL_DNS_DOMAIN" =~ ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ ]] || die \
+ "ExternalDNS domain must be a lowercase DNS suffix without a trailing dot."
+ [[ "$EXTERNAL_DNS_ASSUME_ROLE_ARN" =~ ^arn:[^:]+:iam::[0-9]{12}:role/[A-Za-z0-9_+=,.@/-]+$ ]] || die \
+ "ExternalDNS role ARN must be a valid IAM role ARN."
+ [[ "$EXTERNAL_DNS_OWNER_ID" =~ ^[A-Za-z0-9._-]+$ ]] || die \
+ "ExternalDNS owner ID may contain only letters, numbers, dots, underscores, and hyphens."
+ [ "${#EXTERNAL_DNS_IRSA_ROLE_NAME}" -le 64 ] && \
+ [[ "$EXTERNAL_DNS_IRSA_ROLE_NAME" =~ ^[A-Za-z0-9_+=,.@-]+$ ]] || die \
+ "ExternalDNS IRSA role name must be a valid IAM role name of at most 64 characters."
+}
+
+cluster_exists() {
+ local output
+ local status
+
+ set +e
+ output="$(aws_eks describe-cluster --name "$CLUSTER_NAME" --output json 2>&1)"
+ status=$?
+ set -e
+
+ if [ "$status" -eq 0 ]; then
+ return 0
+ fi
+
+ case "$output" in
+ *ResourceNotFoundException*) return 1 ;;
+ *) die "Unable to query EKS cluster '$CLUSTER_NAME': $output" ;;
+ esac
+}
+
+require_existing_cluster() {
+ cluster_exists || die "EKS cluster '$CLUSTER_NAME' does not exist in region '$AWS_REGION_NAME'."
+}
+
+render_cluster_config() {
+ [ -f "$CONFIG_FILE" ] || die "Cluster configuration '$CONFIG_FILE' does not exist."
+ require_command envsubst
+
+ RENDERED_CONFIG_FILE="$(mktemp "${TMPDIR:-/tmp}/or-eks-cluster.XXXXXX")"
+ export CLUSTER_NAME
+ export AWS_REGION="$AWS_REGION_NAME"
+ envsubst '${CLUSTER_NAME} ${AWS_REGION}' < "$CONFIG_FILE" > "$RENDERED_CONFIG_FILE"
+
+ if grep -q '\${CLUSTER_NAME}\|\${AWS_REGION}' "$RENDERED_CONFIG_FILE"; then
+ die "Cluster configuration still contains an unexpanded cluster name or AWS region."
+ fi
+}
+
+configured_kubernetes_version() {
+ awk '/^[[:space:]]+version:[[:space:]]*/ { gsub(/"/, "", $2); print $2; exit }' "$RENDERED_CONFIG_FILE"
+}
+
+current_kubernetes_version() {
+ aws_eks describe-cluster \
+ --name "$CLUSTER_NAME" \
+ --query 'cluster.version' \
+ --output text
+}
+
+configure_kube_context() {
+ log "Writing kubeconfig context '$KUBE_CONTEXT'."
+ if [ -n "$AWS_PROFILE_NAME" ]; then
+ aws eks update-kubeconfig \
+ --name "$CLUSTER_NAME" \
+ --region "$AWS_REGION_NAME" \
+ --profile "$AWS_PROFILE_NAME" \
+ --alias "$KUBE_CONTEXT" >/dev/null
+ else
+ aws eks update-kubeconfig \
+ --name "$CLUSTER_NAME" \
+ --region "$AWS_REGION_NAME" \
+ --alias "$KUBE_CONTEXT" >/dev/null
+ fi
+}
+
+kube_context_exists() {
+ kubectl config get-contexts "$KUBE_CONTEXT" -o name 2>/dev/null | grep -Fx "$KUBE_CONTEXT" >/dev/null 2>&1
+}
+
+require_kube_context() {
+ local expected_endpoint
+ local context_endpoint
+
+ kube_context_exists || die "Kubeconfig context '$KUBE_CONTEXT' does not exist; run the kubeconfig command first."
+
+ expected_endpoint="$(aws_eks describe-cluster \
+ --name "$CLUSTER_NAME" \
+ --query 'cluster.endpoint' \
+ --output text)"
+ context_endpoint="$(kubectl --context "$KUBE_CONTEXT" config view \
+ --minify \
+ -o jsonpath='{.clusters[0].cluster.server}')"
+
+ [ "$context_endpoint" = "$expected_endpoint" ] || die \
+ "Kubeconfig context '$KUBE_CONTEXT' does not point to EKS cluster '$CLUSTER_NAME'."
+}
+
+reconcile_load_balancer_controller() {
+ local deployed_shared_certificate_arn
+
+ if [ "$MANAGE_LOAD_BALANCER_CONTROLLER" != true ]; then
+ log "Skipping AWS Load Balancer Controller reconciliation."
+ return
+ fi
+
+ [ -n "$LOAD_BALANCER_CONTROLLER_CHART_VERSION" ] || die \
+ "A pinned AWS Load Balancer Controller chart version is required."
+ [ -f "$DEFAULT_INGRESS_CLASS_FILE" ] || die \
+ "IngressClass manifest '$DEFAULT_INGRESS_CLASS_FILE' does not exist."
+
+ require_command helm
+ require_command kubectl
+ require_kube_context
+
+ kubectl --context "$KUBE_CONTEXT" \
+ --namespace kube-system \
+ get serviceaccount aws-load-balancer-controller >/dev/null 2>&1 || die \
+ "ServiceAccount kube-system/aws-load-balancer-controller is missing from the cluster configuration."
+
+ log "Refreshing the AWS EKS Helm repository."
+ helm repo add eks https://aws.github.io/eks-charts --force-update >/dev/null
+ helm repo update eks >/dev/null
+
+ log "Reconciling AWS Load Balancer Controller CRDs."
+ helm show crds eks/aws-load-balancer-controller \
+ --version "$LOAD_BALANCER_CONTROLLER_CHART_VERSION" | \
+ kubectl --context "$KUBE_CONTEXT" apply -f - >/dev/null
+ kubectl --context "$KUBE_CONTEXT" wait \
+ --for=condition=Established \
+ customresourcedefinition/ingressclassparams.elbv2.k8s.aws \
+ --timeout 2m >/dev/null
+
+ log "Reconciling AWS Load Balancer Controller chart $LOAD_BALANCER_CONTROLLER_CHART_VERSION."
+ helm upgrade --install aws-load-balancer-controller eks/aws-load-balancer-controller \
+ --kube-context "$KUBE_CONTEXT" \
+ --namespace kube-system \
+ --version "$LOAD_BALANCER_CONTROLLER_CHART_VERSION" \
+ --set "clusterName=$CLUSTER_NAME" \
+ --set serviceAccount.create=false \
+ --set serviceAccount.name=aws-load-balancer-controller \
+ --wait \
+ --timeout 10m
+
+ log "Reconciling the shared OpenRemote ALB IngressClass."
+ kubectl --context "$KUBE_CONTEXT" apply -f "$DEFAULT_INGRESS_CLASS_FILE"
+
+ if [ -n "$SHARED_CERTIFICATE_ARN" ]; then
+ log "Recording the externally managed shared ACM certificate."
+ kubectl --context "$KUBE_CONTEXT" annotate ingressclass openremote-alb \
+ "$SHARED_CERTIFICATE_ARN_ANNOTATION=$SHARED_CERTIFICATE_ARN" \
+ --overwrite >/dev/null
+ elif [ "$CLEAR_SHARED_CERTIFICATE" = true ]; then
+ deployed_shared_certificate_arn="$(kubectl --context "$KUBE_CONTEXT" \
+ get ingressclass openremote-alb \
+ -o "go-template={{with index .metadata.annotations \"$SHARED_CERTIFICATE_ARN_ANNOTATION\"}}{{.}}{{end}}")" || die \
+ "Unable to inspect the shared ACM certificate configuration."
+ if [ -n "$deployed_shared_certificate_arn" ]; then
+ log "Removing the shared ACM certificate reference."
+ kubectl --context "$KUBE_CONTEXT" annotate ingressclass openremote-alb \
+ "$SHARED_CERTIFICATE_ARN_ANNOTATION-" >/dev/null
+ else
+ log "No shared ACM certificate reference is configured."
+ fi
+ fi
+}
+
+render_external_dns_iam_config() {
+ if [ -n "$EXTERNAL_DNS_IAM_CONFIG_FILE" ] && [ -f "$EXTERNAL_DNS_IAM_CONFIG_FILE" ]; then
+ rm -f "$EXTERNAL_DNS_IAM_CONFIG_FILE"
+ fi
+ EXTERNAL_DNS_IAM_CONFIG_FILE="$(mktemp "${TMPDIR:-/tmp}/or-eks-external-dns-iam.XXXXXX")"
+ cat > "$EXTERNAL_DNS_IAM_CONFIG_FILE" <<EOF
+apiVersion: eksctl.io/v1alpha5
+kind: ClusterConfig
+
+metadata:
+ name: "$CLUSTER_NAME"
+ region: "$AWS_REGION_NAME"
+
+iam:
+ withOIDC: true
+ serviceAccounts:
+ - metadata:
+ name: $EXTERNAL_DNS_SERVICE_ACCOUNT_NAME
+ namespace: kube-system
+ roleName: "$EXTERNAL_DNS_IRSA_ROLE_NAME"
+ attachPolicy:
+ Version: "2012-10-17"
+ Statement:
+ - Effect: Allow
+ Action:
+ - sts:AssumeRole
+ Resource: "$EXTERNAL_DNS_ASSUME_ROLE_ARN"
+EOF
+}
+
+reconcile_external_dns_iam_service_account() {
+ local iam_service_accounts
+ local get_status
+ local service_account_role_arn
+
+ require_command eksctl
+ require_command jq
+ require_command kubectl
+ require_kube_context
+ render_external_dns_iam_config
+
+ set +e
+ iam_service_accounts="$(eksctl_with_profile get iamserviceaccount \
+ --cluster "$CLUSTER_NAME" \
+ --region "$AWS_REGION_NAME" \
+ --namespace kube-system \
+ --name "$EXTERNAL_DNS_SERVICE_ACCOUNT_NAME" \
+ --output json 2>&1)"
+ get_status=$?
+ set -e
+
+ if [ "$get_status" -ne 0 ]; then
+ case "$iam_service_accounts" in
+ *"no iamserviceaccounts"*) iam_service_accounts='[]' ;;
+ *) die "Unable to inspect the ExternalDNS IAM service account: $iam_service_accounts" ;;
+ esac
+ fi
+
+ if jq -e 'type == "array" and length > 0' <<< "$iam_service_accounts" >/dev/null; then
+ log "Reconciling the ExternalDNS IRSA role '$EXTERNAL_DNS_IRSA_ROLE_NAME'."
+ eksctl_with_profile update iamserviceaccount \
+ --config-file "$EXTERNAL_DNS_IAM_CONFIG_FILE" \
+ --include "kube-system/$EXTERNAL_DNS_SERVICE_ACCOUNT_NAME" \
+ --approve
+ else
+ log "Creating the ExternalDNS IRSA role '$EXTERNAL_DNS_IRSA_ROLE_NAME'."
+ eksctl_with_profile create iamserviceaccount \
+ --config-file "$EXTERNAL_DNS_IAM_CONFIG_FILE" \
+ --include "kube-system/$EXTERNAL_DNS_SERVICE_ACCOUNT_NAME" \
+ --override-existing-serviceaccounts \
+ --approve
+ fi
+
+ if ! service_account_role_arn="$(kubectl --context "$KUBE_CONTEXT" \
+ --namespace kube-system \
+ get serviceaccount "$EXTERNAL_DNS_SERVICE_ACCOUNT_NAME" \
+ -o 'jsonpath={.metadata.annotations.eks\.amazonaws\.com/role-arn}')"; then
+ die "Unable to inspect the ExternalDNS ServiceAccount after IRSA reconciliation."
+ fi
+ [ -n "$service_account_role_arn" ] || die \
+ "ServiceAccount kube-system/$EXTERNAL_DNS_SERVICE_ACCOUNT_NAME has no IRSA role annotation."
+}
+
+reconcile_external_dns() {
+ local helm_args
+
+ if [ "$MANAGE_EXTERNAL_DNS" != true ]; then
+ log "Skipping ExternalDNS reconciliation."
+ return
+ fi
+ if ! external_dns_configuration_present; then
+ log "ExternalDNS is not configured; skipping this optional add-on."
+ return
+ fi
+
+ require_command helm
+ reconcile_external_dns_iam_service_account
+
+ log "Refreshing the ExternalDNS Helm repository."
+ helm repo add external-dns https://kubernetes-sigs.github.io/external-dns/ \
+ --force-update >/dev/null
+ helm repo update external-dns >/dev/null
+
+ log "Reconciling the ExternalDNS DNSEndpoint CRD."
+ helm show crds external-dns/external-dns \
+ --version "$EXTERNAL_DNS_CHART_VERSION" | \
+ kubectl --context "$KUBE_CONTEXT" apply -f - >/dev/null
+ kubectl --context "$KUBE_CONTEXT" wait \
+ --for=condition=Established \
+ "customresourcedefinition/$EXTERNAL_DNS_ENDPOINT_CRD" \
+ --timeout 2m >/dev/null
+
+ helm_args=(
+ upgrade --install "$EXTERNAL_DNS_RELEASE_NAME" external-dns/external-dns
+ --kube-context "$KUBE_CONTEXT"
+ --namespace kube-system
+ --version "$EXTERNAL_DNS_CHART_VERSION"
+ --set-string "image.tag=$EXTERNAL_DNS_IMAGE_TAG"
+ --set provider.name=aws
+ --set 'sources[0]=service'
+ --set 'sources[1]=ingress'
+ --set 'sources[2]=crd'
+ --set policy=sync
+ --set registry=txt
+ --set-string "txtOwnerId=$EXTERNAL_DNS_OWNER_ID"
+ --set-string 'labelFilter=openremote.io/managed-dns=true'
+ --set triggerLoopOnEvent=true
+ --set serviceAccount.create=false
+ --set "serviceAccount.name=$EXTERNAL_DNS_SERVICE_ACCOUNT_NAME"
+ --set extraArgs.aws-zone-type=public
+ --set-string "extraArgs.zone-id-filter=$EXTERNAL_DNS_HOSTED_ZONE_ID"
+ --set-string "domainFilters[0]=$EXTERNAL_DNS_DOMAIN"
+ --set-string "extraArgs.aws-assume-role=$EXTERNAL_DNS_ASSUME_ROLE_ARN"
+ --wait
+ --timeout 10m
+ )
+ if [ -n "$EXTERNAL_DNS_ASSUME_ROLE_EXTERNAL_ID" ]; then
+ helm_args+=(--set-string \
+ "extraArgs.aws-assume-role-external-id=$EXTERNAL_DNS_ASSUME_ROLE_EXTERNAL_ID")
+ fi
+ if [ "$EXTERNAL_DNS_DRY_RUN" = true ]; then
+ helm_args+=(--set extraArgs.dry-run=true)
+ warn "ExternalDNS dry-run is enabled; DNS changes will only be logged."
+ fi
+
+ log "Reconciling ExternalDNS chart $EXTERNAL_DNS_CHART_VERSION with image $EXTERNAL_DNS_IMAGE_TAG."
+ helm "${helm_args[@]}"
+}
+
+reconcile_storage_class() {
+ [ -f "$DEFAULT_STORAGE_CLASS_FILE" ] || die \
+ "StorageClass manifest '$DEFAULT_STORAGE_CLASS_FILE' does not exist."
+
+ require_command kubectl
+ require_kube_context
+
+ log "Reconciling the OpenRemote EBS StorageClass."
+ kubectl --context "$KUBE_CONTEXT" apply -f "$DEFAULT_STORAGE_CLASS_FILE"
+}
+
+verify_network_policy_support() {
+ local vpc_cni_containers
+
+ vpc_cni_containers="$(kubectl --context "$KUBE_CONTEXT" --namespace kube-system \
+ get daemonset aws-node \
+ -o 'jsonpath={.spec.template.spec.containers[*].name}')" || die \
+ "Unable to inspect the Amazon VPC CNI DaemonSet."
+
+ case " $vpc_cni_containers " in
+ *" aws-eks-nodeagent "*)
+ ;;
+ *)
+ die "The Amazon VPC CNI NetworkPolicy agent is not installed."
+ ;;
+ esac
+
+ kubectl --context "$KUBE_CONTEXT" get \
+ customresourcedefinition/policyendpoints.networking.k8s.aws >/dev/null 2>&1 || die \
+ "The Amazon VPC CNI NetworkPolicy PolicyEndpoint CRD is unavailable."
+}
+
+show_vpc_cni_diagnostics() {
+ warn "Amazon VPC CNI add-on diagnostics:"
+ aws_eks describe-addon \
+ --cluster-name "$CLUSTER_NAME" \
+ --addon-name vpc-cni \
+ --query 'addon.{Status:status,Version:addonVersion,ServiceAccountRoleArn:serviceAccountRoleArn,Configuration:configurationValues,Issues:health.issues}' \
+ --output json >&2 || true
+ kubectl --context "$KUBE_CONTEXT" --namespace kube-system get daemonset aws-node \
+ -o custom-columns='NAME:.metadata.name,DESIRED:.status.desiredNumberScheduled,UPDATED:.status.updatedNumberScheduled,READY:.status.numberReady,AVAILABLE:.status.numberAvailable,CONTAINERS:.spec.template.spec.containers[*].name' >&2 || true
+ kubectl --context "$KUBE_CONTEXT" --namespace kube-system get pods \
+ --selector k8s-app=aws-node -o wide >&2 || true
+}
+
+reconcile_network_policy_support() {
+ local addon
+ local current_configuration
+ local current_service_account_role_arn
+ local desired_configuration
+ local service_account_role_arn
+ local update_id=""
+ local update_required=false
+ local update_args
+
+ require_command jq
+ require_command kubectl
+ require_kube_context
+
+ if ! addon="$(aws_eks describe-addon \
+ --cluster-name "$CLUSTER_NAME" \
+ --addon-name vpc-cni \
+ --query addon \
+ --output json)"; then
+ die "The managed vpc-cni add-on is required for EKS NetworkPolicy enforcement."
+ fi
+
+ current_configuration="$(jq -r '.configurationValues // "{}"' <<< "$addon")" || die \
+ "The vpc-cni add-on returned an unreadable configuration."
+ service_account_role_arn="$(jq -r '.serviceAccountRoleArn // empty' <<< "$addon")" || die \
+ "The vpc-cni add-on returned an unreadable service account role."
+
+ if [ -z "$current_configuration" ] || [ "$current_configuration" = "None" ]; then
+ current_configuration='{}'
+ fi
+
+ desired_configuration="$(jq -c '.enableNetworkPolicy = "true"' <<< "$current_configuration")" || die \
+ "The vpc-cni add-on returned invalid configuration values."
+
+ if [ -n "$service_account_role_arn" ]; then
+ if ! current_service_account_role_arn="$(kubectl --context "$KUBE_CONTEXT" --namespace kube-system \
+ get serviceaccount aws-node \
+ -o 'jsonpath={.metadata.annotations.eks\.amazonaws\.com/role-arn}')"; then
+ die "Unable to inspect the Amazon VPC CNI aws-node ServiceAccount."
+ fi
+
+ if [ "$current_service_account_role_arn" != "$service_account_role_arn" ]; then
+ log "Associating the Amazon VPC CNI IAM role with the aws-node ServiceAccount."
+ kubectl --context "$KUBE_CONTEXT" --namespace kube-system annotate serviceaccount aws-node \
+ "eks.amazonaws.com/role-arn=$service_account_role_arn" \
+ --overwrite >/dev/null
+ update_required=true
+ fi
+ else
+ warn "The managed vpc-cni add-on has no dedicated service account role; it must obtain its EC2 permissions from the node IAM role."
+ fi
+
+ if jq -e '(.enableNetworkPolicy | tostring) == "true"' \
+ <<< "$current_configuration" >/dev/null; then
+ log "Amazon VPC CNI NetworkPolicy enforcement is already enabled."
+ else
+ log "Enabling Amazon VPC CNI NetworkPolicy enforcement."
+ update_required=true
+ fi
+
+ if [ "$update_required" = true ]; then
+ update_args=(
+ update-addon
+ --cluster-name "$CLUSTER_NAME"
+ --addon-name vpc-cni
+ --resolve-conflicts PRESERVE
+ --configuration-values "$desired_configuration"
+ )
+ if [ -n "$service_account_role_arn" ]; then
+ update_args+=(--service-account-role-arn "$service_account_role_arn")
+ fi
+
+ if ! update_id="$(aws_eks "${update_args[@]}" --query 'update.id' --output text)"; then
+ show_vpc_cni_diagnostics
+ die "Unable to enable NetworkPolicy on the current vpc-cni version; update the add-on to a supported version before retrying."
+ fi
+
+ log "Waiting for Amazon VPC CNI add-on update '$update_id'."
+ fi
+
+ if ! aws_eks wait addon-active \
+ --cluster-name "$CLUSTER_NAME" \
+ --addon-name vpc-cni; then
+ if [ -n "$update_id" ]; then
+ aws_eks describe-update \
+ --name "$CLUSTER_NAME" \
+ --addon-name vpc-cni \
+ --update-id "$update_id" \
+ --query 'update.{Id:id,Status:status,Errors:errors}' \
+ --output json >&2 || true
+ fi
+ show_vpc_cni_diagnostics
+ die "Amazon VPC CNI did not become active; resolve the reported add-on or Pod health issue before retrying."
+ fi
+
+ kubectl --context "$KUBE_CONTEXT" --namespace kube-system rollout status \
+ daemonset/aws-node --timeout 10m
+ verify_network_policy_support
+}
+
+create_cluster() {
+ require_command aws
+ require_command eksctl
+ require_command kubectl
+ validate_shared_certificate_configuration
+ validate_external_dns_configuration
+ render_cluster_config
+
+ if cluster_exists; then
+ die "EKS cluster '$CLUSTER_NAME' already exists; use apply instead of create."
+ fi
+
+ log "Creating EKS cluster '$CLUSTER_NAME' in '$AWS_REGION_NAME'."
+ eksctl_with_profile create cluster --config-file "$RENDERED_CONFIG_FILE"
+ configure_kube_context
+ reconcile_network_policy_support
+ reconcile_storage_class
+ reconcile_load_balancer_controller
+ reconcile_external_dns
+ log "EKS cluster '$CLUSTER_NAME' is ready."
+}
+
+apply_cluster() {
+ local current_version
+ local desired_version
+
+ require_command aws
+ require_command kubectl
+ validate_shared_certificate_configuration
+ validate_external_dns_configuration
+ require_existing_cluster
+ render_cluster_config
+
+ current_version="$(current_kubernetes_version)"
+ desired_version="$(configured_kubernetes_version)"
+ if [ -n "$desired_version" ] && [ "$desired_version" != "$current_version" ]; then
+ die "Cluster configuration requests Kubernetes $desired_version but the cluster runs $current_version; Kubernetes version changes are outside the scope of this first-phase command."
+ fi
+
+ reconcile_network_policy_support
+ reconcile_storage_class
+ reconcile_load_balancer_controller
+ reconcile_external_dns
+ log "Shared cluster resources are reconciled. Node-group infrastructure updates remain outside apply until the cluster definition moves to CloudFormation."
+}
+
+show_external_dns_status() {
+ local crd_installed=false
+ local crd_source_enabled=false
+ local dry_run=false
+ local external_dns_args
+ local owner_id=""
+ local service_account_role_arn=""
+
+ log "ExternalDNS"
+ if ! helm status "$EXTERNAL_DNS_RELEASE_NAME" \
+ --kube-context "$KUBE_CONTEXT" \
+ --namespace kube-system >/dev/null 2>&1; then
+ log "Optional OpenRemote-managed ExternalDNS add-on is not installed."
+ return
+ fi
+ helm status "$EXTERNAL_DNS_RELEASE_NAME" \
+ --kube-context "$KUBE_CONTEXT" \
+ --namespace kube-system
+ kubectl --context "$KUBE_CONTEXT" \
+ --namespace kube-system \
+ get deployments,pods \
+ --selector "app.kubernetes.io/instance=$EXTERNAL_DNS_RELEASE_NAME" \
+ -o wide
+
+ if external_dns_args="$(kubectl --context "$KUBE_CONTEXT" \
+ --namespace kube-system \
+ get deployment "$EXTERNAL_DNS_RELEASE_NAME" \
+ -o 'go-template={{range (index .spec.template.spec.containers 0).args}}{{println .}}{{end}}' \
+ 2>/dev/null)"; then
+ owner_id="$(sed -n 's/^--txt-owner-id=//p' <<< "$external_dns_args" | head -n 1)"
+ if grep -Eq '^--dry-run(=true)?$' <<< "$external_dns_args"; then
+ dry_run=true
+ fi
+ if grep -Fx -- '--source=crd' <<< "$external_dns_args" >/dev/null; then
+ crd_source_enabled=true
+ fi
+ fi
+ if kubectl --context "$KUBE_CONTEXT" get \
+ "customresourcedefinition/$EXTERNAL_DNS_ENDPOINT_CRD" >/dev/null 2>&1; then
+ crd_installed=true
+ fi
+ service_account_role_arn="$(kubectl --context "$KUBE_CONTEXT" \
+ --namespace kube-system \
+ get serviceaccount "$EXTERNAL_DNS_SERVICE_ACCOUNT_NAME" \
+ -o 'jsonpath={.metadata.annotations.eks\.amazonaws\.com/role-arn}' \
+ 2>/dev/null || true)"
+
+ printf 'ExternalDNS TXT owner ID: %s\n' "${owner_id:-unknown}"
+ printf 'ExternalDNS dry run: %s\n' "$dry_run"
+ printf 'ExternalDNS IRSA role: %s\n' "${service_account_role_arn:-unknown}"
+ printf 'ExternalDNS DNSEndpoint CRD installed: %s\n' "$crd_installed"
+ printf 'ExternalDNS DNSEndpoint source enabled: %s\n' "$crd_source_enabled"
+}
+
+show_shared_certificate_status() {
+ local shared_certificate_arn
+
+ shared_certificate_arn="$(kubectl --context "$KUBE_CONTEXT" \
+ get ingressclass openremote-alb \
+ -o "go-template={{with index .metadata.annotations \"$SHARED_CERTIFICATE_ARN_ANNOTATION\"}}{{.}}{{end}}" \
+ 2>/dev/null || true)"
+ printf 'Shared ACM certificate ARN: %s\n' \
+ "${shared_certificate_arn:-not configured}"
+}
+
+show_cluster_status() {
+ require_command aws
+ require_command helm
+ require_command kubectl
+ require_existing_cluster
+
+ log "EKS control plane"
+ aws_eks describe-cluster \
+ --name "$CLUSTER_NAME" \
+ --query 'cluster.{Name:name,Status:status,KubernetesVersion:version,PlatformVersion:platformVersion,Endpoint:endpoint}' \
+ --output table
+
+ log "Managed node groups"
+ aws_eks list-nodegroups \
+ --cluster-name "$CLUSTER_NAME" \
+ --query 'nodegroups' \
+ --output table
+
+ log "EKS managed add-ons"
+ aws_eks list-addons \
+ --cluster-name "$CLUSTER_NAME" \
+ --query 'addons' \
+ --output table
+
+ log "Amazon VPC CNI NetworkPolicy configuration"
+ if ! aws_eks describe-addon \
+ --cluster-name "$CLUSTER_NAME" \
+ --addon-name vpc-cni \
+ --query 'addon.{Status:status,Version:addonVersion,ServiceAccountRoleArn:serviceAccountRoleArn,Configuration:configurationValues,Issues:health.issues}' \
+ --output table; then
+ warn "The managed vpc-cni add-on is unavailable."
+ fi
+
+ if kube_context_exists; then
+ require_kube_context
+ log "Kubernetes nodes"
+ kubectl --context "$KUBE_CONTEXT" get nodes \
+ -o custom-columns='NAME:.metadata.name,STATUS:.status.conditions[-1].type,VERSION:.status.nodeInfo.kubeletVersion,NODEGROUP:.metadata.labels.eks\.amazonaws\.com/nodegroup'
+
+ log "OpenRemote EBS StorageClass"
+ if ! kubectl --context "$KUBE_CONTEXT" get storageclass openremote-ebs; then
+ warn "StorageClass 'openremote-ebs' is not installed."
+ fi
+
+ log "Amazon VPC CNI NetworkPolicy agent"
+ if kubectl --context "$KUBE_CONTEXT" get \
+ customresourcedefinition/policyendpoints.networking.k8s.aws >/dev/null 2>&1; then
+ kubectl --context "$KUBE_CONTEXT" --namespace kube-system get daemonset aws-node \
+ -o custom-columns='NAME:.metadata.name,DESIRED:.status.desiredNumberScheduled,READY:.status.numberReady,CONTAINERS:.spec.template.spec.containers[*].name'
+ else
+ warn "Amazon VPC CNI NetworkPolicy enforcement is not enabled."
+ fi
+
+ if [ "$MANAGE_LOAD_BALANCER_CONTROLLER" = true ]; then
+ log "AWS Load Balancer Controller"
+ if ! helm status aws-load-balancer-controller \
+ --kube-context "$KUBE_CONTEXT" \
+ --namespace kube-system; then
+ warn "AWS Load Balancer Controller is not installed or is unhealthy."
+ fi
+
+ log "Shared OpenRemote ALB IngressClass"
+ if ! kubectl --context "$KUBE_CONTEXT" get ingressclass openremote-alb; then
+ warn "IngressClass 'openremote-alb' is not installed."
+ else
+ show_shared_certificate_status
+ if ! kubectl --context "$KUBE_CONTEXT" get \
+ ingressclassparams.elbv2.k8s.aws openremote-alb; then
+ warn "IngressClassParams 'openremote-alb' is not installed."
+ fi
+ fi
+ fi
+
+ show_external_dns_status
+ else
+ warn "Kubeconfig context '$KUBE_CONTEXT' is unavailable; Kubernetes-side status was skipped."
+ fi
+}
+
+destroy_preflight() {
+ local hostname_registry
+ local dns_endpoint_count=0
+ local dns_endpoint_crd
+ local ingress_count
+ local load_balancer_service_count
+ local pvc_count
+ local workload_pod_count
+ local stack_namespaces
+ local workload_controllers
+ local persistent_volumes
+
+ require_kube_context
+
+ hostname_registry="$(read_hostname_registry)" || die "Unable to inspect hostname reservations."
+ [ "$(jq '(.data // {}) | length' <<< "$hostname_registry")" -eq 0 ] || die \
+ "Hostname reservations remain; finish stack destroy or explicit external cleanup before destroying the cluster."
+
+ # Pods and PVCs alone do not prove that a cluster is empty: internal stacks
+ # need no hostname reservation, controllers can have zero Pods, and retained
+ # PVs can outlive their claims. Keep their metadata until cleanup is explicit.
+ stack_namespaces="$(kubectl --context "$KUBE_CONTEXT" get namespaces -o json | jq -r '
+ .items[] | select(.metadata.labels["openremote.io/stack"] != null or
+ .metadata.labels["app.kubernetes.io/part-of"] == "openremote") |
+ " Namespace/" + .metadata.name')" || die \
+ "Unable to inspect OpenRemote namespaces before cluster destruction."
+ workload_controllers="$(kubectl --context "$KUBE_CONTEXT" get \
+ deployments,statefulsets,daemonsets,replicasets,replicationcontrollers,jobs,cronjobs \
+ --all-namespaces -o json | jq -r '
+ .items[] | select(.metadata.namespace != "kube-system" and
+ .metadata.namespace != "kube-public" and .metadata.namespace != "kube-node-lease") |
+ " " + .kind + "/" + .metadata.namespace + "/" + .metadata.name')" || die \
+ "Unable to inspect workload controllers before cluster destruction."
+ persistent_volumes="$(kubectl --context "$KUBE_CONTEXT" get persistentvolumes -o json | jq -r '
+ .items[] | " PersistentVolume/" + .metadata.name')" || die \
+ "Unable to inspect PersistentVolumes before cluster destruction."
+
+ ingress_count="$(kubectl --context "$KUBE_CONTEXT" get ingress --all-namespaces -o json | jq '.items | length')"
+ load_balancer_service_count="$(kubectl --context "$KUBE_CONTEXT" get services --all-namespaces -o json | jq \
+ '[.items[] | select(.spec.type == "LoadBalancer")] | length')"
+ pvc_count="$(kubectl --context "$KUBE_CONTEXT" get persistentvolumeclaims --all-namespaces -o json | jq '.items | length')"
+ workload_pod_count="$(kubectl --context "$KUBE_CONTEXT" get pods --all-namespaces -o json | jq \
+ '[.items[] | select(.metadata.namespace != "kube-system" and .metadata.namespace != "kube-public" and .metadata.namespace != "kube-node-lease")] | length')"
+ dns_endpoint_crd="$(kubectl --context "$KUBE_CONTEXT" get \
+ customresourcedefinition/dnsendpoints.externaldns.k8s.io \
+ --ignore-not-found -o name)" || die \
+ "Unable to determine whether DNSEndpoint resources exist before cluster destruction."
+ if [ -n "$dns_endpoint_crd" ]; then
+ dns_endpoint_count="$(kubectl --context "$KUBE_CONTEXT" get dnsendpoints \
+ --all-namespaces -o json | jq '.items | length')"
+ fi
+
+ if [ -n "$stack_namespaces" ] || \
+ [ -n "$workload_controllers" ] || \
+ [ -n "$persistent_volumes" ] || \
+ [ "$ingress_count" -gt 0 ] || \
+ [ "$load_balancer_service_count" -gt 0 ] || \
+ [ "$pvc_count" -gt 0 ] || \
+ [ "$workload_pod_count" -gt 0 ] || \
+ [ "$dns_endpoint_count" -gt 0 ]; then
+ warn "Cluster destruction is blocked while stack or user resources remain:"
+ printf ' Ingresses: %s\n' "$ingress_count" >&2
+ printf ' LoadBalancer Services: %s\n' "$load_balancer_service_count" >&2
+ printf ' PersistentVolumeClaims: %s\n' "$pvc_count" >&2
+ printf ' Non-system Pods: %s\n' "$workload_pod_count" >&2
+ printf ' DNSEndpoints: %s\n' "$dns_endpoint_count" >&2
+ if [ -n "$stack_namespaces" ]; then printf '%s\n' "$stack_namespaces" >&2; fi
+ if [ -n "$workload_controllers" ]; then printf '%s\n' "$workload_controllers" >&2; fi
+ if [ -n "$persistent_volumes" ]; then printf '%s\n' "$persistent_volumes" >&2; fi
+ die "Remove stacks and preserve or purge their data explicitly before destroying the cluster."
+ fi
+}
+
+uninstall_external_dns() {
+ if [ "$MANAGE_EXTERNAL_DNS" = true ] && \
+ helm status "$EXTERNAL_DNS_RELEASE_NAME" \
+ --kube-context "$KUBE_CONTEXT" \
+ --namespace kube-system >/dev/null 2>&1; then
+ log "Uninstalling ExternalDNS after confirming that no public stack endpoints remain."
+ helm uninstall "$EXTERNAL_DNS_RELEASE_NAME" \
+ --kube-context "$KUBE_CONTEXT" \
+ --namespace kube-system \
+ --wait
+ fi
+}
+
+destroy_cluster() {
+ require_command aws
+ require_command eksctl
+ require_command helm
+ require_command jq
+ require_command kubectl
+ require_existing_cluster
+
+ [ "$CONFIRM_CLUSTER_NAME" = "$CLUSTER_NAME" ] || die \
+ "Destroy requires --confirm '$CLUSTER_NAME'."
+
+ destroy_preflight
+ uninstall_external_dns
+
+ if [ "$MANAGE_LOAD_BALANCER_CONTROLLER" = true ] && \
+ helm status aws-load-balancer-controller \
+ --kube-context "$KUBE_CONTEXT" \
+ --namespace kube-system >/dev/null 2>&1; then
+ log "Uninstalling the AWS Load Balancer Controller."
+ helm uninstall aws-load-balancer-controller \
+ --kube-context "$KUBE_CONTEXT" \
+ --namespace kube-system \
+ --wait
+ fi
+
+ log "Deleting EKS cluster '$CLUSTER_NAME' without waiting for system PodDisruptionBudgets."
+ eksctl_with_profile delete cluster \
+ --name "$CLUSTER_NAME" \
+ --region "$AWS_REGION_NAME" \
+ --disable-nodegroup-eviction \
+ --wait
+}
+
+main() {
+ parse_arguments "$@"
+
+ case "$COMMAND" in
+ create)
+ create_cluster
+ ;;
+ apply)
+ apply_cluster
+ ;;
+ status)
+ show_cluster_status
+ ;;
+ kubeconfig)
+ require_command aws
+ require_existing_cluster
+ configure_kube_context
+ ;;
+ destroy)
+ destroy_cluster
+ ;;
+ esac
+}
+
+if [ "${BASH_SOURCE[0]}" = "$0" ]; then
+ main "$@"
+fi
diff --git a/kubernetes/or-eks-stack b/kubernetes/or-eks-stack
new file mode 100755
index 000000000..9e56f0d4d
--- /dev/null
+++ b/kubernetes/or-eks-stack
@@ -0,0 +1,3695 @@
+#!/usr/bin/env bash
+
+# AWS-aware orchestration for an OpenRemote stack on an existing EKS cluster.
+# Namespace-scoped Kubernetes resources remain owned by the portable or-stack
+# command; this facade validates shared EKS services and coordinates endpoint
+# readiness around it.
+
+set -Eeuo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+source "$SCRIPT_DIR/lib/hostname-reservations"
+
+DEFAULT_AWS_REGION="eu-west-1"
+DEFAULT_TIMEOUT="20m"
+EXTERNAL_DNS_RELEASE_NAME="openremote-external-dns"
+EXTERNAL_DNS_SERVICE_ACCOUNT_NAME="external-dns"
+CERTIFICATE_MANAGED_BY_TAG_KEY="openremote.io/managed-by"
+CERTIFICATE_MANAGED_BY_TAG_VALUE="or-eks-stack"
+CERTIFICATE_CLUSTER_TAG_KEY="openremote.io/cluster"
+CERTIFICATE_STACK_TAG_KEY="openremote.io/stack"
+CERTIFICATE_HOSTNAME_TAG_KEY="openremote.io/hostname"
+CERTIFICATE_ENDPOINT_TAG_KEY="openremote.io/endpoint"
+CERTIFICATE_MODE_ANNOTATION="openremote.io/certificate-mode"
+CERTIFICATE_ARN_ANNOTATION="openremote.io/certificate-arn"
+HOSTNAME_ANNOTATION="openremote.io/hostname"
+MQTTS_CERTIFICATE_MODE_ANNOTATION="openremote.io/mqtts-certificate-mode"
+MQTTS_CERTIFICATE_ARN_ANNOTATION="openremote.io/mqtts-certificate-arn"
+MQTTS_HOSTNAME_ANNOTATION="openremote.io/mqtts-hostname"
+SHARED_CERTIFICATE_ARN_ANNOTATION="openremote.io/shared-certificate-arn"
+ACM_VALIDATION_DNS_ENDPOINT_NAME="acm-validation"
+ACM_MQTTS_VALIDATION_DNS_ENDPOINT_NAME="acm-validation-mqtts"
+
+COMMAND=""
+STACK_NAME="${OR_EKS_STACK_NAME:-}"
+CLUSTER_NAME="${OR_EKS_STACK_CLUSTER:-}"
+AWS_REGION_NAME="${OR_EKS_STACK_AWS_REGION:-${AWS_REGION:-$DEFAULT_AWS_REGION}}"
+# Only --profile selects an explicit profile; otherwise leave credential
+# resolution to AWS, including its standard environment settings.
+AWS_PROFILE_NAME=""
+KUBE_CONTEXT="${OR_EKS_STACK_KUBE_CONTEXT:-}"
+EXPOSURE="${OR_EKS_STACK_EXPOSURE:-haproxy}"
+HOSTNAME_VALUE="${OR_EKS_STACK_HOSTNAME:-}"
+CERTIFICATE_MODE="${OR_EKS_STACK_CERTIFICATE_MODE:-}"
+CERTIFICATE_ARN="${OR_EKS_STACK_CERTIFICATE_ARN:-}"
+VALUES_DIR="${OR_EKS_STACK_VALUES_DIR:-}"
+TIMEOUT="${OR_EKS_STACK_TIMEOUT:-$DEFAULT_TIMEOUT}"
+HTTP_PORT="${OR_EKS_STACK_HTTP_PORT:-80}"
+HTTPS_PORT="${OR_EKS_STACK_HTTPS_PORT:-443}"
+MQTTS_ENABLED="${OR_EKS_STACK_MQTTS:-false}"
+MQTTS_PORT="${OR_EKS_STACK_MQTTS_PORT:-8883}"
+MQTTS_PORT_CONFIGURED=false
+MQTTS_HOSTNAME_VALUE="${OR_EKS_STACK_MQTTS_HOSTNAME:-}"
+MQTTS_CERTIFICATE_MODE="${OR_EKS_STACK_MQTTS_CERTIFICATE_MODE:-}"
+MQTTS_CERTIFICATE_ARN="${OR_EKS_STACK_MQTTS_CERTIFICATE_ARN:-}"
+CONFIRM_STACK_NAME=""
+OR_STACK_COMMAND="${OR_EKS_STACK_OR_STACK:-$SCRIPT_DIR/or-stack}"
+LOAD_BALANCER_ADDRESS=""
+MQTTS_LOAD_BALANCER_ADDRESS=""
+EXTERNAL_DNS_ARGS=""
+CERTIFICATE_JSON=""
+CERTIFICATE_EXISTS=false
+MQTTS_CERTIFICATE_JSON=""
+MQTTS_CERTIFICATE_EXISTS=false
+STACK_NAMESPACE_EXISTS=false
+VALIDATION_DNS_RECORDS=""
+MQTTS_VALIDATION_DNS_RECORDS=""
+MANAGED_DNS_DOMAIN=""
+AUTHORITATIVE_NAME_SERVERS=""
+STACK_INGRESS_JSON=""
+STACK_MQTTS_SERVICE_JSON=""
+VALIDATION_DNS_ENDPOINT_EXISTS=false
+MQTTS_VALIDATION_DNS_ENDPOINT_EXISTS=false
+MANAGED_CERTIFICATE_DESTROY=false
+MANAGED_MQTTS_CERTIFICATE_DESTROY=false
+
+if [ "${OR_EKS_STACK_MQTTS_PORT+x}" = "x" ]; then
+ MQTTS_PORT_CONFIGURED=true
+fi
+
+usage() {
+ cat <<'EOF'
+Manage an OpenRemote stack on an existing Amazon EKS cluster.
+
+Usage:
+ or-eks-stack apply --name <name> --cluster <cluster> --hostname <host> [options]
+ or-eks-stack status --name <name> --cluster <cluster> [options]
+ or-eks-stack credentials --name <name> --cluster <cluster> [options]
+ or-eks-stack uninstall --name <name> --cluster <cluster> [options]
+ or-eks-stack destroy --name <name> --cluster <cluster> --confirm <name> [options]
+
+Commands:
+ apply Install or upgrade the stack, wait for its load balancer
+ and DNS, resolve TLS, and verify trusted HTTPS and any
+ enabled MQTTS endpoint.
+ status Show the stack's Kubernetes status.
+ credentials Show the Manager administrator login credentials.
+ uninstall Remove workloads while retaining namespace, credentials,
+ PVCs, and certificate state.
+ destroy Delete the stack namespace and its data after exact-name
+ confirmation.
+
+Options:
+ --name <name> Stack and namespace name. Env: OR_EKS_STACK_NAME.
+ --cluster <name> Existing EKS cluster name. Env: OR_EKS_STACK_CLUSTER.
+ --region <region> AWS region. Env: OR_EKS_STACK_AWS_REGION.
+ Default: eu-west-1.
+ --profile <name> Explicit AWS profile. Default: no profile argument;
+ use the normal AWS credential chain. An empty value
+ also leaves profile selection to AWS.
+ --kube-context <c> Kubernetes context. Defaults to <cluster>@<region>.
+ Env: OR_EKS_STACK_KUBE_CONTEXT.
+ --exposure <mode> Public exposure: haproxy or ingress. Env:
+ OR_EKS_STACK_EXPOSURE. Default: haproxy.
+ --hostname <host> Public, ExternalDNS-managed hostname. Required by apply.
+ Must be a strict subdomain of a managed DNS domain.
+ Env: OR_EKS_STACK_HOSTNAME.
+ --certificate-mode <mode>
+ Certificate ownership mode for Ingress: existing,
+ shared, or managed. Reapply preserves the recorded mode.
+ On first selection, omission chooses a configured shared
+ certificate, otherwise managed. In-place mode migration
+ is not supported. Env:
+ OR_EKS_STACK_CERTIFICATE_MODE.
+ --certificate-arn <arn>
+ Existing ACM ARN for mode existing; reapply can omit it
+ to reuse the recorded ARN. Env:
+ OR_EKS_STACK_CERTIFICATE_ARN. Mode shared reads the ARN
+ from the cluster's openremote-alb IngressClass.
+ --http-port <port> Public HAProxy Service HTTP port. Env:
+ OR_EKS_STACK_HTTP_PORT. HAProxy only; must remain 80 for
+ automated HTTP-01 certificate issuance.
+ --https-port <port> Public HAProxy Service HTTPS port. Env:
+ OR_EKS_STACK_HTTPS_PORT. HAProxy only; default: 443.
+ --mqtts Expose MQTTS. HAProxy uses the web hostname and its
+ own certificate; Ingress uses a dedicated NLB,
+ hostname, and ACM certificate. Env:
+ OR_EKS_STACK_MQTTS. Default: false. Plaintext MQTT is
+ not exposed.
+ --mqtts-port <port> Public MQTTS port. Env: OR_EKS_STACK_MQTTS_PORT.
+ Default: 8883. Requires --mqtts.
+ --mqtts-hostname <host>
+ Dedicated ExternalDNS-managed hostname for Ingress
+ MQTTS. Required on first apply; reapply may reuse the
+ recorded hostname. Env: OR_EKS_STACK_MQTTS_HOSTNAME.
+ --mqtts-certificate-mode <mode>
+ Certificate ownership mode for Ingress MQTTS: existing,
+ shared, or managed. An explicit ARN implies existing;
+ otherwise first apply selects the cluster's configured
+ shared certificate, falling back to managed. Reapply
+ preserves the recorded mode. Env:
+ OR_EKS_STACK_MQTTS_CERTIFICATE_MODE.
+ --mqtts-certificate-arn <arn>
+ Existing ACM certificate for the Ingress MQTTS NLB in
+ mode existing. Reapply may reuse the recorded ARN. Env:
+ OR_EKS_STACK_MQTTS_CERTIFICATE_ARN.
+ --values-dir <path> Optional per-component values directory passed to
+ or-stack. Env: OR_EKS_STACK_VALUES_DIR.
+ --timeout <duration> Timeout used by Helm and each endpoint readiness phase.
+ Accepts seconds or an s, m, or h suffix. Env:
+ OR_EKS_STACK_TIMEOUT. Default: 20m.
+ --confirm <name> Required exact stack name for destroy.
+ -h, --help Show this help.
+
+This is an AWS-aware facade over or-stack. It requires the OpenRemote-managed
+ExternalDNS release installed by or-eks-cluster, with active Route 53 writes
+for the requested hostname. The portable or-stack command remains available
+for local clusters, unmanaged DNS, and internal-only stacks.
+
+Before workloads or ACM requests, the facade prepares the namespace and reserves
+its hostname through or-stack. A hostname cannot belong to two stacks or change
+on an existing stack. Reservations survive uninstall and failed operations;
+destroy releases them only after external cleanup. See kubernetes/README.md for
+registry permissions, required stack metadata, and manual recovery. Destroy
+requires a hostname reservation in every mode; missing state is an error and
+never bypasses DNS cleanup.
+
+Ingress accepts an explicit existing ACM ARN, the externally managed shared
+certificate configured by or-eks-cluster, or a stack-owned managed certificate.
+Managed mode requests or reuses an exact-hostname ACM certificate and retains
+its namespaced DNS-validation record through uninstall. Destroy removes the
+stack route and namespace, waits for its public and validation DNS records to
+disappear, then deletes only the exactly owned ACM certificate.
+Managed mode requires exclusive use of the certificate and its validation
+records by this stack. Other certificates must not depend on those records;
+this deployment requirement is not established by the script's ownership checks.
+The first certificate mode is retained across failed applies and uninstall.
+Changing mode requires destroying and recreating the stack; back up any data
+to retain first. Certificate rotation within existing or shared mode is supported.
+Ingress MQTTS accepts an existing user-owned ACM certificate, the cluster's
+externally managed shared certificate, or a dedicated stack-managed
+certificate. Managed mode owns a separate ACM certificate and DNS-validation
+resource for the MQTTS endpoint; destroy deletes only that exactly tagged
+certificate after the NLB and validation records are removed.
+EOF
+}
+
+log() {
+ printf '[or-eks-stack] %s\n' "$*"
+}
+
+warn() {
+ printf '[or-eks-stack] WARNING: %s\n' "$*" >&2
+}
+
+die() {
+ printf '[or-eks-stack] ERROR: %s\n' "$*" >&2
+ exit 1
+}
+
+require_command() {
+ command -v "$1" >/dev/null 2>&1 || die \
+ "Required command '$1' was not found in PATH."
+}
+
+aws_cli() {
+ if [ -n "$AWS_PROFILE_NAME" ]; then
+ aws "$@" --region "$AWS_REGION_NAME" --profile "$AWS_PROFILE_NAME"
+ else
+ aws "$@" --region "$AWS_REGION_NAME"
+ fi
+}
+
+aws_eks() {
+ aws_cli eks "$@"
+}
+
+aws_acm() {
+ aws_cli acm "$@"
+}
+
+aws_elbv2() {
+ aws_cli elbv2 "$@"
+}
+
+aws_sts() {
+ aws_cli sts "$@"
+}
+
+run_or_stack() {
+ "$OR_STACK_COMMAND" "$@"
+}
+
+parse_arguments() {
+ if [ "$#" -eq 0 ]; then
+ usage >&2
+ exit 1
+ fi
+
+ case "$1" in
+ -h|--help)
+ usage
+ exit 0
+ ;;
+ apply|status|credentials|uninstall|destroy)
+ COMMAND="$1"
+ shift
+ ;;
+ *)
+ die "Unknown command '$1'. Run with --help for usage."
+ ;;
+ esac
+
+ while [ "$#" -gt 0 ]; do
+ case "$1" in
+ --name)
+ [ "$#" -ge 2 ] || die "Option --name requires a value."
+ STACK_NAME="$2"
+ shift 2
+ ;;
+ --cluster)
+ [ "$#" -ge 2 ] || die "Option --cluster requires a value."
+ CLUSTER_NAME="$2"
+ shift 2
+ ;;
+ --region)
+ [ "$#" -ge 2 ] || die "Option --region requires a value."
+ AWS_REGION_NAME="$2"
+ shift 2
+ ;;
+ --profile)
+ [ "$#" -ge 2 ] || die "Option --profile requires a value."
+ AWS_PROFILE_NAME="$2"
+ shift 2
+ ;;
+ --kube-context)
+ [ "$#" -ge 2 ] || die "Option --kube-context requires a value."
+ KUBE_CONTEXT="$2"
+ shift 2
+ ;;
+ --exposure)
+ [ "$#" -ge 2 ] || die "Option --exposure requires a value."
+ EXPOSURE="$2"
+ shift 2
+ ;;
+ --hostname)
+ [ "$#" -ge 2 ] || die "Option --hostname requires a value."
+ HOSTNAME_VALUE="$2"
+ shift 2
+ ;;
+ --certificate-mode)
+ [ "$#" -ge 2 ] || die "Option --certificate-mode requires a value."
+ CERTIFICATE_MODE="$2"
+ shift 2
+ ;;
+ --certificate-arn)
+ [ "$#" -ge 2 ] || die "Option --certificate-arn requires a value."
+ CERTIFICATE_ARN="$2"
+ shift 2
+ ;;
+ --http-port)
+ [ "$#" -ge 2 ] || die "Option --http-port requires a value."
+ HTTP_PORT="$2"
+ shift 2
+ ;;
+ --https-port)
+ [ "$#" -ge 2 ] || die "Option --https-port requires a value."
+ HTTPS_PORT="$2"
+ shift 2
+ ;;
+ --mqtts)
+ MQTTS_ENABLED=true
+ shift
+ ;;
+ --mqtts-port)
+ [ "$#" -ge 2 ] || die "Option --mqtts-port requires a value."
+ MQTTS_PORT="$2"
+ MQTTS_PORT_CONFIGURED=true
+ shift 2
+ ;;
+ --mqtts-hostname)
+ [ "$#" -ge 2 ] || die "Option --mqtts-hostname requires a value."
+ MQTTS_HOSTNAME_VALUE="$2"
+ shift 2
+ ;;
+ --mqtts-certificate-mode)
+ [ "$#" -ge 2 ] || die "Option --mqtts-certificate-mode requires a value."
+ MQTTS_CERTIFICATE_MODE="$2"
+ shift 2
+ ;;
+ --mqtts-certificate-arn)
+ [ "$#" -ge 2 ] || die "Option --mqtts-certificate-arn requires a value."
+ MQTTS_CERTIFICATE_ARN="$2"
+ shift 2
+ ;;
+ --values-dir)
+ [ "$#" -ge 2 ] || die "Option --values-dir requires a value."
+ VALUES_DIR="$2"
+ shift 2
+ ;;
+ --timeout)
+ [ "$#" -ge 2 ] || die "Option --timeout requires a value."
+ TIMEOUT="$2"
+ shift 2
+ ;;
+ --confirm)
+ [ "$#" -ge 2 ] || die "Option --confirm requires a value."
+ CONFIRM_STACK_NAME="$2"
+ shift 2
+ ;;
+ -h|--help)
+ usage
+ exit 0
+ ;;
+ *)
+ die "Unknown option '$1'. Run with --help for usage."
+ ;;
+ esac
+ done
+}
+
+validate_stack_name() {
+ [ -n "$STACK_NAME" ] || die "A stack name is required; use --name."
+ [ "${#STACK_NAME}" -le 63 ] || die \
+ "Stack name must contain at most 63 characters."
+ if [[ ! "$STACK_NAME" =~ ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ ]]; then
+ die "Stack name must be a lowercase Kubernetes namespace name."
+ fi
+
+ case "$STACK_NAME" in
+ default|kube-system|kube-public|kube-node-lease)
+ die "Namespace '$STACK_NAME' is reserved and cannot be used as a stack name."
+ ;;
+ esac
+}
+
+validate_cluster_name() {
+ [ -n "$CLUSTER_NAME" ] || die "An EKS cluster name is required; use --cluster."
+ [ "${#CLUSTER_NAME}" -le 100 ] || die \
+ "EKS cluster name must contain at most 100 characters."
+ if [[ ! "$CLUSTER_NAME" =~ ^[A-Za-z0-9][A-Za-z0-9_-]*$ ]]; then
+ die "EKS cluster name may contain only letters, numbers, underscores, and hyphens."
+ fi
+}
+
+validate_port() {
+ local port="$1"
+ local name="$2"
+
+ [[ "$port" =~ ^[0-9]+$ ]] || die \
+ "$name port must be a number from 1 to 65535."
+ [ "$port" -ge 1 ] && [ "$port" -le 65535 ] || die \
+ "$name port must be a number from 1 to 65535."
+}
+
+duration_seconds() {
+ local duration="$1"
+ local multiplier
+
+ if [[ ! "$duration" =~ ^([0-9]+)([smh]?)$ ]]; then
+ return 1
+ fi
+
+ case "${BASH_REMATCH[2]}" in
+ ""|s) multiplier=1 ;;
+ m) multiplier=60 ;;
+ h) multiplier=3600 ;;
+ *) return 1 ;;
+ esac
+
+ printf '%s\n' "$((10#${BASH_REMATCH[1]} * multiplier))"
+}
+
+validate_common_options() {
+ local timeout_seconds
+
+ validate_stack_name
+ validate_cluster_name
+ [ -n "$AWS_REGION_NAME" ] || die "AWS region cannot be empty."
+ if [ -z "$KUBE_CONTEXT" ]; then
+ KUBE_CONTEXT="${CLUSTER_NAME}@${AWS_REGION_NAME}"
+ fi
+ [ -n "$KUBE_CONTEXT" ] || die "Kubernetes context cannot be empty."
+ if ! timeout_seconds="$(duration_seconds "$TIMEOUT")"; then
+ die "Timeout must be seconds or a duration with an s, m, or h suffix."
+ fi
+ [ "$timeout_seconds" -gt 0 ] || die "Timeout must be greater than zero."
+}
+
+validate_hostname_value() {
+ local hostname="$1"
+ local description="$2"
+
+ [ -n "$hostname" ] || die "$description is required."
+ [ "${#hostname}" -le 253 ] || die \
+ "$description must contain at most 253 characters."
+ if [[ ! "$hostname" =~ ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)+$ ]]; then
+ die "$description must be a lowercase, fully qualified DNS name."
+ fi
+}
+
+validate_hostname() {
+ HOSTNAME_VALUE="$(printf '%s' "$HOSTNAME_VALUE" | tr '[:upper:]' '[:lower:]')"
+ HOSTNAME_VALUE="${HOSTNAME_VALUE%.}"
+ [ -n "$HOSTNAME_VALUE" ] || die \
+ "A public hostname is required for apply; use --hostname."
+ validate_hostname_value "$HOSTNAME_VALUE" "Hostname"
+}
+
+normalize_and_validate_mqtts_hostname() {
+ MQTTS_HOSTNAME_VALUE="$(printf '%s' "$MQTTS_HOSTNAME_VALUE" | tr '[:upper:]' '[:lower:]')"
+ MQTTS_HOSTNAME_VALUE="${MQTTS_HOSTNAME_VALUE%.}"
+ [ -n "$MQTTS_HOSTNAME_VALUE" ] || die \
+ "An MQTTS hostname is required for Ingress exposure; use --mqtts-hostname."
+ validate_hostname_value "$MQTTS_HOSTNAME_VALUE" "MQTTS hostname"
+}
+
+validate_apply_options() {
+ validate_hostname
+
+ case "$MQTTS_ENABLED" in
+ true|false)
+ ;;
+ *)
+ die "OR_EKS_STACK_MQTTS must be true or false."
+ ;;
+ esac
+ if [ "$MQTTS_ENABLED" != true ] && [ "$MQTTS_PORT_CONFIGURED" = true ]; then
+ die "Option --mqtts-port requires --mqtts."
+ fi
+ if [ "$MQTTS_ENABLED" != true ] && \
+ { [ -n "$MQTTS_HOSTNAME_VALUE" ] || [ -n "$MQTTS_CERTIFICATE_MODE" ] || \
+ [ -n "$MQTTS_CERTIFICATE_ARN" ]; }; then
+ die "Options --mqtts-hostname, --mqtts-certificate-mode, and --mqtts-certificate-arn require --mqtts."
+ fi
+
+ case "$EXPOSURE" in
+ haproxy)
+ [ -z "$CERTIFICATE_MODE" ] || die \
+ "Option --certificate-mode is only valid with Ingress exposure."
+ [ -z "$CERTIFICATE_ARN" ] || die \
+ "Option --certificate-arn is only valid with Ingress exposure."
+ [ -z "$MQTTS_HOSTNAME_VALUE" ] || die \
+ "Option --mqtts-hostname is only valid with Ingress exposure."
+ [ -z "$MQTTS_CERTIFICATE_MODE" ] || die \
+ "Option --mqtts-certificate-mode is only valid with Ingress exposure."
+ [ -z "$MQTTS_CERTIFICATE_ARN" ] || die \
+ "Option --mqtts-certificate-arn is only valid with Ingress exposure."
+ validate_port "$HTTP_PORT" "HTTP"
+ validate_port "$HTTPS_PORT" "HTTPS"
+ [ "$HTTP_PORT" != "$HTTPS_PORT" ] || die \
+ "HAProxy HTTP and HTTPS ports must be different."
+ if [ "$MQTTS_ENABLED" = true ]; then
+ validate_port "$MQTTS_PORT" "MQTTS"
+ [ "$MQTTS_PORT" != "$HTTP_PORT" ] && [ "$MQTTS_PORT" != "$HTTPS_PORT" ] || die \
+ "HAProxy HTTP, HTTPS, and MQTTS ports must be different."
+ fi
+ [ "$HTTP_PORT" = "80" ] || die \
+ "The automated HAProxy ACME workflow requires public HTTP port 80. Use or-stack directly for a custom HTTP port."
+ ;;
+ ingress)
+ if [ "$MQTTS_ENABLED" = true ]; then
+ validate_port "$MQTTS_PORT" "MQTTS"
+ if [ -n "$MQTTS_HOSTNAME_VALUE" ]; then
+ normalize_and_validate_mqtts_hostname
+ [ "$MQTTS_HOSTNAME_VALUE" != "$HOSTNAME_VALUE" ] || die \
+ "Ingress web and MQTTS hostnames must be different."
+ fi
+ if [ -n "$MQTTS_CERTIFICATE_ARN" ] && \
+ [[ ! "$MQTTS_CERTIFICATE_ARN" =~ ^arn:[^:]+:acm:[a-z0-9-]+:[0-9]{12}:certificate/[A-Za-z0-9-]+$ ]]; then
+ die "MQTTS certificate ARN must be a valid ACM certificate ARN."
+ fi
+ case "$MQTTS_CERTIFICATE_MODE" in
+ existing|shared|managed|"") ;;
+ *)
+ die "Unsupported Ingress MQTTS certificate mode '$MQTTS_CERTIFICATE_MODE'; expected existing, shared, or managed."
+ ;;
+ esac
+ if { [ "$MQTTS_CERTIFICATE_MODE" = shared ] || \
+ [ "$MQTTS_CERTIFICATE_MODE" = managed ]; } && \
+ [ -n "$MQTTS_CERTIFICATE_ARN" ]; then
+ die "Option --mqtts-certificate-arn requires --mqtts-certificate-mode existing; shared and managed modes resolve it automatically."
+ fi
+ fi
+ case "$CERTIFICATE_MODE" in
+ existing)
+ # An existing stack may reuse its recorded ARN; resolve it after
+ # reading namespace metadata, before validating the certificate.
+ if [ -n "$CERTIFICATE_ARN" ] && [[ ! "$CERTIFICATE_ARN" =~ ^arn:[^:]+:acm:[a-z0-9-]+:[0-9]{12}:certificate/[A-Za-z0-9-]+$ ]]; then
+ die "Certificate ARN must be a valid ACM certificate ARN."
+ fi
+ ;;
+ shared|managed|"")
+ [ -z "$CERTIFICATE_ARN" ] || die \
+ "Option --certificate-arn requires --certificate-mode existing; shared and managed modes resolve it automatically."
+ ;;
+ *)
+ die "Unsupported Ingress certificate mode '$CERTIFICATE_MODE'; expected existing, shared, or managed."
+ ;;
+ esac
+ [ "$HTTP_PORT" = "80" ] && [ "$HTTPS_PORT" = "443" ] || die \
+ "Options --http-port and --https-port are only configurable with HAProxy exposure."
+ ;;
+ *)
+ die "Unsupported exposure '$EXPOSURE'; expected haproxy or ingress."
+ ;;
+ esac
+
+ if [ -n "$VALUES_DIR" ] && [ ! -d "$VALUES_DIR" ]; then
+ die "Values directory '$VALUES_DIR' does not exist."
+ fi
+}
+
+validate_command_options() {
+ if [ "$COMMAND" != "destroy" ] && [ -n "$CONFIRM_STACK_NAME" ]; then
+ die "Option --confirm is only valid with destroy."
+ fi
+ if [ "$COMMAND" = "destroy" ] && [ "$CONFIRM_STACK_NAME" != "$STACK_NAME" ]; then
+ die "Destroy requires --confirm '$STACK_NAME'."
+ fi
+ if [ "$COMMAND" = "apply" ]; then
+ validate_apply_options
+ fi
+}
+
+require_active_cluster() {
+ local cluster_status
+ local output
+ local status
+
+ set +e
+ output="$(aws_eks describe-cluster \
+ --name "$CLUSTER_NAME" \
+ --query 'cluster.status' \
+ --output text 2>&1)"
+ status=$?
+ set -e
+
+ if [ "$status" -ne 0 ]; then
+ case "$output" in
+ *ResourceNotFoundException*)
+ die "EKS cluster '$CLUSTER_NAME' does not exist in region '$AWS_REGION_NAME'."
+ ;;
+ *)
+ die "Unable to query EKS cluster '$CLUSTER_NAME': $output"
+ ;;
+ esac
+ fi
+
+ cluster_status="$output"
+ [ "$cluster_status" = "ACTIVE" ] || die \
+ "EKS cluster '$CLUSTER_NAME' is '$cluster_status', not ACTIVE."
+}
+
+require_matching_kube_context() {
+ local configured_context
+ local context_endpoint
+ local expected_endpoint
+
+ configured_context="$(kubectl config get-contexts "$KUBE_CONTEXT" -o name \
+ 2>/dev/null || true)"
+ [ "$configured_context" = "$KUBE_CONTEXT" ] || die \
+ "Kubernetes context '$KUBE_CONTEXT' does not exist; run or-eks-cluster kubeconfig first."
+
+ expected_endpoint="$(aws_eks describe-cluster \
+ --name "$CLUSTER_NAME" \
+ --query 'cluster.endpoint' \
+ --output text)" || die \
+ "Unable to read the endpoint for EKS cluster '$CLUSTER_NAME'."
+ context_endpoint="$(kubectl --context "$KUBE_CONTEXT" config view \
+ --minify \
+ -o 'jsonpath={.clusters[0].cluster.server}')" || die \
+ "Unable to inspect Kubernetes context '$KUBE_CONTEXT'."
+
+ [ "$context_endpoint" = "$expected_endpoint" ] || die \
+ "Kubernetes context '$KUBE_CONTEXT' does not point to EKS cluster '$CLUSTER_NAME'."
+ kubectl --context "$KUBE_CONTEXT" get namespace kube-system >/dev/null 2>&1 || die \
+ "Kubernetes context '$KUBE_CONTEXT' cannot reach the EKS cluster."
+}
+
+require_eks_target() {
+ require_command aws
+ require_command kubectl
+ require_active_cluster
+ require_matching_kube_context
+}
+
+require_external_dns_argument() {
+ local external_dns_args="$1"
+ local required_argument="$2"
+
+ grep -Fx -- "$required_argument" <<< "$external_dns_args" >/dev/null || die \
+ "ExternalDNS is missing required argument '$required_argument'; run or-eks-cluster apply with the managed DNS configuration."
+}
+
+dns_name_is_covered() {
+ local dns_name="$1"
+ local domain_filter="$2"
+
+ dns_name="${dns_name%.}"
+ domain_filter="${domain_filter%.}"
+ [ "$dns_name" = "$domain_filter" ] || \
+ [[ "$dns_name" == *."$domain_filter" ]]
+}
+
+hostname_is_covered() {
+ local domain_filter="$1"
+ local hostname="${2:-$HOSTNAME_VALUE}"
+ dns_name_is_covered "$hostname" "$domain_filter"
+}
+
+require_managed_external_dns() {
+ local domain_filter
+ local external_dns_args
+ local hostname_covered=false
+ local mqtts_hostname_covered=false
+ local service_account_role_arn
+
+ helm status "$EXTERNAL_DNS_RELEASE_NAME" \
+ --kube-context "$KUBE_CONTEXT" \
+ --namespace kube-system >/dev/null 2>&1 || die \
+ "OpenRemote-managed ExternalDNS is not installed; configure it with or-eks-cluster apply first."
+
+ kubectl --context "$KUBE_CONTEXT" --namespace kube-system rollout status \
+ "deployment/$EXTERNAL_DNS_RELEASE_NAME" --timeout "$TIMEOUT" >/dev/null || die \
+ "ExternalDNS is not ready; inspect deployment/$EXTERNAL_DNS_RELEASE_NAME in kube-system."
+
+ external_dns_args="$(kubectl --context "$KUBE_CONTEXT" \
+ --namespace kube-system \
+ get deployment "$EXTERNAL_DNS_RELEASE_NAME" \
+ -o 'go-template={{range (index .spec.template.spec.containers 0).args}}{{println .}}{{end}}')" || die \
+ "Unable to inspect the ExternalDNS deployment."
+
+ if [ "$EXPOSURE" = "ingress" ]; then
+ require_external_dns_argument "$external_dns_args" "--source=ingress"
+ if [ "$MQTTS_ENABLED" = true ]; then
+ require_external_dns_argument "$external_dns_args" "--source=service"
+ fi
+ else
+ require_external_dns_argument "$external_dns_args" "--source=service"
+ fi
+ require_external_dns_argument "$external_dns_args" "--policy=sync"
+ require_external_dns_argument "$external_dns_args" "--registry=txt"
+ require_external_dns_argument "$external_dns_args" \
+ "--label-filter=openremote.io/managed-dns=true"
+ if [ "$CERTIFICATE_MODE" = "managed" ] || \
+ [ "$MQTTS_CERTIFICATE_MODE" = "managed" ]; then
+ require_external_dns_argument "$external_dns_args" "--source=crd"
+ kubectl --context "$KUBE_CONTEXT" get \
+ customresourcedefinition/dnsendpoints.externaldns.k8s.io >/dev/null 2>&1 || die \
+ "ExternalDNS DNSEndpoint support is missing; run or-eks-cluster apply before using managed certificates."
+ fi
+ grep -Eq '^--txt-owner-id=.+$' <<< "$external_dns_args" || die \
+ "ExternalDNS has no TXT owner ID; reconcile its managed configuration before applying a stack."
+ if grep -Eq '^--dry-run(=true)?$' <<< "$external_dns_args"; then
+ die "ExternalDNS dry-run is enabled; disable it with or-eks-cluster apply before applying a stack."
+ fi
+
+ while IFS= read -r domain_filter; do
+ [ -n "$domain_filter" ] || continue
+ if hostname_is_covered "$domain_filter"; then
+ hostname_covered=true
+ fi
+ if [ "$MQTTS_ENABLED" = true ] && [ "$EXPOSURE" = ingress ] && \
+ hostname_is_covered "$domain_filter" "$MQTTS_HOSTNAME_VALUE"; then
+ mqtts_hostname_covered=true
+ fi
+ done < <(sed -n 's/^--domain-filter=//p' <<< "$external_dns_args")
+
+ [ "$hostname_covered" = true ] || die \
+ "ExternalDNS domain filters do not cover '$HOSTNAME_VALUE'; reconcile the cluster DNS configuration first."
+ if [ "$MQTTS_ENABLED" = true ] && [ "$EXPOSURE" = ingress ]; then
+ [ "$mqtts_hostname_covered" = true ] || die \
+ "ExternalDNS domain filters do not cover MQTTS hostname '$MQTTS_HOSTNAME_VALUE'; reconcile the cluster DNS configuration first."
+ fi
+
+ service_account_role_arn="$(kubectl --context "$KUBE_CONTEXT" \
+ --namespace kube-system \
+ get serviceaccount "$EXTERNAL_DNS_SERVICE_ACCOUNT_NAME" \
+ -o 'jsonpath={.metadata.annotations.eks\.amazonaws\.com/role-arn}')" || die \
+ "Unable to inspect the ExternalDNS ServiceAccount."
+ [ -n "$service_account_role_arn" ] || die \
+ "ExternalDNS ServiceAccount kube-system/$EXTERNAL_DNS_SERVICE_ACCOUNT_NAME has no IRSA role annotation."
+
+ EXTERNAL_DNS_ARGS="$external_dns_args"
+}
+
+require_stack_hostname_subdomain() {
+ local domain_filter
+ local hostname="${1:-$HOSTNAME_VALUE}"
+ # ExternalDNS prefixes ownership TXT names with the record type. At the
+ # managed domain itself that name leaves the IAM domain boundary (for example,
+ # example.com -> a-example.com). Nested subdomains keep it inside the boundary.
+ # This is an apply-only check so it never prevents cleanup of existing state.
+ while IFS= read -r domain_filter; do
+ domain_filter="${domain_filter%.}"
+ [ -n "$domain_filter" ] || continue
+ if [[ "$hostname" == *."$domain_filter" ]]; then
+ return
+ fi
+ done < <(sed -n 's/^--domain-filter=//p' <<< "$EXTERNAL_DNS_ARGS")
+ die "Hostname '$hostname' must be a strict subdomain of a managed ExternalDNS domain; using the domain itself would place ownership TXT records outside the permitted DNS boundary."
+}
+
+read_certificate_namespace() {
+ local namespace_json
+ namespace_json="$(kubectl --context "$KUBE_CONTEXT" get namespace "$STACK_NAME" \
+ --ignore-not-found -o json)" || die "Unable to read the stack's certificate configuration."
+ if [ -n "$namespace_json" ]; then
+ jq -e --arg stack "$STACK_NAME" '.metadata.labels["openremote.io/stack"] == $stack' \
+ <<< "$namespace_json" >/dev/null || die \
+ "Namespace '$STACK_NAME' is not owned by this stack; refusing to use its certificate configuration."
+ fi
+ printf '%s\n' "$namespace_json"
+}
+
+resolve_ingress_certificate() {
+ local shared_certificate_arn namespace_json recorded_mode recorded_arn
+
+ namespace_json="$(read_certificate_namespace)" || die "Unable to resolve the stack's certificate mode."
+ recorded_mode="$(jq -r '.metadata.annotations["openremote.io/certificate-mode"] // empty' <<< "$namespace_json")"
+ recorded_arn="$(jq -r '.metadata.annotations["openremote.io/certificate-arn"] // empty' <<< "$namespace_json")"
+ if [ -n "$recorded_mode" ]; then
+ case "$recorded_mode" in
+ existing|shared|managed) ;;
+ *) die "Stack '$STACK_NAME' records incompatible certificate mode '$recorded_mode'." ;;
+ esac
+ [ -z "$CERTIFICATE_MODE" ] || [ "$CERTIFICATE_MODE" = "$recorded_mode" ] || die \
+ "Stack '$STACK_NAME' uses certificate mode '$recorded_mode'; in-place certificate mode migration to '$CERTIFICATE_MODE' is not supported."
+ # Cluster defaults apply only to first selection. Re-evaluating them on
+ # reapply can abandon a managed certificate and make destroy inconsistent.
+ CERTIFICATE_MODE="$recorded_mode"
+ fi
+
+ case "$CERTIFICATE_MODE" in
+ existing)
+ CERTIFICATE_ARN="${CERTIFICATE_ARN:-$recorded_arn}"
+ [ -n "$CERTIFICATE_ARN" ] || die "Ingress with certificate mode existing requires --certificate-arn on first apply."
+ return
+ ;;
+ managed)
+ return
+ ;;
+ esac
+
+ if ! shared_certificate_arn="$(kubectl --context "$KUBE_CONTEXT" \
+ get ingressclass openremote-alb \
+ -o "go-template={{with .metadata.annotations}}{{with index . \"$SHARED_CERTIFICATE_ARN_ANNOTATION\"}}{{.}}{{end}}{{end}}" \
+ 2>/dev/null)"; then
+ die "Unable to inspect shared certificate configuration on IngressClass 'openremote-alb'."
+ fi
+ if [ -z "$shared_certificate_arn" ]; then
+ if [ "$CERTIFICATE_MODE" = "shared" ]; then
+ die "Ingress certificate mode shared requires a certificate configured with or-eks-cluster --shared-certificate-arn."
+ fi
+ CERTIFICATE_MODE=managed
+ log "No shared ACM certificate is configured; selected stack-managed certificate mode."
+ return
+ fi
+
+ CERTIFICATE_MODE=shared
+ CERTIFICATE_ARN="$shared_certificate_arn"
+ log "Selected the cluster's shared ACM certificate '$CERTIFICATE_ARN'."
+}
+
+resolve_ingress_mqtts_configuration() {
+ local namespace_json
+ local recorded_arn
+ local recorded_hostname
+ local recorded_mode
+ local requested_arn="$MQTTS_CERTIFICATE_ARN"
+ local requested_mode="$MQTTS_CERTIFICATE_MODE"
+ local shared_certificate_arn=""
+
+ namespace_json="$(read_certificate_namespace)" || die \
+ "Unable to resolve the stack's MQTTS certificate configuration."
+ recorded_hostname="$(jq -r '.metadata.annotations["openremote.io/mqtts-hostname"] // empty' \
+ <<< "$namespace_json")"
+ recorded_mode="$(jq -r '.metadata.annotations["openremote.io/mqtts-certificate-mode"] // empty' \
+ <<< "$namespace_json")"
+ recorded_arn="$(jq -r '.metadata.annotations["openremote.io/mqtts-certificate-arn"] // empty' \
+ <<< "$namespace_json")"
+
+ if [ "$MQTTS_ENABLED" != true ]; then
+ if [ -n "$recorded_hostname" ] || [ -n "$recorded_mode" ] || [ -n "$recorded_arn" ]; then
+ die "Stack '$STACK_NAME' records an Ingress MQTTS endpoint; repeat --mqtts to preserve it. Disabling it in place is not supported."
+ fi
+ return
+ fi
+
+ case "$recorded_mode" in
+ existing|shared|managed|"") ;;
+ *) die "Stack '$STACK_NAME' records incompatible MQTTS certificate mode '$recorded_mode'." ;;
+ esac
+ if [ -z "$recorded_mode" ] && [ -n "$recorded_arn" ]; then
+ die "Stack '$STACK_NAME' has incomplete MQTTS certificate state; inspect its namespace annotations before retrying."
+ fi
+ if [ -n "$recorded_mode" ]; then
+ [ -z "$requested_mode" ] || [ "$requested_mode" = "$recorded_mode" ] || die \
+ "Stack '$STACK_NAME' uses MQTTS certificate mode '$recorded_mode'; in-place certificate mode migration to '$requested_mode' is not supported."
+ MQTTS_CERTIFICATE_MODE="$recorded_mode"
+ elif [ -z "$MQTTS_CERTIFICATE_MODE" ]; then
+ if [ -n "$requested_arn" ]; then
+ MQTTS_CERTIFICATE_MODE=existing
+ else
+ if ! shared_certificate_arn="$(kubectl --context "$KUBE_CONTEXT" \
+ get ingressclass openremote-alb \
+ -o "go-template={{with .metadata.annotations}}{{with index . \"$SHARED_CERTIFICATE_ARN_ANNOTATION\"}}{{.}}{{end}}{{end}}" \
+ 2>/dev/null)"; then
+ die "Unable to inspect shared MQTTS certificate configuration on IngressClass 'openremote-alb'."
+ fi
+ if [ -n "$shared_certificate_arn" ]; then
+ MQTTS_CERTIFICATE_MODE=shared
+ else
+ MQTTS_CERTIFICATE_MODE=managed
+ log "No shared MQTTS certificate is configured; selected stack-managed MQTTS certificate mode."
+ fi
+ fi
+ fi
+
+ if [ -n "$recorded_hostname" ]; then
+ if [ -n "$MQTTS_HOSTNAME_VALUE" ] && \
+ [ "${MQTTS_HOSTNAME_VALUE%.}" != "${recorded_hostname%.}" ]; then
+ die "Stack '$STACK_NAME' uses MQTTS hostname '$recorded_hostname'; in-place hostname migration to '$MQTTS_HOSTNAME_VALUE' is not supported."
+ fi
+ MQTTS_HOSTNAME_VALUE="$recorded_hostname"
+ fi
+ [ -n "$MQTTS_HOSTNAME_VALUE" ] || die \
+ "Ingress MQTTS requires --mqtts-hostname on first apply."
+ normalize_and_validate_mqtts_hostname
+ [ "$MQTTS_HOSTNAME_VALUE" != "$HOSTNAME_VALUE" ] || die \
+ "Ingress web and MQTTS hostnames must be different."
+
+ case "$MQTTS_CERTIFICATE_MODE" in
+ existing)
+ MQTTS_CERTIFICATE_ARN="${requested_arn:-$recorded_arn}"
+ [ -n "$MQTTS_CERTIFICATE_ARN" ] || die \
+ "Ingress MQTTS certificate mode existing requires --mqtts-certificate-arn on first apply."
+ ;;
+ shared)
+ [ -z "$requested_arn" ] || die \
+ "Option --mqtts-certificate-arn cannot be used with shared MQTTS certificate mode."
+ if [ -z "$shared_certificate_arn" ]; then
+ if ! shared_certificate_arn="$(kubectl --context "$KUBE_CONTEXT" \
+ get ingressclass openremote-alb \
+ -o "go-template={{with .metadata.annotations}}{{with index . \"$SHARED_CERTIFICATE_ARN_ANNOTATION\"}}{{.}}{{end}}{{end}}" \
+ 2>/dev/null)"; then
+ die "Unable to inspect shared MQTTS certificate configuration on IngressClass 'openremote-alb'."
+ fi
+ fi
+ [ -n "$shared_certificate_arn" ] || die \
+ "Ingress MQTTS certificate mode shared requires a certificate configured with or-eks-cluster --shared-certificate-arn."
+ MQTTS_CERTIFICATE_ARN="$shared_certificate_arn"
+ log "Selected the cluster's shared ACM certificate '$MQTTS_CERTIFICATE_ARN' for MQTTS."
+ ;;
+ managed)
+ [ -z "$requested_arn" ] || die \
+ "Option --mqtts-certificate-arn cannot be used with managed MQTTS certificate mode."
+ # A recorded managed ARN is loaded and ownership-checked after the
+ # namespace and hostname reservations have been prepared.
+ MQTTS_CERTIFICATE_ARN=""
+ ;;
+ esac
+}
+
+record_initial_certificate_mode() {
+ local namespace_json recorded_mode resource_version
+ local annotations=("$CERTIFICATE_MODE_ANNOTATION=$CERTIFICATE_MODE")
+ namespace_json="$(read_certificate_namespace)" || die "Unable to record the stack's certificate mode."
+ [ -n "$namespace_json" ] || die "Prepared namespace '$STACK_NAME' is missing; unable to record certificate mode."
+ recorded_mode="$(jq -r '.metadata.annotations["openremote.io/certificate-mode"] // empty' <<< "$namespace_json")"
+ if [ -n "$recorded_mode" ]; then
+ [ "$recorded_mode" = "$CERTIFICATE_MODE" ] || die \
+ "Stack '$STACK_NAME' now uses certificate mode '$recorded_mode'; in-place certificate mode migration is not supported. Retry apply with that mode."
+ return
+ fi
+ resource_version="$(jq -er '.metadata.resourceVersion | select(type == "string" and length > 0)' \
+ <<< "$namespace_json")" || die "Unable to verify namespace version before recording certificate mode."
+ if [ "$CERTIFICATE_MODE" = existing ] || [ "$CERTIFICATE_MODE" = shared ]; then
+ annotations+=("$CERTIFICATE_ARN_ANNOTATION=$CERTIFICATE_ARN")
+ fi
+ # Pin the mode before ACM requests or Helm. The conditional write prevents
+ # two initial applies from choosing different modes. Retain it after failure;
+ # managed mode without an ARN resumes through existing tagged-ACM discovery.
+ kubectl --context "$KUBE_CONTEXT" annotate namespace "$STACK_NAME" \
+ "${annotations[@]}" --overwrite --resource-version="$resource_version" >/dev/null || die \
+ "Unable to record certificate mode; the namespace may have changed concurrently. Retry apply."
+}
+
+record_initial_mqtts_configuration() {
+ local namespace_json
+ local recorded_hostname
+ local recorded_mode
+ local resource_version
+
+ [ "$EXPOSURE" = ingress ] && [ "$MQTTS_ENABLED" = true ] || return 0
+ namespace_json="$(read_certificate_namespace)" || die \
+ "Unable to record the stack's MQTTS lifecycle configuration."
+ [ -n "$namespace_json" ] || die \
+ "Prepared namespace '$STACK_NAME' is missing; unable to record MQTTS lifecycle configuration."
+ recorded_mode="$(jq -r '.metadata.annotations["openremote.io/mqtts-certificate-mode"] // empty' \
+ <<< "$namespace_json")"
+ recorded_hostname="$(jq -r '.metadata.annotations["openremote.io/mqtts-hostname"] // empty' \
+ <<< "$namespace_json")"
+ if [ -n "$recorded_mode" ]; then
+ [ "$recorded_mode" = "$MQTTS_CERTIFICATE_MODE" ] || die \
+ "Stack '$STACK_NAME' now uses MQTTS certificate mode '$recorded_mode', not '$MQTTS_CERTIFICATE_MODE'."
+ [ -n "$recorded_hostname" ] || die \
+ "Stack '$STACK_NAME' has incomplete MQTTS lifecycle configuration."
+ [ "$recorded_hostname" = "$MQTTS_HOSTNAME_VALUE" ] || die \
+ "Stack '$STACK_NAME' now records MQTTS hostname '$recorded_hostname', not '$MQTTS_HOSTNAME_VALUE'."
+ return
+ fi
+ resource_version="$(jq -er '.metadata.resourceVersion | select(type == "string" and length > 0)' \
+ <<< "$namespace_json")" || die \
+ "Unable to verify namespace version before recording MQTTS lifecycle configuration."
+ # Record the facade's ownership mode before ACM requests or Helm. If apply is
+ # interrupted, destroy can still distinguish this workflow from a low-level
+ # or-stack endpoint. Existing-mode ARNs are recorded only after Helm succeeds
+ # so rotation never advances namespace state ahead of the Service. Managed
+ # mode records its ARN immediately after an owned certificate is resolved.
+ kubectl --context "$KUBE_CONTEXT" annotate namespace "$STACK_NAME" \
+ "$MQTTS_CERTIFICATE_MODE_ANNOTATION=$MQTTS_CERTIFICATE_MODE" \
+ "$MQTTS_HOSTNAME_ANNOTATION=$MQTTS_HOSTNAME_VALUE" \
+ --overwrite --resource-version="$resource_version" >/dev/null || die \
+ "Unable to record MQTTS lifecycle configuration; the namespace may have changed concurrently. Retry apply."
+}
+
+certificate_name_covers_hostname() {
+ local certificate_name
+ local hostname="${2:-$HOSTNAME_VALUE}"
+ local wildcard_suffix
+
+ certificate_name="$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]')"
+ certificate_name="${certificate_name%.}"
+ if [ "$certificate_name" = "$hostname" ]; then
+ return 0
+ fi
+
+ case "$certificate_name" in
+ \*.*)
+ wildcard_suffix="${certificate_name#*.}"
+ [ "${hostname#*.}" = "$wildcard_suffix" ]
+ ;;
+ *)
+ return 1
+ ;;
+ esac
+}
+
+require_selected_mqtts_acm_certificate() {
+ local certificate_account
+ local certificate_json
+ local certificate_name
+ local certificate_region
+ local certificate_status
+ local certificate_tags
+ local current_account
+ local describe_status
+ local hostname_covered=false
+ local tags_status
+
+ if [[ ! "$MQTTS_CERTIFICATE_ARN" =~ ^arn:[^:]+:acm:([^:]+):([0-9]{12}):certificate/[A-Za-z0-9-]+$ ]]; then
+ die "MQTTS certificate ARN must be a valid ACM certificate ARN."
+ fi
+ certificate_region="${BASH_REMATCH[1]}"
+ certificate_account="${BASH_REMATCH[2]}"
+ [ "$certificate_region" = "$AWS_REGION_NAME" ] || die \
+ "MQTTS certificate '$MQTTS_CERTIFICATE_ARN' is in region '$certificate_region', but cluster '$CLUSTER_NAME' is in '$AWS_REGION_NAME'."
+ current_account="$(aws_sts get-caller-identity --query Account --output text)" || die \
+ "Unable to determine the EKS AWS account."
+ [ "$certificate_account" = "$current_account" ] || die \
+ "MQTTS certificate '$MQTTS_CERTIFICATE_ARN' belongs to account '$certificate_account', but the EKS account is '$current_account'."
+
+ set +e
+ certificate_json="$(aws_acm describe-certificate \
+ --certificate-arn "$MQTTS_CERTIFICATE_ARN" \
+ --output json 2>&1)"
+ describe_status=$?
+ set -e
+ [ "$describe_status" -eq 0 ] || die \
+ "Unable to describe MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN': $certificate_json"
+ certificate_status="$(jq -r '.Certificate.Status // empty' <<< "$certificate_json")" || die \
+ "Unable to read MQTTS ACM certificate metadata."
+ [ "$certificate_status" = ISSUED ] || die \
+ "The MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN' is '$certificate_status', not ISSUED."
+
+ while IFS= read -r certificate_name; do
+ [ -n "$certificate_name" ] || continue
+ if certificate_name_covers_hostname "$certificate_name" "$MQTTS_HOSTNAME_VALUE"; then
+ hostname_covered=true
+ break
+ fi
+ done < <(jq -r \
+ '[.Certificate.DomainName, .Certificate.SubjectAlternativeNames[]?] | unique[] | select(. != null)' \
+ <<< "$certificate_json")
+ [ "$hostname_covered" = true ] || die \
+ "The MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN' does not cover '$MQTTS_HOSTNAME_VALUE'."
+
+ set +e
+ certificate_tags="$(aws_acm list-tags-for-certificate \
+ --certificate-arn "$MQTTS_CERTIFICATE_ARN" \
+ --output json 2>&1)"
+ tags_status=$?
+ set -e
+ [ "$tags_status" -eq 0 ] || die \
+ "Unable to inspect ownership tags on MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN': $certificate_tags"
+ if jq -e \
+ --arg key "$CERTIFICATE_MANAGED_BY_TAG_KEY" \
+ --arg value "$CERTIFICATE_MANAGED_BY_TAG_VALUE" \
+ '.Tags[]? | select(.Key == $key and .Value == $value)' \
+ <<< "$certificate_tags" >/dev/null; then
+ if [ "$MQTTS_CERTIFICATE_MODE" = shared ]; then
+ die "MQTTS certificate '$MQTTS_CERTIFICATE_ARN' is marked as managed by or-eks-stack and cannot be used in shared mode."
+ fi
+ die "MQTTS certificate '$MQTTS_CERTIFICATE_ARN' is marked as managed by or-eks-stack and cannot be used as an existing certificate."
+ fi
+
+ if [ "$MQTTS_CERTIFICATE_MODE" = shared ]; then
+ log "Validated externally managed shared ACM certificate '$MQTTS_CERTIFICATE_ARN' for MQTTS hostname '$MQTTS_HOSTNAME_VALUE'."
+ else
+ log "Validated user-owned ACM certificate '$MQTTS_CERTIFICATE_ARN' for MQTTS hostname '$MQTTS_HOSTNAME_VALUE'."
+ fi
+}
+
+require_selected_acm_certificate() {
+ local certificate_account
+ local certificate_json
+ local certificate_name
+ local certificate_region
+ local certificate_status
+ local certificate_tags
+ local current_account
+ local describe_status
+ local hostname_covered=false
+ local tags_status
+
+ if [[ ! "$CERTIFICATE_ARN" =~ ^arn:[^:]+:acm:([^:]+):([0-9]{12}):certificate/[A-Za-z0-9-]+$ ]]; then
+ die "Certificate ARN must be a valid ACM certificate ARN."
+ fi
+ certificate_region="${BASH_REMATCH[1]}"
+ certificate_account="${BASH_REMATCH[2]}"
+
+ [ "$certificate_region" = "$AWS_REGION_NAME" ] || die \
+ "Certificate '$CERTIFICATE_ARN' is in region '$certificate_region', but cluster '$CLUSTER_NAME' is in '$AWS_REGION_NAME'."
+ current_account="$(aws_sts get-caller-identity --query Account --output text)" || die \
+ "Unable to determine the EKS AWS account."
+ [ "$certificate_account" = "$current_account" ] || die \
+ "Certificate '$CERTIFICATE_ARN' belongs to account '$certificate_account', but the EKS account is '$current_account'."
+
+ set +e
+ certificate_json="$(aws_acm describe-certificate \
+ --certificate-arn "$CERTIFICATE_ARN" \
+ --output json 2>&1)"
+ describe_status=$?
+ set -e
+ [ "$describe_status" -eq 0 ] || die \
+ "Unable to describe $CERTIFICATE_MODE ACM certificate '$CERTIFICATE_ARN': $certificate_json"
+
+ certificate_status="$(jq -r '.Certificate.Status // empty' \
+ <<< "$certificate_json")" || die \
+ "Unable to read $CERTIFICATE_MODE ACM certificate metadata."
+ [ "$certificate_status" = "ISSUED" ] || die \
+ "The $CERTIFICATE_MODE ACM certificate '$CERTIFICATE_ARN' is '$certificate_status', not ISSUED."
+
+ while IFS= read -r certificate_name; do
+ [ -n "$certificate_name" ] || continue
+ if certificate_name_covers_hostname "$certificate_name"; then
+ hostname_covered=true
+ break
+ fi
+ done < <(jq -r \
+ '[.Certificate.DomainName, .Certificate.SubjectAlternativeNames[]?] | unique[] | select(. != null)' \
+ <<< "$certificate_json")
+ [ "$hostname_covered" = true ] || die \
+ "The $CERTIFICATE_MODE ACM certificate '$CERTIFICATE_ARN' does not cover '$HOSTNAME_VALUE'."
+
+ set +e
+ certificate_tags="$(aws_acm list-tags-for-certificate \
+ --certificate-arn "$CERTIFICATE_ARN" \
+ --output json 2>&1)"
+ tags_status=$?
+ set -e
+ [ "$tags_status" -eq 0 ] || die \
+ "Unable to inspect ownership tags on $CERTIFICATE_MODE ACM certificate '$CERTIFICATE_ARN': $certificate_tags"
+ if jq -e \
+ --arg key "$CERTIFICATE_MANAGED_BY_TAG_KEY" \
+ --arg value "$CERTIFICATE_MANAGED_BY_TAG_VALUE" \
+ '.Tags[]? | select(.Key == $key and .Value == $value)' \
+ <<< "$certificate_tags" >/dev/null; then
+ die "Certificate '$CERTIFICATE_ARN' is marked as managed by or-eks-stack and cannot be used in $CERTIFICATE_MODE mode."
+ fi
+
+ if [ "$CERTIFICATE_MODE" = "shared" ]; then
+ log "Validated externally managed shared ACM certificate '$CERTIFICATE_ARN' for '$HOSTNAME_VALUE'."
+ else
+ log "Validated user-owned ACM certificate '$CERTIFICATE_ARN' for '$HOSTNAME_VALUE'."
+ fi
+}
+
+managed_certificate_tags_match_for_endpoint() {
+ local tags_json="$1"
+ local hostname="$2"
+ local endpoint="$3"
+
+ jq -e \
+ --arg managed_by_key "$CERTIFICATE_MANAGED_BY_TAG_KEY" \
+ --arg managed_by_value "$CERTIFICATE_MANAGED_BY_TAG_VALUE" \
+ --arg cluster_key "$CERTIFICATE_CLUSTER_TAG_KEY" \
+ --arg cluster_value "$CLUSTER_NAME" \
+ --arg stack_key "$CERTIFICATE_STACK_TAG_KEY" \
+ --arg stack_value "$STACK_NAME" \
+ --arg hostname_key "$CERTIFICATE_HOSTNAME_TAG_KEY" \
+ --arg hostname_value "$hostname" \
+ --arg endpoint_key "$CERTIFICATE_ENDPOINT_TAG_KEY" \
+ --arg endpoint_value "$endpoint" \
+ 'def values_for($key): [.Tags[]? | select(.Key == $key) | .Value];
+ values_for($managed_by_key) == [$managed_by_value] and
+ values_for($cluster_key) == [$cluster_value] and
+ values_for($stack_key) == [$stack_value] and
+ values_for($hostname_key) == [$hostname_value] and
+ (values_for($endpoint_key) == [$endpoint_value] or
+ ($endpoint_value == "web" and values_for($endpoint_key) == []))' \
+ <<< "$tags_json" >/dev/null
+}
+
+managed_certificate_tags_match_stack_for_endpoint() {
+ local tags_json="$1"
+ local endpoint="$2"
+
+ jq -e \
+ --arg managed_by_key "$CERTIFICATE_MANAGED_BY_TAG_KEY" \
+ --arg managed_by_value "$CERTIFICATE_MANAGED_BY_TAG_VALUE" \
+ --arg cluster_key "$CERTIFICATE_CLUSTER_TAG_KEY" \
+ --arg cluster_value "$CLUSTER_NAME" \
+ --arg stack_key "$CERTIFICATE_STACK_TAG_KEY" \
+ --arg stack_value "$STACK_NAME" \
+ --arg endpoint_key "$CERTIFICATE_ENDPOINT_TAG_KEY" \
+ --arg endpoint_value "$endpoint" \
+ 'def values_for($key): [.Tags[]? | select(.Key == $key) | .Value];
+ values_for($managed_by_key) == [$managed_by_value] and
+ values_for($cluster_key) == [$cluster_value] and
+ values_for($stack_key) == [$stack_value] and
+ (values_for($endpoint_key) == [$endpoint_value] or
+ ($endpoint_value == "web" and values_for($endpoint_key) == []))' \
+ <<< "$tags_json" >/dev/null
+}
+
+managed_certificate_tags_match() {
+ managed_certificate_tags_match_for_endpoint "$1" "$HOSTNAME_VALUE" web
+}
+
+managed_certificate_tags_match_stack() {
+ managed_certificate_tags_match_stack_for_endpoint "$1" web
+}
+
+managed_mqtts_certificate_tags_match() {
+ managed_certificate_tags_match_for_endpoint "$1" "$MQTTS_HOSTNAME_VALUE" mqtts
+}
+
+managed_mqtts_certificate_tags_match_stack() {
+ managed_certificate_tags_match_stack_for_endpoint "$1" mqtts
+}
+
+require_managed_certificate_ownership() {
+ local certificate_account
+ local certificate_region
+ local certificate_tags
+ local current_account
+
+ if [[ ! "$CERTIFICATE_ARN" =~ ^arn:[^:]+:acm:([^:]+):([0-9]{12}):certificate/[A-Za-z0-9-]+$ ]]; then
+ die "Managed certificate ARN '$CERTIFICATE_ARN' is invalid."
+ fi
+ certificate_region="${BASH_REMATCH[1]}"
+ certificate_account="${BASH_REMATCH[2]}"
+ [ "$certificate_region" = "$AWS_REGION_NAME" ] || die \
+ "Managed certificate '$CERTIFICATE_ARN' is in region '$certificate_region', not '$AWS_REGION_NAME'."
+
+ current_account="$(aws_sts get-caller-identity --query Account --output text)" || die \
+ "Unable to determine the EKS AWS account."
+ [ "$certificate_account" = "$current_account" ] || die \
+ "Managed certificate '$CERTIFICATE_ARN' belongs to account '$certificate_account', not '$current_account'."
+
+ certificate_tags="$(aws_acm list-tags-for-certificate \
+ --certificate-arn "$CERTIFICATE_ARN" \
+ --output json)" || die \
+ "Unable to inspect ownership tags on managed certificate '$CERTIFICATE_ARN'."
+ managed_certificate_tags_match "$certificate_tags" || die \
+ "Certificate '$CERTIFICATE_ARN' does not carry the exact managed ownership tags for stack '$STACK_NAME' and hostname '$HOSTNAME_VALUE'."
+}
+
+discover_managed_certificate() {
+ local candidate_arn
+ local candidate_arns=""
+ local candidate_count=0
+ local certificate_domain
+ local certificate_list
+ local certificate_tagged_hostname
+ local certificate_tags
+
+ certificate_list="$(aws_acm list-certificates \
+ --certificate-statuses \
+ PENDING_VALIDATION ISSUED INACTIVE EXPIRED VALIDATION_TIMED_OUT REVOKED FAILED \
+ --output json)" || die "Unable to list ACM certificates for managed-certificate discovery."
+
+ while IFS=$'\t' read -r candidate_arn certificate_domain; do
+ [ -n "$candidate_arn" ] || continue
+ certificate_tags="$(aws_acm list-tags-for-certificate \
+ --certificate-arn "$candidate_arn" \
+ --output json)" || die \
+ "Unable to inspect ownership tags on ACM certificate '$candidate_arn'."
+ managed_certificate_tags_match_stack "$certificate_tags" || continue
+
+ certificate_tagged_hostname="$(jq -r \
+ --arg key "$CERTIFICATE_HOSTNAME_TAG_KEY" \
+ '.Tags[]? | select(.Key == $key) | .Value' \
+ <<< "$certificate_tags")"
+ if ! managed_certificate_tags_match "$certificate_tags" || \
+ [ "${certificate_domain%.}" != "$HOSTNAME_VALUE" ]; then
+ die "Managed ACM certificate '$candidate_arn' is already owned by stack '$STACK_NAME' for hostname '${certificate_tagged_hostname:-${certificate_domain%.}}'; refusing to create another certificate for '$HOSTNAME_VALUE'."
+ fi
+
+ candidate_count="$((candidate_count + 1))"
+ candidate_arns+="${candidate_arns:+$'\n'}$candidate_arn"
+ done < <(jq -r \
+ '.CertificateSummaryList[]? | [.CertificateArn, (.DomainName | ascii_downcase)] | @tsv' \
+ <<< "$certificate_list")
+
+ [ "$candidate_count" -le 1 ] || die \
+ "Found $candidate_count managed ACM certificates for stack '$STACK_NAME' and hostname '$HOSTNAME_VALUE'; refusing to choose one."
+ if [ "$candidate_count" -eq 1 ]; then
+ CERTIFICATE_ARN="$candidate_arns"
+ log "Reusing tagged managed ACM certificate '$CERTIFICATE_ARN'."
+ fi
+}
+
+request_managed_certificate() {
+ local idempotency_token
+
+ idempotency_token="$(printf '%s' \
+ "$CLUSTER_NAME|$AWS_REGION_NAME|$STACK_NAME|$HOSTNAME_VALUE" | \
+ cksum | awk '{print "or" $1}')"
+
+ log "Requesting a managed ACM certificate for '$HOSTNAME_VALUE'."
+ CERTIFICATE_ARN="$(aws_acm request-certificate \
+ --domain-name "$HOSTNAME_VALUE" \
+ --validation-method DNS \
+ --idempotency-token "$idempotency_token" \
+ --tags \
+ "Key=$CERTIFICATE_MANAGED_BY_TAG_KEY,Value=$CERTIFICATE_MANAGED_BY_TAG_VALUE" \
+ "Key=$CERTIFICATE_CLUSTER_TAG_KEY,Value=$CLUSTER_NAME" \
+ "Key=$CERTIFICATE_STACK_TAG_KEY,Value=$STACK_NAME" \
+ "Key=$CERTIFICATE_HOSTNAME_TAG_KEY,Value=$HOSTNAME_VALUE" \
+ "Key=$CERTIFICATE_ENDPOINT_TAG_KEY,Value=web" \
+ --query CertificateArn \
+ --output text)" || die \
+ "Unable to request a managed ACM certificate for '$HOSTNAME_VALUE'."
+ [ -n "$CERTIFICATE_ARN" ] && [ "$CERTIFICATE_ARN" != "None" ] || die \
+ "ACM did not return a certificate ARN for '$HOSTNAME_VALUE'."
+ log "Requested managed ACM certificate '$CERTIFICATE_ARN'."
+}
+
+stack_certificate_mode() {
+ kubectl --context "$KUBE_CONTEXT" get namespace "$STACK_NAME" \
+ -o 'go-template={{with index .metadata.annotations "openremote.io/certificate-mode"}}{{.}}{{end}}'
+}
+
+stack_certificate_arn() {
+ kubectl --context "$KUBE_CONTEXT" get namespace "$STACK_NAME" \
+ -o 'go-template={{with index .metadata.annotations "openremote.io/certificate-arn"}}{{.}}{{end}}'
+}
+
+stack_hostname() {
+ kubectl --context "$KUBE_CONTEXT" get namespace "$STACK_NAME" \
+ -o 'go-template={{with index .metadata.annotations "openremote.io/hostname"}}{{.}}{{end}}'
+}
+
+stack_mqtts_certificate_mode() {
+ kubectl --context "$KUBE_CONTEXT" get namespace "$STACK_NAME" \
+ -o 'go-template={{with index .metadata.annotations "openremote.io/mqtts-certificate-mode"}}{{.}}{{end}}'
+}
+
+stack_mqtts_certificate_arn() {
+ kubectl --context "$KUBE_CONTEXT" get namespace "$STACK_NAME" \
+ -o 'go-template={{with index .metadata.annotations "openremote.io/mqtts-certificate-arn"}}{{.}}{{end}}'
+}
+
+stack_mqtts_hostname() {
+ kubectl --context "$KUBE_CONTEXT" get namespace "$STACK_NAME" \
+ -o 'go-template={{with index .metadata.annotations "openremote.io/mqtts-hostname"}}{{.}}{{end}}'
+}
+
+inspect_stack_namespace_for_destroy() {
+ local namespace_identity
+ local namespace_name
+ local stack_label
+
+ namespace_identity="$(kubectl --context "$KUBE_CONTEXT" \
+ get namespace "$STACK_NAME" \
+ --ignore-not-found \
+ -o 'go-template={{.metadata.name}}|{{with index .metadata.labels "openremote.io/stack"}}{{.}}{{end}}')" || die \
+ "Unable to inspect namespace '$STACK_NAME' before destroy."
+ if [ -z "$namespace_identity" ]; then
+ STACK_NAMESPACE_EXISTS=false
+ return
+ fi
+
+ IFS='|' read -r namespace_name stack_label <<< "$namespace_identity"
+ [ "$namespace_name" = "$STACK_NAME" ] || die \
+ "Unable to identify namespace '$STACK_NAME' before destroy."
+ [ "$stack_label" = "$STACK_NAME" ] || die \
+ "Namespace '$STACK_NAME' does not have the matching openremote.io/stack label; refusing to modify it."
+ STACK_NAMESPACE_EXISTS=true
+}
+
+prepare_portable_stack() {
+ local stack_args=(
+ prepare
+ --name "$STACK_NAME" \
+ --kube-context "$KUBE_CONTEXT" \
+ --target eks \
+ --exposure "$EXPOSURE" \
+ --dns external-dns \
+ --hostname "$HOSTNAME_VALUE"
+ )
+ if [ "$EXPOSURE" = ingress ] && [ "$MQTTS_ENABLED" = true ]; then
+ stack_args+=(--mqtts --mqtts-hostname "$MQTTS_HOSTNAME_VALUE")
+ fi
+ run_or_stack "${stack_args[@]}"
+}
+
+resolve_managed_certificate() {
+ local recorded_certificate_arn
+ local recorded_certificate_mode
+
+ recorded_certificate_mode="$(stack_certificate_mode)" || die "Unable to read the recorded certificate mode."
+ recorded_certificate_arn="$(stack_certificate_arn)" || die "Unable to read the recorded certificate ARN."
+ if [ "$recorded_certificate_mode" = "managed" ] && [ -n "$recorded_certificate_arn" ]; then
+ CERTIFICATE_ARN="$recorded_certificate_arn"
+ log "Reusing the managed ACM certificate recorded by stack '$STACK_NAME'."
+ else
+ discover_managed_certificate
+ if [ -z "$CERTIFICATE_ARN" ]; then
+ request_managed_certificate
+ fi
+ fi
+
+ require_managed_certificate_ownership
+ record_certificate_configuration
+}
+
+require_managed_mqtts_certificate_ownership() {
+ local certificate_account
+ local certificate_region
+ local certificate_tags
+ local current_account
+
+ if [[ ! "$MQTTS_CERTIFICATE_ARN" =~ ^arn:[^:]+:acm:([^:]+):([0-9]{12}):certificate/[A-Za-z0-9-]+$ ]]; then
+ die "Managed MQTTS certificate ARN '$MQTTS_CERTIFICATE_ARN' is invalid."
+ fi
+ certificate_region="${BASH_REMATCH[1]}"
+ certificate_account="${BASH_REMATCH[2]}"
+ [ "$certificate_region" = "$AWS_REGION_NAME" ] || die \
+ "Managed MQTTS certificate '$MQTTS_CERTIFICATE_ARN' is in region '$certificate_region', not '$AWS_REGION_NAME'."
+
+ current_account="$(aws_sts get-caller-identity --query Account --output text)" || die \
+ "Unable to determine the EKS AWS account."
+ [ "$certificate_account" = "$current_account" ] || die \
+ "Managed MQTTS certificate '$MQTTS_CERTIFICATE_ARN' belongs to account '$certificate_account', not '$current_account'."
+
+ certificate_tags="$(aws_acm list-tags-for-certificate \
+ --certificate-arn "$MQTTS_CERTIFICATE_ARN" \
+ --output json)" || die \
+ "Unable to inspect ownership tags on managed MQTTS certificate '$MQTTS_CERTIFICATE_ARN'."
+ managed_mqtts_certificate_tags_match "$certificate_tags" || die \
+ "Certificate '$MQTTS_CERTIFICATE_ARN' does not carry the exact managed MQTTS ownership tags for stack '$STACK_NAME' and hostname '$MQTTS_HOSTNAME_VALUE'."
+}
+
+discover_managed_mqtts_certificate() {
+ local candidate_arn
+ local candidate_arns=""
+ local candidate_count=0
+ local certificate_domain
+ local certificate_list
+ local certificate_tagged_hostname
+ local certificate_tags
+
+ certificate_list="$(aws_acm list-certificates \
+ --certificate-statuses \
+ PENDING_VALIDATION ISSUED INACTIVE EXPIRED VALIDATION_TIMED_OUT REVOKED FAILED \
+ --output json)" || die \
+ "Unable to list ACM certificates for managed MQTTS certificate discovery."
+
+ while IFS=$'\t' read -r candidate_arn certificate_domain; do
+ [ -n "$candidate_arn" ] || continue
+ certificate_tags="$(aws_acm list-tags-for-certificate \
+ --certificate-arn "$candidate_arn" \
+ --output json)" || die \
+ "Unable to inspect ownership tags on ACM certificate '$candidate_arn'."
+ managed_mqtts_certificate_tags_match_stack "$certificate_tags" || continue
+
+ certificate_tagged_hostname="$(jq -r \
+ --arg key "$CERTIFICATE_HOSTNAME_TAG_KEY" \
+ '.Tags[]? | select(.Key == $key) | .Value' \
+ <<< "$certificate_tags")"
+ if ! managed_mqtts_certificate_tags_match "$certificate_tags" || \
+ [ "${certificate_domain%.}" != "$MQTTS_HOSTNAME_VALUE" ]; then
+ die "Managed MQTTS ACM certificate '$candidate_arn' is already owned by stack '$STACK_NAME' for hostname '${certificate_tagged_hostname:-${certificate_domain%.}}'; refusing to create another certificate for '$MQTTS_HOSTNAME_VALUE'."
+ fi
+
+ candidate_count="$((candidate_count + 1))"
+ candidate_arns+="${candidate_arns:+$'\n'}$candidate_arn"
+ done < <(jq -r \
+ '.CertificateSummaryList[]? | [.CertificateArn, (.DomainName | ascii_downcase)] | @tsv' \
+ <<< "$certificate_list")
+
+ [ "$candidate_count" -le 1 ] || die \
+ "Found $candidate_count managed MQTTS ACM certificates for stack '$STACK_NAME' and hostname '$MQTTS_HOSTNAME_VALUE'; refusing to choose one."
+ if [ "$candidate_count" -eq 1 ]; then
+ MQTTS_CERTIFICATE_ARN="$candidate_arns"
+ log "Reusing tagged managed MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN'."
+ fi
+}
+
+request_managed_mqtts_certificate() {
+ local idempotency_token
+
+ idempotency_token="$(printf '%s' \
+ "$CLUSTER_NAME|$AWS_REGION_NAME|$STACK_NAME|mqtts|$MQTTS_HOSTNAME_VALUE" | \
+ cksum | awk '{print "or" $1}')"
+
+ log "Requesting a managed MQTTS ACM certificate for '$MQTTS_HOSTNAME_VALUE'."
+ MQTTS_CERTIFICATE_ARN="$(aws_acm request-certificate \
+ --domain-name "$MQTTS_HOSTNAME_VALUE" \
+ --validation-method DNS \
+ --idempotency-token "$idempotency_token" \
+ --tags \
+ "Key=$CERTIFICATE_MANAGED_BY_TAG_KEY,Value=$CERTIFICATE_MANAGED_BY_TAG_VALUE" \
+ "Key=$CERTIFICATE_CLUSTER_TAG_KEY,Value=$CLUSTER_NAME" \
+ "Key=$CERTIFICATE_STACK_TAG_KEY,Value=$STACK_NAME" \
+ "Key=$CERTIFICATE_HOSTNAME_TAG_KEY,Value=$MQTTS_HOSTNAME_VALUE" \
+ "Key=$CERTIFICATE_ENDPOINT_TAG_KEY,Value=mqtts" \
+ --query CertificateArn \
+ --output text)" || die \
+ "Unable to request a managed MQTTS ACM certificate for '$MQTTS_HOSTNAME_VALUE'."
+ [ -n "$MQTTS_CERTIFICATE_ARN" ] && [ "$MQTTS_CERTIFICATE_ARN" != "None" ] || die \
+ "ACM did not return a certificate ARN for '$MQTTS_HOSTNAME_VALUE'."
+ log "Requested managed MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN'."
+}
+
+resolve_managed_mqtts_certificate() {
+ local recorded_certificate_arn
+ local recorded_certificate_mode
+
+ recorded_certificate_mode="$(stack_mqtts_certificate_mode)" || die \
+ "Unable to read the recorded MQTTS certificate mode."
+ recorded_certificate_arn="$(stack_mqtts_certificate_arn)" || die \
+ "Unable to read the recorded MQTTS certificate ARN."
+ [ "$recorded_certificate_mode" = managed ] || die \
+ "Stack '$STACK_NAME' no longer records managed MQTTS certificate mode."
+ if [ -n "$recorded_certificate_arn" ]; then
+ MQTTS_CERTIFICATE_ARN="$recorded_certificate_arn"
+ log "Reusing the managed MQTTS ACM certificate recorded by stack '$STACK_NAME'."
+ else
+ discover_managed_mqtts_certificate
+ if [ -z "$MQTTS_CERTIFICATE_ARN" ]; then
+ request_managed_mqtts_certificate
+ fi
+ fi
+
+ require_managed_mqtts_certificate_ownership
+ record_mqtts_certificate_configuration
+}
+
+require_actionable_managed_mqtts_certificate_status() {
+ local certificate_status="$1"
+
+ case "$certificate_status" in
+ PENDING_VALIDATION|ISSUED) ;;
+ *)
+ die "Managed MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN' is '$certificate_status' and cannot be reconciled automatically."
+ ;;
+ esac
+}
+
+wait_for_managed_mqtts_certificate_validation_records() {
+ local certificate_json
+ local certificate_status="unknown"
+ local deadline
+ local describe_status
+ local last_error=""
+ local next_notice
+ local now
+ local timeout_seconds
+
+ timeout_seconds="$(duration_seconds "$TIMEOUT")"
+ now="$(date +%s)"
+ deadline="$((now + timeout_seconds))"
+ next_notice="$((now + 30))"
+ log "Waiting for ACM to publish MQTTS DNS validation records."
+
+ while [ "$now" -lt "$deadline" ]; do
+ set +e
+ certificate_json="$(aws_acm describe-certificate \
+ --certificate-arn "$MQTTS_CERTIFICATE_ARN" \
+ --output json 2>&1)"
+ describe_status=$?
+ set -e
+ if [ "$describe_status" -eq 0 ]; then
+ certificate_status="$(jq -r '.Certificate.Status // empty' \
+ <<< "$certificate_json")" || die \
+ "Unable to read managed MQTTS ACM certificate metadata."
+ require_actionable_managed_mqtts_certificate_status "$certificate_status"
+ jq -e \
+ --arg hostname "$MQTTS_HOSTNAME_VALUE" \
+ '(.Certificate.DomainName | ascii_downcase) == $hostname and
+ ([.Certificate.DomainValidationOptions[]?] | length) > 0 and
+ ([.Certificate.DomainValidationOptions[]?] |
+ all((.ResourceRecord.Name // "") != "" and
+ (.ResourceRecord.Type // "") != "" and
+ (.ResourceRecord.Value // "") != ""))' \
+ <<< "$certificate_json" >/dev/null && {
+ MQTTS_CERTIFICATE_JSON="$certificate_json"
+ return
+ }
+ last_error="ACM has not published every MQTTS validation record yet"
+ else
+ last_error="$certificate_json"
+ fi
+
+ if [ "$now" -ge "$next_notice" ]; then
+ log "ACM MQTTS validation records are still pending."
+ next_notice="$((now + 30))"
+ fi
+ sleep 5
+ now="$(date +%s)"
+ done
+
+ die "ACM did not publish DNS validation records for MQTTS certificate '$MQTTS_CERTIFICATE_ARN' within '$TIMEOUT': $last_error"
+}
+
+validate_managed_mqtts_certificate_records() {
+ local record_name
+ local record_target
+ local record_type
+
+ while IFS=$'\t' read -r record_name record_type record_target; do
+ [ -n "$record_name" ] && [ -n "$record_target" ] || die \
+ "ACM returned an incomplete DNS validation record for MQTTS certificate '$MQTTS_CERTIFICATE_ARN'."
+ [ "$record_type" = CNAME ] || die \
+ "ACM returned unsupported MQTTS validation record type '$record_type'."
+ validation_record_is_in_managed_domain "$record_name" || die \
+ "ACM MQTTS validation record '$record_name' is outside the configured ExternalDNS domains."
+ done < <(jq -r \
+ '.Certificate.DomainValidationOptions[]?.ResourceRecord |
+ [.Name, .Type, .Value] | @tsv' <<< "$MQTTS_CERTIFICATE_JSON")
+}
+
+require_owned_mqtts_validation_dns_endpoint() {
+ local existing_certificate_arn
+ local existing_endpoint_kind
+ local existing_endpoint
+ local existing_managed_by
+
+ # Only NotFound permits creation; a failed read cannot establish ownership.
+ existing_endpoint="$(kubectl --context "$KUBE_CONTEXT" \
+ --namespace "$STACK_NAME" \
+ get dnsendpoint "$ACM_MQTTS_VALIDATION_DNS_ENDPOINT_NAME" \
+ --ignore-not-found -o json)" || die \
+ "Unable to inspect DNSEndpoint '$STACK_NAME/$ACM_MQTTS_VALIDATION_DNS_ENDPOINT_NAME'; refusing to write ACM MQTTS validation records without ownership checks."
+ [ -n "$existing_endpoint" ] || return 0
+
+ existing_managed_by="$(jq -r \
+ '.metadata.labels["app.kubernetes.io/managed-by"] // empty' \
+ <<< "$existing_endpoint")"
+ existing_certificate_arn="$(jq -r \
+ '.metadata.annotations["openremote.io/certificate-arn"] // empty' \
+ <<< "$existing_endpoint")"
+ existing_endpoint_kind="$(jq -r \
+ '.metadata.annotations["openremote.io/endpoint"] // empty' \
+ <<< "$existing_endpoint")"
+ [ "$existing_managed_by" = "$CERTIFICATE_MANAGED_BY_TAG_VALUE" ] && \
+ [ "$existing_certificate_arn" = "$MQTTS_CERTIFICATE_ARN" ] && \
+ [ "$existing_endpoint_kind" = mqtts ] || die \
+ "DNSEndpoint '$STACK_NAME/$ACM_MQTTS_VALIDATION_DNS_ENDPOINT_NAME' is not owned by managed MQTTS certificate '$MQTTS_CERTIFICATE_ARN'; refusing to replace it."
+}
+
+apply_managed_mqtts_certificate_validation_records() {
+ local endpoint_manifest
+
+ validate_managed_mqtts_certificate_records
+ require_owned_mqtts_validation_dns_endpoint
+ endpoint_manifest="$(jq -c \
+ --arg name "$ACM_MQTTS_VALIDATION_DNS_ENDPOINT_NAME" \
+ --arg namespace "$STACK_NAME" \
+ --arg certificate_arn "$MQTTS_CERTIFICATE_ARN" \
+ --arg managed_by "$CERTIFICATE_MANAGED_BY_TAG_VALUE" \
+ '{
+ apiVersion: "externaldns.k8s.io/v1alpha1",
+ kind: "DNSEndpoint",
+ metadata: {
+ name: $name,
+ namespace: $namespace,
+ labels: {
+ "app.kubernetes.io/managed-by": $managed_by,
+ "openremote.io/managed-dns": "true"
+ },
+ annotations: {
+ "openremote.io/certificate-arn": $certificate_arn,
+ "openremote.io/endpoint": "mqtts"
+ }
+ },
+ spec: {
+ endpoints: ([.Certificate.DomainValidationOptions[]?.ResourceRecord | {
+ dnsName: .Name,
+ recordType: .Type,
+ recordTTL: 300,
+ targets: [.Value]
+ }] | unique_by(.dnsName, .recordType, .targets[0]))
+ }
+ }' <<< "$MQTTS_CERTIFICATE_JSON")" || die \
+ "Unable to render ACM MQTTS validation records as a DNSEndpoint."
+
+ log "Reconciling ACM MQTTS DNS validation records in '$STACK_NAME/$ACM_MQTTS_VALIDATION_DNS_ENDPOINT_NAME'."
+ kubectl --context "$KUBE_CONTEXT" \
+ --namespace "$STACK_NAME" \
+ apply -f - <<< "$endpoint_manifest" >/dev/null || die \
+ "Unable to apply ACM MQTTS validation DNSEndpoint '$STACK_NAME/$ACM_MQTTS_VALIDATION_DNS_ENDPOINT_NAME'."
+}
+
+wait_for_managed_mqtts_certificate() {
+ local certificate_status="unknown"
+ local deadline
+ local next_notice
+ local now
+ local timeout_seconds
+
+ timeout_seconds="$(duration_seconds "$TIMEOUT")"
+ now="$(date +%s)"
+ deadline="$((now + timeout_seconds))"
+ next_notice="$((now + 30))"
+ log "Waiting for managed MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN' to be issued."
+
+ while [ "$now" -lt "$deadline" ]; do
+ certificate_status="$(aws_acm describe-certificate \
+ --certificate-arn "$MQTTS_CERTIFICATE_ARN" \
+ --query 'Certificate.Status' \
+ --output text 2>/dev/null || true)"
+ if [ "$certificate_status" = ISSUED ]; then
+ log "Managed MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN' is issued."
+ return
+ fi
+ if [ -n "$certificate_status" ] && [ "$certificate_status" != None ]; then
+ require_actionable_managed_mqtts_certificate_status "$certificate_status"
+ fi
+ if [ "$now" -ge "$next_notice" ]; then
+ log "Managed MQTTS ACM certificate validation is still pending."
+ next_notice="$((now + 30))"
+ fi
+ sleep 5
+ now="$(date +%s)"
+ done
+
+ die "Managed MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN' remained '$certificate_status' beyond '$TIMEOUT'. The validation DNSEndpoint is retained for a later retry."
+}
+
+discover_managed_certificate_for_destroy() {
+ local certificate_arn
+ local certificate_count=0
+ local certificate_list
+ local certificate_tags
+ local discovered_certificate_arn=""
+
+ certificate_list="$(aws_acm list-certificates \
+ --certificate-statuses \
+ PENDING_VALIDATION ISSUED INACTIVE EXPIRED VALIDATION_TIMED_OUT REVOKED FAILED \
+ --output json)" || die \
+ "Unable to check for stack-managed ACM certificates before destroy."
+
+ while IFS= read -r certificate_arn; do
+ [ -n "$certificate_arn" ] || continue
+ certificate_tags="$(aws_acm list-tags-for-certificate \
+ --certificate-arn "$certificate_arn" \
+ --output json)" || die \
+ "Unable to inspect ownership tags on ACM certificate '$certificate_arn'."
+ if managed_certificate_tags_match_stack "$certificate_tags"; then
+ certificate_count="$((certificate_count + 1))"
+ discovered_certificate_arn="$certificate_arn"
+ fi
+ done < <(jq -r '.CertificateSummaryList[]?.CertificateArn' \
+ <<< "$certificate_list")
+
+ [ "$certificate_count" -le 1 ] || die \
+ "Found $certificate_count managed ACM certificates for stack '$STACK_NAME'; refusing destructive cleanup."
+ if [ "$certificate_count" -eq 1 ]; then
+ printf '%s\n' "$discovered_certificate_arn"
+ fi
+}
+
+load_managed_certificate_for_destroy() {
+ local certificate_domain
+ local certificate_json
+ local certificate_tags
+ local describe_status
+ local tagged_hostname
+
+ set +e
+ certificate_json="$(aws_acm describe-certificate \
+ --certificate-arn "$CERTIFICATE_ARN" \
+ --output json 2>&1)"
+ describe_status=$?
+ set -e
+ if [ "$describe_status" -ne 0 ]; then
+ case "$certificate_json" in
+ *ResourceNotFoundException*)
+ CERTIFICATE_EXISTS=false
+ CERTIFICATE_JSON=""
+ return
+ ;;
+ *)
+ die "Unable to describe managed ACM certificate '$CERTIFICATE_ARN' before destroy: $certificate_json"
+ ;;
+ esac
+ fi
+
+ certificate_tags="$(aws_acm list-tags-for-certificate \
+ --certificate-arn "$CERTIFICATE_ARN" \
+ --output json)" || die \
+ "Unable to inspect ownership tags on managed certificate '$CERTIFICATE_ARN'."
+ managed_certificate_tags_match_stack "$certificate_tags" || die \
+ "Certificate '$CERTIFICATE_ARN' does not carry the exact managed ownership tags for stack '$STACK_NAME'."
+ tagged_hostname="$(jq -r \
+ --arg key "$CERTIFICATE_HOSTNAME_TAG_KEY" \
+ '[.Tags[]? | select(.Key == $key) | .Value] |
+ if length == 1 then .[0] else empty end' \
+ <<< "$certificate_tags")" || die \
+ "Unable to read the managed hostname tag from certificate '$CERTIFICATE_ARN'."
+ [ -n "$tagged_hostname" ] || die \
+ "Certificate '$CERTIFICATE_ARN' does not have exactly one managed hostname tag."
+ if [ -n "$HOSTNAME_VALUE" ] && [ "$HOSTNAME_VALUE" != "$tagged_hostname" ]; then
+ die "Namespace '$STACK_NAME' records hostname '$HOSTNAME_VALUE', but its managed certificate records '$tagged_hostname'."
+ fi
+ HOSTNAME_VALUE="$tagged_hostname"
+ validate_hostname
+
+ certificate_domain="$(jq -r '.Certificate.DomainName // empty | ascii_downcase' \
+ <<< "$certificate_json")" || die \
+ "Unable to read the domain of managed certificate '$CERTIFICATE_ARN'."
+ [ "${certificate_domain%.}" = "$HOSTNAME_VALUE" ] || die \
+ "Managed certificate '$CERTIFICATE_ARN' is for '$certificate_domain', not '$HOSTNAME_VALUE'."
+
+ require_managed_certificate_ownership
+ CERTIFICATE_EXISTS=true
+ CERTIFICATE_JSON="$certificate_json"
+}
+
+capture_stack_ingress_for_destroy() {
+ STACK_INGRESS_JSON=""
+ [ "$STACK_NAMESPACE_EXISTS" = true ] || return 0
+
+ STACK_INGRESS_JSON="$(kubectl --context "$KUBE_CONTEXT" \
+ --namespace "$STACK_NAME" \
+ get ingress proxy \
+ --ignore-not-found \
+ -o json)" || die \
+ "Unable to inspect Ingress '$STACK_NAME/proxy' before destroy."
+}
+
+resolve_managed_certificate_for_destroy() {
+ local discovered_certificate_arn=""
+ local validation_endpoint
+ local recorded_certificate_arn=""
+ local recorded_certificate_mode=""
+ local recorded_hostname=""
+
+ MANAGED_CERTIFICATE_DESTROY=false
+
+ if [ "$STACK_NAMESPACE_EXISTS" = true ]; then
+ recorded_certificate_mode="$(stack_certificate_mode)" || die \
+ "Unable to read certificate mode from namespace '$STACK_NAME'."
+ recorded_certificate_arn="$(stack_certificate_arn)" || die \
+ "Unable to read certificate ARN from namespace '$STACK_NAME'."
+ recorded_hostname="$(stack_hostname)" || die \
+ "Unable to read hostname from namespace '$STACK_NAME'."
+ [ -z "$recorded_hostname" ] || [ "$recorded_hostname" = "$HOSTNAME_VALUE" ] || die \
+ "Namespace '$STACK_NAME' records hostname '$recorded_hostname', but reserves '$HOSTNAME_VALUE'; refusing destructive cleanup."
+ fi
+
+ if ! discovered_certificate_arn="$(discover_managed_certificate_for_destroy)"; then
+ die "Unable to safely discover a unique managed ACM certificate for stack '$STACK_NAME'."
+ fi
+ if [ "$recorded_certificate_mode" = "managed" ]; then
+ if [ -n "$recorded_certificate_arn" ] && \
+ [ -n "$discovered_certificate_arn" ] && \
+ [ "$recorded_certificate_arn" != "$discovered_certificate_arn" ]; then
+ die "Namespace '$STACK_NAME' records managed certificate '$recorded_certificate_arn', but ownership tags identify '$discovered_certificate_arn'."
+ fi
+ CERTIFICATE_ARN="${recorded_certificate_arn:-$discovered_certificate_arn}"
+ if [ -z "$CERTIFICATE_ARN" ]; then
+ # First apply may have stopped after recording mode but before requesting
+ # ACM. With no tagged certificate or validation resource, ordinary stack
+ # cleanup is sufficient; do not strand the namespace and hostname claim.
+ validation_endpoint="$(kubectl --context "$KUBE_CONTEXT" --namespace "$STACK_NAME" \
+ get dnsendpoint "$ACM_VALIDATION_DNS_ENDPOINT_NAME" --ignore-not-found -o json)" || die \
+ "Unable to inspect validation resources for the incomplete managed certificate configuration."
+ [ -z "$validation_endpoint" ] || die \
+ "Stack '$STACK_NAME' has validation resources without an identifiable managed certificate; refusing destructive cleanup."
+ return 0
+ fi
+ elif [ -n "$discovered_certificate_arn" ]; then
+ CERTIFICATE_ARN="$discovered_certificate_arn"
+ log "Recovered tagged managed ACM certificate '$CERTIFICATE_ARN' for stack '$STACK_NAME'."
+ else
+ return 0
+ fi
+
+ load_managed_certificate_for_destroy
+ validate_hostname
+ MANAGED_CERTIFICATE_DESTROY=true
+}
+
+discover_managed_mqtts_certificate_for_destroy() {
+ local certificate_arn
+ local certificate_count=0
+ local certificate_list
+ local certificate_tags
+ local discovered_certificate_arn=""
+
+ certificate_list="$(aws_acm list-certificates \
+ --certificate-statuses \
+ PENDING_VALIDATION ISSUED INACTIVE EXPIRED VALIDATION_TIMED_OUT REVOKED FAILED \
+ --output json)" || die \
+ "Unable to check for stack-managed MQTTS ACM certificates before destroy."
+
+ while IFS= read -r certificate_arn; do
+ [ -n "$certificate_arn" ] || continue
+ certificate_tags="$(aws_acm list-tags-for-certificate \
+ --certificate-arn "$certificate_arn" \
+ --output json)" || die \
+ "Unable to inspect ownership tags on ACM certificate '$certificate_arn'."
+ if managed_mqtts_certificate_tags_match_stack "$certificate_tags"; then
+ certificate_count="$((certificate_count + 1))"
+ discovered_certificate_arn="$certificate_arn"
+ fi
+ done < <(jq -r '.CertificateSummaryList[]?.CertificateArn' \
+ <<< "$certificate_list")
+
+ [ "$certificate_count" -le 1 ] || die \
+ "Found $certificate_count managed MQTTS ACM certificates for stack '$STACK_NAME'; refusing destructive cleanup."
+ if [ "$certificate_count" -eq 1 ]; then
+ printf '%s\n' "$discovered_certificate_arn"
+ fi
+}
+
+load_managed_mqtts_certificate_for_destroy() {
+ local certificate_domain
+ local certificate_json
+ local certificate_tags
+ local describe_status
+ local tagged_hostname
+
+ set +e
+ certificate_json="$(aws_acm describe-certificate \
+ --certificate-arn "$MQTTS_CERTIFICATE_ARN" \
+ --output json 2>&1)"
+ describe_status=$?
+ set -e
+ if [ "$describe_status" -ne 0 ]; then
+ case "$certificate_json" in
+ *ResourceNotFoundException*)
+ MQTTS_CERTIFICATE_EXISTS=false
+ MQTTS_CERTIFICATE_JSON=""
+ return
+ ;;
+ *)
+ die "Unable to describe managed MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN' before destroy: $certificate_json"
+ ;;
+ esac
+ fi
+
+ certificate_tags="$(aws_acm list-tags-for-certificate \
+ --certificate-arn "$MQTTS_CERTIFICATE_ARN" \
+ --output json)" || die \
+ "Unable to inspect ownership tags on managed MQTTS certificate '$MQTTS_CERTIFICATE_ARN'."
+ managed_mqtts_certificate_tags_match_stack "$certificate_tags" || die \
+ "Certificate '$MQTTS_CERTIFICATE_ARN' does not carry the exact managed MQTTS ownership tags for stack '$STACK_NAME'."
+ tagged_hostname="$(jq -r \
+ --arg key "$CERTIFICATE_HOSTNAME_TAG_KEY" \
+ '[.Tags[]? | select(.Key == $key) | .Value] |
+ if length == 1 then .[0] else empty end' \
+ <<< "$certificate_tags")" || die \
+ "Unable to read the managed hostname tag from MQTTS certificate '$MQTTS_CERTIFICATE_ARN'."
+ [ -n "$tagged_hostname" ] || die \
+ "MQTTS certificate '$MQTTS_CERTIFICATE_ARN' does not have exactly one managed hostname tag."
+ if [ -n "$MQTTS_HOSTNAME_VALUE" ] && \
+ [ "$MQTTS_HOSTNAME_VALUE" != "$tagged_hostname" ]; then
+ die "Namespace '$STACK_NAME' records MQTTS hostname '$MQTTS_HOSTNAME_VALUE', but its managed MQTTS certificate records '$tagged_hostname'."
+ fi
+ MQTTS_HOSTNAME_VALUE="$tagged_hostname"
+ normalize_and_validate_mqtts_hostname
+
+ certificate_domain="$(jq -r '.Certificate.DomainName // empty | ascii_downcase' \
+ <<< "$certificate_json")" || die \
+ "Unable to read the domain of managed MQTTS certificate '$MQTTS_CERTIFICATE_ARN'."
+ [ "${certificate_domain%.}" = "$MQTTS_HOSTNAME_VALUE" ] || die \
+ "Managed MQTTS certificate '$MQTTS_CERTIFICATE_ARN' is for '$certificate_domain', not '$MQTTS_HOSTNAME_VALUE'."
+
+ require_managed_mqtts_certificate_ownership
+ MQTTS_CERTIFICATE_EXISTS=true
+ MQTTS_CERTIFICATE_JSON="$certificate_json"
+}
+
+resolve_managed_mqtts_certificate_for_destroy() {
+ local discovered_certificate_arn=""
+ local recorded_certificate_arn="$MQTTS_CERTIFICATE_ARN"
+ local recorded_certificate_mode="$MQTTS_CERTIFICATE_MODE"
+ local validation_endpoint
+
+ MANAGED_MQTTS_CERTIFICATE_DESTROY=false
+ if [ "$STACK_NAMESPACE_EXISTS" = true ]; then
+ recorded_certificate_mode="$(stack_mqtts_certificate_mode)" || die \
+ "Unable to read MQTTS certificate mode from namespace '$STACK_NAME'."
+ recorded_certificate_arn="$(stack_mqtts_certificate_arn)" || die \
+ "Unable to read MQTTS certificate ARN from namespace '$STACK_NAME'."
+ fi
+
+ if ! discovered_certificate_arn="$(discover_managed_mqtts_certificate_for_destroy)"; then
+ die "Unable to safely discover a unique managed MQTTS ACM certificate for stack '$STACK_NAME'."
+ fi
+ if [ "$recorded_certificate_mode" = managed ]; then
+ if [ -n "$recorded_certificate_arn" ] && \
+ [ -n "$discovered_certificate_arn" ] && \
+ [ "$recorded_certificate_arn" != "$discovered_certificate_arn" ]; then
+ die "Namespace '$STACK_NAME' records managed MQTTS certificate '$recorded_certificate_arn', but ownership tags identify '$discovered_certificate_arn'."
+ fi
+ MQTTS_CERTIFICATE_ARN="${recorded_certificate_arn:-$discovered_certificate_arn}"
+ if [ -z "$MQTTS_CERTIFICATE_ARN" ]; then
+ validation_endpoint="$(kubectl --context "$KUBE_CONTEXT" --namespace "$STACK_NAME" \
+ get dnsendpoint "$ACM_MQTTS_VALIDATION_DNS_ENDPOINT_NAME" \
+ --ignore-not-found -o json)" || die \
+ "Unable to inspect validation resources for the incomplete managed MQTTS certificate configuration."
+ [ -z "$validation_endpoint" ] || die \
+ "Stack '$STACK_NAME' has MQTTS validation resources without an identifiable managed certificate; refusing destructive cleanup."
+ return 0
+ fi
+ elif [ -n "$discovered_certificate_arn" ]; then
+ [ -z "$recorded_certificate_mode" ] || die \
+ "Stack '$STACK_NAME' records MQTTS certificate mode '$recorded_certificate_mode', but also owns managed MQTTS certificate '$discovered_certificate_arn'."
+ MQTTS_CERTIFICATE_ARN="$discovered_certificate_arn"
+ log "Recovered tagged managed MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN' for stack '$STACK_NAME'."
+ else
+ return 0
+ fi
+
+ load_managed_mqtts_certificate_for_destroy
+ MQTTS_CERTIFICATE_MODE=managed
+ MANAGED_MQTTS_CERTIFICATE_DESTROY=true
+}
+
+capture_mqtts_validation_dns_records_for_destroy() {
+ local certificate_records_json='[]'
+ local endpoint_certificate_arn
+ local endpoint_kind
+ local endpoint_managed_by
+ local endpoint_managed_dns
+ local endpoint_records_json='[]'
+ local validation_endpoint=""
+
+ MQTTS_VALIDATION_DNS_ENDPOINT_EXISTS=false
+ MQTTS_VALIDATION_DNS_RECORDS=""
+ if [ "$MQTTS_CERTIFICATE_EXISTS" = true ]; then
+ certificate_records_json="$(jq -c \
+ '[.Certificate.DomainValidationOptions[]?.ResourceRecord |
+ select((.Name // "") != "" and (.Type // "") != "" and
+ (.Value // "") != "") | {
+ dnsName: .Name,
+ recordType: .Type,
+ targets: [.Value]
+ }] | unique_by(.dnsName, .recordType, .targets[0]) |
+ sort_by(.dnsName, .recordType, .targets[0])' \
+ <<< "$MQTTS_CERTIFICATE_JSON")" || die \
+ "Unable to read validation records from managed MQTTS certificate '$MQTTS_CERTIFICATE_ARN'."
+ fi
+
+ if [ "$STACK_NAMESPACE_EXISTS" = true ]; then
+ validation_endpoint="$(kubectl --context "$KUBE_CONTEXT" \
+ --namespace "$STACK_NAME" \
+ get dnsendpoint "$ACM_MQTTS_VALIDATION_DNS_ENDPOINT_NAME" \
+ --ignore-not-found -o json)" || die \
+ "Unable to inspect DNSEndpoint '$STACK_NAME/$ACM_MQTTS_VALIDATION_DNS_ENDPOINT_NAME' before destroy."
+ fi
+
+ if [ -n "$validation_endpoint" ]; then
+ MQTTS_VALIDATION_DNS_ENDPOINT_EXISTS=true
+ endpoint_managed_by="$(jq -r \
+ '.metadata.labels["app.kubernetes.io/managed-by"] // empty' \
+ <<< "$validation_endpoint")"
+ endpoint_managed_dns="$(jq -r \
+ '.metadata.labels["openremote.io/managed-dns"] // empty' \
+ <<< "$validation_endpoint")"
+ endpoint_certificate_arn="$(jq -r \
+ '.metadata.annotations["openremote.io/certificate-arn"] // empty' \
+ <<< "$validation_endpoint")"
+ endpoint_kind="$(jq -r \
+ '.metadata.annotations["openremote.io/endpoint"] // empty' \
+ <<< "$validation_endpoint")"
+ [ "$endpoint_managed_by" = "$CERTIFICATE_MANAGED_BY_TAG_VALUE" ] && \
+ [ "$endpoint_managed_dns" = true ] && \
+ [ "$endpoint_certificate_arn" = "$MQTTS_CERTIFICATE_ARN" ] && \
+ [ "$endpoint_kind" = mqtts ] || die \
+ "DNSEndpoint '$STACK_NAME/$ACM_MQTTS_VALIDATION_DNS_ENDPOINT_NAME' does not exactly match managed MQTTS certificate '$MQTTS_CERTIFICATE_ARN'; refusing to delete it."
+
+ endpoint_records_json="$(jq -c \
+ '[.spec.endpoints[]? | {
+ dnsName: (.dnsName // ""),
+ recordType: (.recordType // ""),
+ targets: ((.targets // []) | sort)
+ }] | unique_by(.dnsName, .recordType, .targets[0]) |
+ sort_by(.dnsName, .recordType, .targets[0])' \
+ <<< "$validation_endpoint")" || die \
+ "Unable to read records from DNSEndpoint '$STACK_NAME/$ACM_MQTTS_VALIDATION_DNS_ENDPOINT_NAME'."
+ [ "$(jq 'length' <<< "$endpoint_records_json")" -gt 0 ] || die \
+ "DNSEndpoint '$STACK_NAME/$ACM_MQTTS_VALIDATION_DNS_ENDPOINT_NAME' has no validation records."
+ if [ "$MQTTS_CERTIFICATE_EXISTS" = true ] && \
+ [ "$endpoint_records_json" != "$certificate_records_json" ]; then
+ die "DNSEndpoint '$STACK_NAME/$ACM_MQTTS_VALIDATION_DNS_ENDPOINT_NAME' does not contain exactly the records returned by ACM; refusing destructive cleanup."
+ fi
+ elif [ "$MQTTS_CERTIFICATE_EXISTS" = false ]; then
+ die "Managed MQTTS certificate '$MQTTS_CERTIFICATE_ARN' and its validation DNSEndpoint are both absent; validation-record ownership cannot be verified."
+ fi
+
+ if [ "$MQTTS_VALIDATION_DNS_ENDPOINT_EXISTS" = true ]; then
+ MQTTS_VALIDATION_DNS_RECORDS="$(jq -r \
+ '.[] | [.dnsName, .recordType] | @tsv' \
+ <<< "$endpoint_records_json")"
+ else
+ MQTTS_VALIDATION_DNS_RECORDS="$(jq -r \
+ '.[] | [.dnsName, .recordType] | @tsv' \
+ <<< "$certificate_records_json")"
+ fi
+
+ while IFS=$'\t' read -r record_name record_type; do
+ [ -n "$record_name" ] || continue
+ [ "$record_type" = CNAME ] || die \
+ "Managed MQTTS validation record '$record_name' has unsupported type '$record_type'."
+ validation_record_is_in_managed_domain "$record_name" || die \
+ "Managed MQTTS validation record '$record_name' is outside the configured ExternalDNS domains."
+ done <<< "$MQTTS_VALIDATION_DNS_RECORDS"
+}
+
+validate_stack_ingress_for_managed_destroy() {
+ local ingress_certificate_arn
+ local ingress_hostname
+ local managed_dns_label
+
+ [ -n "$STACK_INGRESS_JSON" ] || return 0
+
+ managed_dns_label="$(jq -r \
+ '.metadata.labels["openremote.io/managed-dns"] // empty' \
+ <<< "$STACK_INGRESS_JSON")"
+ ingress_hostname="$(jq -r \
+ '.metadata.annotations["external-dns.kubernetes.io/hostname"] // empty' <<< "$STACK_INGRESS_JSON")"
+ ingress_certificate_arn="$(jq -r \
+ '.metadata.annotations["alb.ingress.kubernetes.io/certificate-arn"] // empty' \
+ <<< "$STACK_INGRESS_JSON")"
+
+ [ "$managed_dns_label" = "true" ] || die \
+ "Ingress '$STACK_NAME/proxy' is not marked for OpenRemote-managed DNS; refusing to delete it."
+ [ "$ingress_hostname" = "$HOSTNAME_VALUE" ] || die \
+ "Ingress '$STACK_NAME/proxy' records hostname '$ingress_hostname', not '$HOSTNAME_VALUE'."
+ [ "$ingress_certificate_arn" = "$CERTIFICATE_ARN" ] || die \
+ "Ingress '$STACK_NAME/proxy' uses certificate '$ingress_certificate_arn', not owned certificate '$CERTIFICATE_ARN'."
+}
+
+capture_validation_dns_records_for_destroy() {
+ local certificate_records_json='[]'
+ local endpoint_certificate_arn
+ local endpoint_managed_by
+ local endpoint_managed_dns
+ local endpoint_records_json='[]'
+ local validation_endpoint=""
+
+ VALIDATION_DNS_ENDPOINT_EXISTS=false
+ VALIDATION_DNS_RECORDS=""
+ if [ "$CERTIFICATE_EXISTS" = true ]; then
+ certificate_records_json="$(jq -c \
+ '[.Certificate.DomainValidationOptions[]?.ResourceRecord |
+ select((.Name // "") != "" and (.Type // "") != "" and
+ (.Value // "") != "") | {
+ dnsName: .Name,
+ recordType: .Type,
+ targets: [.Value]
+ }] | unique_by(.dnsName, .recordType, .targets[0]) |
+ sort_by(.dnsName, .recordType, .targets[0])' \
+ <<< "$CERTIFICATE_JSON")" || die \
+ "Unable to read validation records from managed certificate '$CERTIFICATE_ARN'."
+ fi
+
+ if [ "$STACK_NAMESPACE_EXISTS" = true ]; then
+ validation_endpoint="$(kubectl --context "$KUBE_CONTEXT" \
+ --namespace "$STACK_NAME" \
+ get dnsendpoint "$ACM_VALIDATION_DNS_ENDPOINT_NAME" \
+ --ignore-not-found \
+ -o json)" || die \
+ "Unable to inspect DNSEndpoint '$STACK_NAME/$ACM_VALIDATION_DNS_ENDPOINT_NAME' before destroy."
+ fi
+
+ if [ -n "$validation_endpoint" ]; then
+ VALIDATION_DNS_ENDPOINT_EXISTS=true
+ endpoint_managed_by="$(jq -r \
+ '.metadata.labels["app.kubernetes.io/managed-by"] // empty' \
+ <<< "$validation_endpoint")"
+ endpoint_managed_dns="$(jq -r \
+ '.metadata.labels["openremote.io/managed-dns"] // empty' \
+ <<< "$validation_endpoint")"
+ endpoint_certificate_arn="$(jq -r \
+ '.metadata.annotations["openremote.io/certificate-arn"] // empty' \
+ <<< "$validation_endpoint")"
+ [ "$endpoint_managed_by" = "$CERTIFICATE_MANAGED_BY_TAG_VALUE" ] && \
+ [ "$endpoint_managed_dns" = "true" ] && \
+ [ "$endpoint_certificate_arn" = "$CERTIFICATE_ARN" ] || die \
+ "DNSEndpoint '$STACK_NAME/$ACM_VALIDATION_DNS_ENDPOINT_NAME' does not exactly match managed certificate '$CERTIFICATE_ARN'; refusing to delete it."
+
+ endpoint_records_json="$(jq -c \
+ '[.spec.endpoints[]? | {
+ dnsName: (.dnsName // ""),
+ recordType: (.recordType // ""),
+ targets: ((.targets // []) | sort)
+ }] | unique_by(.dnsName, .recordType, .targets[0]) |
+ sort_by(.dnsName, .recordType, .targets[0])' \
+ <<< "$validation_endpoint")" || die \
+ "Unable to read records from DNSEndpoint '$STACK_NAME/$ACM_VALIDATION_DNS_ENDPOINT_NAME'."
+ [ "$(jq 'length' <<< "$endpoint_records_json")" -gt 0 ] || die \
+ "DNSEndpoint '$STACK_NAME/$ACM_VALIDATION_DNS_ENDPOINT_NAME' has no validation records."
+ if [ "$CERTIFICATE_EXISTS" = true ] && \
+ [ "$endpoint_records_json" != "$certificate_records_json" ]; then
+ die "DNSEndpoint '$STACK_NAME/$ACM_VALIDATION_DNS_ENDPOINT_NAME' does not contain exactly the records returned by ACM; refusing destructive cleanup."
+ fi
+ elif [ "$CERTIFICATE_EXISTS" = false ]; then
+ die "Managed certificate '$CERTIFICATE_ARN' and its validation DNSEndpoint are both absent; validation-record ownership cannot be verified."
+ fi
+
+ if [ "$VALIDATION_DNS_ENDPOINT_EXISTS" = true ]; then
+ VALIDATION_DNS_RECORDS="$(jq -r \
+ '.[] | [.dnsName, .recordType] | @tsv' \
+ <<< "$endpoint_records_json")"
+ else
+ VALIDATION_DNS_RECORDS="$(jq -r \
+ '.[] | [.dnsName, .recordType] | @tsv' \
+ <<< "$certificate_records_json")"
+ fi
+
+ while IFS=$'\t' read -r record_name record_type; do
+ [ -n "$record_name" ] || continue
+ [ "$record_type" = "CNAME" ] || die \
+ "Managed validation record '$record_name' has unsupported type '$record_type'."
+ validation_record_is_in_managed_domain "$record_name" || die \
+ "Managed validation record '$record_name' is outside the configured ExternalDNS domains."
+ done <<< "$VALIDATION_DNS_RECORDS"
+}
+
+require_actionable_managed_certificate_status() {
+ local certificate_status="$1"
+
+ case "$certificate_status" in
+ PENDING_VALIDATION|ISSUED)
+ ;;
+ *)
+ die "Managed ACM certificate '$CERTIFICATE_ARN' is '$certificate_status' and cannot be reconciled automatically."
+ ;;
+ esac
+}
+
+wait_for_managed_certificate_validation_records() {
+ local certificate_json
+ local certificate_status="unknown"
+ local deadline
+ local describe_status
+ local last_error=""
+ local next_notice
+ local now
+ local timeout_seconds
+
+ timeout_seconds="$(duration_seconds "$TIMEOUT")"
+ now="$(date +%s)"
+ deadline="$((now + timeout_seconds))"
+ next_notice="$((now + 30))"
+ log "Waiting for ACM to publish DNS validation records."
+
+ while [ "$now" -lt "$deadline" ]; do
+ set +e
+ certificate_json="$(aws_acm describe-certificate \
+ --certificate-arn "$CERTIFICATE_ARN" \
+ --output json 2>&1)"
+ describe_status=$?
+ set -e
+ if [ "$describe_status" -eq 0 ]; then
+ certificate_status="$(jq -r '.Certificate.Status // empty' \
+ <<< "$certificate_json")" || die \
+ "Unable to read managed ACM certificate metadata."
+ require_actionable_managed_certificate_status "$certificate_status"
+ jq -e \
+ --arg hostname "$HOSTNAME_VALUE" \
+ '(.Certificate.DomainName | ascii_downcase) == $hostname and
+ ([.Certificate.DomainValidationOptions[]?] | length) > 0 and
+ ([.Certificate.DomainValidationOptions[]?] |
+ all((.ResourceRecord.Name // "") != "" and
+ (.ResourceRecord.Type // "") != "" and
+ (.ResourceRecord.Value // "") != ""))' \
+ <<< "$certificate_json" >/dev/null && {
+ CERTIFICATE_JSON="$certificate_json"
+ return
+ }
+ last_error="ACM has not published every validation record yet"
+ else
+ last_error="$certificate_json"
+ fi
+
+ if [ "$now" -ge "$next_notice" ]; then
+ log "ACM validation records are still pending."
+ next_notice="$((now + 30))"
+ fi
+ sleep 5
+ now="$(date +%s)"
+ done
+
+ die "ACM did not publish DNS validation records for '$CERTIFICATE_ARN' within '$TIMEOUT': $last_error"
+}
+
+validation_record_is_in_managed_domain() {
+ local domain_filter
+ local record_name="$1"
+
+ while IFS= read -r domain_filter; do
+ [ -n "$domain_filter" ] || continue
+ if dns_name_is_covered "$record_name" "$domain_filter"; then
+ return 0
+ fi
+ done < <(sed -n 's/^--domain-filter=//p' <<< "$EXTERNAL_DNS_ARGS")
+ return 1
+}
+
+validate_managed_certificate_records() {
+ local record_name
+ local record_target
+ local record_type
+
+ while IFS=$'\t' read -r record_name record_type record_target; do
+ [ -n "$record_name" ] && [ -n "$record_target" ] || die \
+ "ACM returned an incomplete DNS validation record for '$CERTIFICATE_ARN'."
+ [ "$record_type" = "CNAME" ] || die \
+ "ACM returned unsupported validation record type '$record_type'."
+ validation_record_is_in_managed_domain "$record_name" || die \
+ "ACM validation record '$record_name' is outside the configured ExternalDNS domains."
+ done < <(jq -r \
+ '.Certificate.DomainValidationOptions[]?.ResourceRecord |
+ [.Name, .Type, .Value] | @tsv' <<< "$CERTIFICATE_JSON")
+}
+
+require_owned_validation_dns_endpoint() {
+ local existing_endpoint
+ local existing_certificate_arn
+ local existing_managed_by
+
+ # Only NotFound permits creation; a failed read cannot establish ownership.
+ existing_endpoint="$(kubectl --context "$KUBE_CONTEXT" \
+ --namespace "$STACK_NAME" \
+ get dnsendpoint "$ACM_VALIDATION_DNS_ENDPOINT_NAME" \
+ --ignore-not-found -o json)" || die \
+ "Unable to inspect DNSEndpoint '$STACK_NAME/$ACM_VALIDATION_DNS_ENDPOINT_NAME'; refusing to write ACM validation records without ownership checks."
+ [ -n "$existing_endpoint" ] || return 0
+
+ existing_managed_by="$(jq -r \
+ '.metadata.labels["app.kubernetes.io/managed-by"] // empty' \
+ <<< "$existing_endpoint")"
+ existing_certificate_arn="$(jq -r \
+ '.metadata.annotations["openremote.io/certificate-arn"] // empty' \
+ <<< "$existing_endpoint")"
+ [ "$existing_managed_by" = "$CERTIFICATE_MANAGED_BY_TAG_VALUE" ] && \
+ [ "$existing_certificate_arn" = "$CERTIFICATE_ARN" ] || die \
+ "DNSEndpoint '$STACK_NAME/$ACM_VALIDATION_DNS_ENDPOINT_NAME' is not owned by managed certificate '$CERTIFICATE_ARN'; refusing to replace it."
+}
+
+apply_managed_certificate_validation_records() {
+ local endpoint_manifest
+
+ validate_managed_certificate_records
+ require_owned_validation_dns_endpoint
+ endpoint_manifest="$(jq -c \
+ --arg name "$ACM_VALIDATION_DNS_ENDPOINT_NAME" \
+ --arg namespace "$STACK_NAME" \
+ --arg certificate_arn "$CERTIFICATE_ARN" \
+ --arg managed_by "$CERTIFICATE_MANAGED_BY_TAG_VALUE" \
+ '{
+ apiVersion: "externaldns.k8s.io/v1alpha1",
+ kind: "DNSEndpoint",
+ metadata: {
+ name: $name,
+ namespace: $namespace,
+ labels: {
+ "app.kubernetes.io/managed-by": $managed_by,
+ "openremote.io/managed-dns": "true"
+ },
+ annotations: {
+ "openremote.io/certificate-arn": $certificate_arn
+ }
+ },
+ spec: {
+ endpoints: ([.Certificate.DomainValidationOptions[]?.ResourceRecord | {
+ dnsName: .Name,
+ recordType: .Type,
+ recordTTL: 300,
+ targets: [.Value]
+ }] | unique_by(.dnsName, .recordType, .targets[0]))
+ }
+ }' <<< "$CERTIFICATE_JSON")" || die \
+ "Unable to render ACM validation records as a DNSEndpoint."
+
+ log "Reconciling ACM DNS validation records in '$STACK_NAME/$ACM_VALIDATION_DNS_ENDPOINT_NAME'."
+ kubectl --context "$KUBE_CONTEXT" \
+ --namespace "$STACK_NAME" \
+ apply -f - <<< "$endpoint_manifest" >/dev/null || die \
+ "Unable to apply ACM validation DNSEndpoint '$STACK_NAME/$ACM_VALIDATION_DNS_ENDPOINT_NAME'."
+}
+
+wait_for_managed_certificate() {
+ local certificate_status="unknown"
+ local deadline
+ local next_notice
+ local now
+ local timeout_seconds
+
+ timeout_seconds="$(duration_seconds "$TIMEOUT")"
+ now="$(date +%s)"
+ deadline="$((now + timeout_seconds))"
+ next_notice="$((now + 30))"
+ log "Waiting for managed ACM certificate '$CERTIFICATE_ARN' to be issued."
+
+ while [ "$now" -lt "$deadline" ]; do
+ certificate_status="$(aws_acm describe-certificate \
+ --certificate-arn "$CERTIFICATE_ARN" \
+ --query 'Certificate.Status' \
+ --output text 2>/dev/null || true)"
+ if [ "$certificate_status" = "ISSUED" ]; then
+ log "Managed ACM certificate '$CERTIFICATE_ARN' is issued."
+ return
+ fi
+ if [ -n "$certificate_status" ] && [ "$certificate_status" != "None" ]; then
+ require_actionable_managed_certificate_status "$certificate_status"
+ fi
+ if [ "$now" -ge "$next_notice" ]; then
+ log "Managed ACM certificate validation is still pending."
+ next_notice="$((now + 30))"
+ fi
+ sleep 5
+ now="$(date +%s)"
+ done
+
+ die "Managed ACM certificate '$CERTIFICATE_ARN' remained '$certificate_status' beyond '$TIMEOUT'. The validation DNSEndpoint is retained for a later retry."
+}
+
+apply_portable_stack() {
+ local stack_args=(
+ apply
+ --name "$STACK_NAME"
+ --kube-context "$KUBE_CONTEXT"
+ --target eks
+ --exposure "$EXPOSURE"
+ --dns external-dns
+ --hostname "$HOSTNAME_VALUE"
+ --timeout "$TIMEOUT"
+ )
+
+ if [ "$EXPOSURE" = "haproxy" ]; then
+ stack_args+=(--http-port "$HTTP_PORT" --https-port "$HTTPS_PORT")
+ if [ "$MQTTS_ENABLED" = true ]; then
+ stack_args+=(--mqtts --mqtts-port "$MQTTS_PORT")
+ fi
+ else
+ stack_args+=(--certificate-arn "$CERTIFICATE_ARN")
+ if [ "$MQTTS_ENABLED" = true ]; then
+ stack_args+=(
+ --mqtts
+ --mqtts-port "$MQTTS_PORT"
+ --mqtts-hostname "$MQTTS_HOSTNAME_VALUE"
+ --mqtts-certificate-arn "$MQTTS_CERTIFICATE_ARN"
+ )
+ fi
+ fi
+ if [ -n "$VALUES_DIR" ]; then
+ stack_args+=(--values-dir "$VALUES_DIR")
+ fi
+
+ run_or_stack "${stack_args[@]}"
+}
+
+record_certificate_configuration() {
+ local certificate_mode=acme
+
+ if [ "$EXPOSURE" = "ingress" ]; then
+ certificate_mode="$CERTIFICATE_MODE"
+ kubectl --context "$KUBE_CONTEXT" annotate namespace "$STACK_NAME" \
+ "$CERTIFICATE_MODE_ANNOTATION=$certificate_mode" \
+ "$CERTIFICATE_ARN_ANNOTATION=$CERTIFICATE_ARN" \
+ "$HOSTNAME_ANNOTATION=$HOSTNAME_VALUE" \
+ --overwrite >/dev/null
+ else
+ kubectl --context "$KUBE_CONTEXT" annotate namespace "$STACK_NAME" \
+ "$CERTIFICATE_MODE_ANNOTATION=$certificate_mode" \
+ "$HOSTNAME_ANNOTATION=$HOSTNAME_VALUE" \
+ --overwrite >/dev/null
+ fi
+}
+
+record_mqtts_certificate_configuration() {
+ [ "$EXPOSURE" = ingress ] && [ "$MQTTS_ENABLED" = true ] || return 0
+ kubectl --context "$KUBE_CONTEXT" annotate namespace "$STACK_NAME" \
+ "$MQTTS_CERTIFICATE_MODE_ANNOTATION=$MQTTS_CERTIFICATE_MODE" \
+ "$MQTTS_CERTIFICATE_ARN_ANNOTATION=$MQTTS_CERTIFICATE_ARN" \
+ "$MQTTS_HOSTNAME_ANNOTATION=$MQTTS_HOSTNAME_VALUE" \
+ --overwrite >/dev/null || die \
+ "Unable to record the Ingress MQTTS certificate configuration."
+}
+
+wait_for_load_balancer() {
+ local deadline
+ local load_balancer_kind
+ local next_notice
+ local now
+ local resource_kind
+ local timeout_seconds
+
+ if [ "$EXPOSURE" = "ingress" ]; then
+ load_balancer_kind="shared ALB"
+ resource_kind="ingress"
+ else
+ load_balancer_kind="stack NLB"
+ resource_kind="service"
+ fi
+ timeout_seconds="$(duration_seconds "$TIMEOUT")"
+ now="$(date +%s)"
+ deadline="$((now + timeout_seconds))"
+ next_notice="$((now + 30))"
+ LOAD_BALANCER_ADDRESS=""
+ log "Waiting for the $load_balancer_kind address."
+
+ while [ "$now" -lt "$deadline" ]; do
+ LOAD_BALANCER_ADDRESS="$(kubectl --context "$KUBE_CONTEXT" \
+ --namespace "$STACK_NAME" \
+ get "$resource_kind" proxy \
+ -o 'jsonpath={.status.loadBalancer.ingress[0].ip}{.status.loadBalancer.ingress[0].hostname}' \
+ 2>/dev/null || true)"
+ if [ -n "$LOAD_BALANCER_ADDRESS" ]; then
+ log "The $load_balancer_kind is '$LOAD_BALANCER_ADDRESS'."
+ return
+ fi
+ if [ "$now" -ge "$next_notice" ]; then
+ log "The $load_balancer_kind address is still pending."
+ next_notice="$((now + 30))"
+ fi
+ sleep 5
+ now="$(date +%s)"
+ done
+
+ die "The proxy $resource_kind did not receive a $load_balancer_kind address within '$TIMEOUT'; inspect '$STACK_NAME/proxy' and the AWS Load Balancer Controller."
+}
+
+wait_for_mqtts_load_balancer() {
+ local deadline
+ local next_notice
+ local now
+ local timeout_seconds
+
+ timeout_seconds="$(duration_seconds "$TIMEOUT")"
+ now="$(date +%s)"
+ deadline="$((now + timeout_seconds))"
+ next_notice="$((now + 30))"
+ MQTTS_LOAD_BALANCER_ADDRESS=""
+ log "Waiting for the stack MQTTS NLB address."
+
+ while [ "$now" -lt "$deadline" ]; do
+ MQTTS_LOAD_BALANCER_ADDRESS="$(kubectl --context "$KUBE_CONTEXT" \
+ --namespace "$STACK_NAME" \
+ get service proxy-mqtts \
+ -o 'jsonpath={.status.loadBalancer.ingress[0].ip}{.status.loadBalancer.ingress[0].hostname}' \
+ 2>/dev/null || true)"
+ if [ -n "$MQTTS_LOAD_BALANCER_ADDRESS" ]; then
+ log "The stack MQTTS NLB is '$MQTTS_LOAD_BALANCER_ADDRESS'."
+ return
+ fi
+ if [ "$now" -ge "$next_notice" ]; then
+ log "The stack MQTTS NLB address is still pending."
+ next_notice="$((now + 30))"
+ fi
+ sleep 5
+ now="$(date +%s)"
+ done
+
+ die "Service '$STACK_NAME/proxy-mqtts' did not receive an NLB address within '$TIMEOUT'; inspect the Service and the AWS Load Balancer Controller."
+}
+
+resolve_addresses() {
+ local dns_name="$1"
+
+ {
+ dig +short "$dns_name" A 2>/dev/null || true
+ dig +short "$dns_name" AAAA 2>/dev/null || true
+ } | awk '/^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$/ || index($0, ":") > 0 { print }' | sort -u
+}
+
+addresses_intersect() {
+ local first_addresses="$1"
+ local second_addresses="$2"
+ local address
+
+ while IFS= read -r address; do
+ [ -n "$address" ] || continue
+ if grep -Fx -- "$address" <<< "$second_addresses" >/dev/null; then
+ return 0
+ fi
+ done <<< "$first_addresses"
+ return 1
+}
+
+wait_for_dns() {
+ local deadline
+ local hostname_addresses
+ local load_balancer_address="$1"
+ local hostname="${2:-$HOSTNAME_VALUE}"
+ local load_balancer_addresses
+ local next_notice
+ local now
+ local timeout_seconds
+
+ timeout_seconds="$(duration_seconds "$TIMEOUT")"
+ now="$(date +%s)"
+ deadline="$((now + timeout_seconds))"
+ next_notice="$((now + 30))"
+ log "Waiting for '$hostname' to resolve to the current load balancer '$load_balancer_address'."
+
+ while [ "$now" -lt "$deadline" ]; do
+ hostname_addresses="$(resolve_addresses "$hostname")"
+ if [[ "$load_balancer_address" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]] || \
+ [[ "$load_balancer_address" == *:* ]]; then
+ load_balancer_addresses="$load_balancer_address"
+ else
+ load_balancer_addresses="$(resolve_addresses "$load_balancer_address")"
+ fi
+
+ if [ -n "$hostname_addresses" ] && [ -n "$load_balancer_addresses" ] && \
+ addresses_intersect "$hostname_addresses" "$load_balancer_addresses"; then
+ log "DNS now resolves to the current load balancer."
+ return
+ fi
+ if [ "$now" -ge "$next_notice" ]; then
+ log "DNS is still pending or points to another load balancer."
+ next_notice="$((now + 30))"
+ fi
+ sleep 5
+ now="$(date +%s)"
+ done
+
+ die "DNS '$hostname' did not resolve to the current load balancer within '$TIMEOUT'. Inspect deployment/$EXTERNAL_DNS_RELEASE_NAME logs and allow for resolver negative caching before retrying."
+}
+
+select_managed_dns_domain() {
+ local domain_filter
+ local hostname="${1:-$HOSTNAME_VALUE}"
+
+ MANAGED_DNS_DOMAIN=""
+ while IFS= read -r domain_filter; do
+ domain_filter="${domain_filter%.}"
+ [ -n "$domain_filter" ] || continue
+ if dns_name_is_covered "$hostname" "$domain_filter" && \
+ [ "${#domain_filter}" -gt "${#MANAGED_DNS_DOMAIN}" ]; then
+ MANAGED_DNS_DOMAIN="$domain_filter"
+ fi
+ done < <(sed -n 's/^--domain-filter=//p' <<< "$EXTERNAL_DNS_ARGS")
+
+ [ -n "$MANAGED_DNS_DOMAIN" ] || die \
+ "Unable to select an ExternalDNS domain for '$hostname'."
+}
+
+discover_authoritative_name_servers() {
+ local candidate_domain="$MANAGED_DNS_DOMAIN"
+ local name_servers
+
+ AUTHORITATIVE_NAME_SERVERS=""
+ while [[ "$candidate_domain" == *.* ]]; do
+ name_servers="$(dig +time=5 +tries=2 +short "$candidate_domain" NS)" || die \
+ "Unable to discover authoritative name servers for '$candidate_domain'."
+ name_servers="$(awk \
+ '/^[A-Za-z0-9]([A-Za-z0-9.-]*[A-Za-z0-9])?\.?$/ { print }' \
+ <<< "$name_servers" | sort -u)"
+ if [ -n "$name_servers" ]; then
+ AUTHORITATIVE_NAME_SERVERS="$name_servers"
+ MANAGED_DNS_DOMAIN="$candidate_domain"
+ log "Using authoritative DNS for '$MANAGED_DNS_DOMAIN' while verifying record removal."
+ return
+ fi
+ candidate_domain="${candidate_domain#*.}"
+ done
+
+ die "Unable to find authoritative name servers for managed DNS domain '$MANAGED_DNS_DOMAIN'."
+}
+
+authoritative_dns_record_exists() {
+ local answer
+ local name_server
+ local query_status
+ local query_failed=false
+ local record_name="$1"
+ local record_type="$2"
+
+ while IFS= read -r name_server; do
+ [ -n "$name_server" ] || continue
+ set +e
+ answer="$(dig +time=3 +tries=1 +norecurse +noall +comments +answer +authority \
+ "@$name_server" "$record_name" "$record_type" 2>/dev/null)"
+ query_status=$?
+ set -e
+ # A successful dig process can carry SERVFAIL/REFUSED or a referral. Those
+ # are not evidence of deletion and must never allow a hostname to be reused.
+ if [ "$query_status" -ne 0 ] ||
+ ! grep -Eq '^;; ->>HEADER<<-.*status: (NOERROR|NXDOMAIN),' <<< "$answer" ||
+ ! grep -Eq '^;; flags:([^;]* )?aa[ ;]' <<< "$answer"; then
+ query_failed=true
+ continue
+ fi
+ if awk -v type="$record_type" '!/^;/ && NF >= 5 && ($4 == type || $4 == "CNAME") { found=1 } END { exit !found }' \
+ <<< "$answer"; then
+ return 0
+ fi
+ done <<< "$AUTHORITATIVE_NAME_SERVERS"
+
+ [ "$query_failed" = false ] || return 2
+ return 1
+}
+
+wait_for_dns_records_removed() {
+ local deadline
+ local description="$1"
+ local next_notice
+ local now
+ local query_status
+ local record_name
+ local record_type
+ local records="$2"
+ local records_remain
+ local timeout_seconds
+
+ [ -n "$records" ] || return 0
+ timeout_seconds="$(duration_seconds "$TIMEOUT")"
+ now="$(date +%s)"
+ deadline="$((now + timeout_seconds))"
+ next_notice="$((now + 30))"
+ log "Waiting for $description DNS records to be removed."
+
+ while [ "$now" -lt "$deadline" ]; do
+ records_remain=false
+ while IFS=$'\t' read -r record_name record_type; do
+ [ -n "$record_name" ] || continue
+ if authoritative_dns_record_exists "$record_name" "$record_type"; then
+ query_status=0
+ else
+ query_status=$?
+ fi
+ if [ "$query_status" -ne 1 ]; then
+ records_remain=true
+ break
+ fi
+ done <<< "$records"
+
+ if [ "$records_remain" = false ]; then
+ log "The $description DNS records are removed."
+ return
+ fi
+ if [ "$now" -ge "$next_notice" ]; then
+ log "The $description DNS records are still present or could not be verified."
+ next_notice="$((now + 30))"
+ fi
+ sleep 5
+ now="$(date +%s)"
+ done
+
+ die "The $description DNS records were not confirmed removed within '$TIMEOUT'. External resources were retained so destroy can be retried safely."
+}
+
+remove_managed_stack_ingress() {
+ if [ -z "$STACK_INGRESS_JSON" ]; then
+ log "Ingress '$STACK_NAME/proxy' is already absent."
+ return
+ fi
+
+ log "Deleting Ingress '$STACK_NAME/proxy' before managed certificate cleanup."
+ kubectl --context "$KUBE_CONTEXT" \
+ --namespace "$STACK_NAME" \
+ delete ingress proxy \
+ --wait=true \
+ --timeout "$TIMEOUT" || die \
+ "Unable to delete Ingress '$STACK_NAME/proxy'."
+ STACK_INGRESS_JSON=""
+}
+
+wait_for_managed_certificate_detachment() {
+ local certificate_json
+ local deadline
+ local describe_status
+ local in_use_count
+ local last_error=""
+ local next_notice
+ local now
+ local timeout_seconds
+
+ [ "$CERTIFICATE_EXISTS" = true ] || return 0
+ timeout_seconds="$(duration_seconds "$TIMEOUT")"
+ now="$(date +%s)"
+ deadline="$((now + timeout_seconds))"
+ next_notice="$((now + 30))"
+ log "Waiting for managed ACM certificate '$CERTIFICATE_ARN' to be detached."
+
+ while [ "$now" -lt "$deadline" ]; do
+ set +e
+ certificate_json="$(aws_acm describe-certificate \
+ --certificate-arn "$CERTIFICATE_ARN" \
+ --output json 2>&1)"
+ describe_status=$?
+ set -e
+ if [ "$describe_status" -eq 0 ]; then
+ in_use_count="$(jq -r '.Certificate.InUseBy // [] | length' \
+ <<< "$certificate_json")" || die \
+ "Unable to read attachment state for managed certificate '$CERTIFICATE_ARN'."
+ if [ "$in_use_count" -eq 0 ]; then
+ CERTIFICATE_JSON="$certificate_json"
+ log "Managed ACM certificate '$CERTIFICATE_ARN' is detached."
+ return
+ fi
+ last_error="certificate is still attached to $in_use_count AWS resource(s)"
+ elif [[ "$certificate_json" == *ResourceNotFoundException* ]]; then
+ CERTIFICATE_EXISTS=false
+ CERTIFICATE_JSON=""
+ log "Managed ACM certificate '$CERTIFICATE_ARN' is already absent."
+ return
+ else
+ last_error="$certificate_json"
+ fi
+
+ if [ "$now" -ge "$next_notice" ]; then
+ log "Managed ACM certificate detachment is still pending."
+ next_notice="$((now + 30))"
+ fi
+ sleep 5
+ now="$(date +%s)"
+ done
+
+ die "Managed ACM certificate '$CERTIFICATE_ARN' did not detach within '$TIMEOUT': $last_error"
+}
+
+delete_managed_certificate() {
+ local delete_output
+ local delete_status
+ local describe_output
+ local describe_status
+ local deadline
+ local next_notice
+ local now
+ local timeout_seconds
+
+ [ "$CERTIFICATE_EXISTS" = true ] || return 0
+ require_managed_certificate_ownership
+ log "Deleting managed ACM certificate '$CERTIFICATE_ARN'."
+ set +e
+ delete_output="$(aws_acm delete-certificate \
+ --certificate-arn "$CERTIFICATE_ARN" 2>&1)"
+ delete_status=$?
+ set -e
+ if [ "$delete_status" -ne 0 ]; then
+ case "$delete_output" in
+ *ResourceNotFoundException*)
+ CERTIFICATE_EXISTS=false
+ log "Managed ACM certificate '$CERTIFICATE_ARN' is already absent."
+ return
+ ;;
+ *)
+ die "Unable to delete managed ACM certificate '$CERTIFICATE_ARN': $delete_output"
+ ;;
+ esac
+ fi
+
+ timeout_seconds="$(duration_seconds "$TIMEOUT")"
+ now="$(date +%s)"
+ deadline="$((now + timeout_seconds))"
+ next_notice="$((now + 30))"
+ while [ "$now" -lt "$deadline" ]; do
+ set +e
+ describe_output="$(aws_acm describe-certificate \
+ --certificate-arn "$CERTIFICATE_ARN" \
+ --output json 2>&1)"
+ describe_status=$?
+ set -e
+ if [ "$describe_status" -ne 0 ] && \
+ [[ "$describe_output" == *ResourceNotFoundException* ]]; then
+ CERTIFICATE_EXISTS=false
+ log "Managed ACM certificate '$CERTIFICATE_ARN' was deleted."
+ return
+ fi
+ if [ "$now" -ge "$next_notice" ]; then
+ log "Managed ACM certificate deletion is still propagating."
+ next_notice="$((now + 30))"
+ fi
+ sleep 5
+ now="$(date +%s)"
+ done
+
+ die "Deletion of managed ACM certificate '$CERTIFICATE_ARN' was not confirmed within '$TIMEOUT'."
+}
+
+wait_for_managed_mqtts_certificate_detachment() {
+ local certificate_json
+ local deadline
+ local describe_status
+ local in_use_count
+ local last_error=""
+ local next_notice
+ local now
+ local timeout_seconds
+
+ [ "$MQTTS_CERTIFICATE_EXISTS" = true ] || return 0
+ timeout_seconds="$(duration_seconds "$TIMEOUT")"
+ now="$(date +%s)"
+ deadline="$((now + timeout_seconds))"
+ next_notice="$((now + 30))"
+ log "Waiting for managed MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN' to be detached."
+
+ while [ "$now" -lt "$deadline" ]; do
+ set +e
+ certificate_json="$(aws_acm describe-certificate \
+ --certificate-arn "$MQTTS_CERTIFICATE_ARN" \
+ --output json 2>&1)"
+ describe_status=$?
+ set -e
+ if [ "$describe_status" -eq 0 ]; then
+ in_use_count="$(jq -r '.Certificate.InUseBy // [] | length' \
+ <<< "$certificate_json")" || die \
+ "Unable to read attachment state for managed MQTTS certificate '$MQTTS_CERTIFICATE_ARN'."
+ if [ "$in_use_count" -eq 0 ]; then
+ MQTTS_CERTIFICATE_JSON="$certificate_json"
+ log "Managed MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN' is detached."
+ return
+ fi
+ last_error="certificate is still attached to $in_use_count AWS resource(s)"
+ elif [[ "$certificate_json" == *ResourceNotFoundException* ]]; then
+ MQTTS_CERTIFICATE_EXISTS=false
+ MQTTS_CERTIFICATE_JSON=""
+ log "Managed MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN' is already absent."
+ return
+ else
+ last_error="$certificate_json"
+ fi
+
+ if [ "$now" -ge "$next_notice" ]; then
+ log "Managed MQTTS ACM certificate detachment is still pending."
+ next_notice="$((now + 30))"
+ fi
+ sleep 5
+ now="$(date +%s)"
+ done
+
+ die "Managed MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN' did not detach within '$TIMEOUT': $last_error"
+}
+
+delete_managed_mqtts_certificate() {
+ local delete_output
+ local delete_status
+ local describe_output
+ local describe_status
+ local deadline
+ local next_notice
+ local now
+ local timeout_seconds
+
+ [ "$MQTTS_CERTIFICATE_EXISTS" = true ] || return 0
+ require_managed_mqtts_certificate_ownership
+ log "Deleting managed MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN'."
+ set +e
+ delete_output="$(aws_acm delete-certificate \
+ --certificate-arn "$MQTTS_CERTIFICATE_ARN" 2>&1)"
+ delete_status=$?
+ set -e
+ if [ "$delete_status" -ne 0 ]; then
+ case "$delete_output" in
+ *ResourceNotFoundException*)
+ MQTTS_CERTIFICATE_EXISTS=false
+ log "Managed MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN' is already absent."
+ return
+ ;;
+ *)
+ die "Unable to delete managed MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN': $delete_output"
+ ;;
+ esac
+ fi
+
+ timeout_seconds="$(duration_seconds "$TIMEOUT")"
+ now="$(date +%s)"
+ deadline="$((now + timeout_seconds))"
+ next_notice="$((now + 30))"
+ while [ "$now" -lt "$deadline" ]; do
+ set +e
+ describe_output="$(aws_acm describe-certificate \
+ --certificate-arn "$MQTTS_CERTIFICATE_ARN" \
+ --output json 2>&1)"
+ describe_status=$?
+ set -e
+ if [ "$describe_status" -ne 0 ] && \
+ [[ "$describe_output" == *ResourceNotFoundException* ]]; then
+ MQTTS_CERTIFICATE_EXISTS=false
+ log "Managed MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN' was deleted."
+ return
+ fi
+ if [ "$now" -ge "$next_notice" ]; then
+ log "Managed MQTTS ACM certificate deletion is still propagating."
+ next_notice="$((now + 30))"
+ fi
+ sleep 5
+ now="$(date +%s)"
+ done
+
+ die "Deletion of managed MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN' was not confirmed within '$TIMEOUT'."
+}
+
+public_manager_url() {
+ printf 'https://%s' "$HOSTNAME_VALUE"
+ if [ "$EXPOSURE" = "haproxy" ] && [ "$HTTPS_PORT" != "443" ]; then
+ printf ':%s' "$HTTPS_PORT"
+ fi
+ printf '/manager/'
+}
+
+endpoint_is_ready() {
+ local endpoint_url="$1"
+
+ curl --disable --fail --silent --show-error --location \
+ --proto '=https' \
+ --proto-redir '=https' \
+ --max-redirs 5 \
+ --connect-timeout 10 \
+ --max-time 30 \
+ --output /dev/null \
+ "$endpoint_url"
+}
+
+proxy_certificate_state() {
+ local state
+ state="$(kubectl --context "$KUBE_CONTEXT" \
+ --namespace "$STACK_NAME" exec deployment/proxy --container proxy \
+ -- sh -c '
+ lineage="$LE_DIR/live/$1"
+ if [ -s "$lineage/privkey.pem" ] && [ -s "$lineage/fullchain.pem" ]; then
+ printf "present\n"
+ elif [ -e "$lineage" ] || [ -L "$lineage" ]; then
+ printf "incomplete\n"
+ else
+ printf "absent\n"
+ fi
+ ' sh "$HOSTNAME_VALUE")" || die \
+ "Unable to inspect HAProxy certificate state; refusing to infer that issuance is needed."
+ case "$state" in
+ present|incomplete|absent) printf '%s\n' "$state" ;;
+ *) die "Unexpected HAProxy certificate state '$state'; inspect deployment/proxy before retrying." ;;
+ esac
+}
+
+trigger_acme() {
+ local state
+ # Manager errors and connection failures say nothing about persisted TLS
+ # state. add rejects an existing lineage; renew would force fresh issuance.
+ # Leave existing certificates to the proxy's normal renewal lifecycle.
+ state="$(proxy_certificate_state)" || die "Unable to determine whether HAProxy needs ACME issuance."
+ case "$state" in
+ present)
+ log "Reusing HAProxy certificate state for '$HOSTNAME_VALUE'."
+ return
+ ;;
+ incomplete)
+ die "HAProxy certificate state for '$HOSTNAME_VALUE' is incomplete. Inspect its persisted letsencrypt directory and proxy logs before retrying; refusing to overwrite existing state."
+ ;;
+ esac
+
+ log "Triggering HAProxy ACME issuance for '$HOSTNAME_VALUE'."
+ if kubectl --context "$KUBE_CONTEXT" \
+ --namespace "$STACK_NAME" \
+ exec deployment/proxy \
+ --container proxy \
+ -- /entrypoint.sh add "$HOSTNAME_VALUE"; then
+ return
+ fi
+ # Startup cert_init runs concurrently and may have created the certificate
+ # after our first inspection. Only accept the failure if both files now exist;
+ # trusted TLS and Manager readiness are still checked independently below.
+ state="$(proxy_certificate_state)" || die "Unable to inspect HAProxy state after ACME issuance failed."
+ if [ "$state" = present ]; then
+ log "HAProxy certificate state appeared during issuance; waiting for trusted HTTPS."
+ return
+ fi
+ die "HAProxy ACME issuance failed (certificate state: $state). DNS is ready; inspect deployment/proxy logs and retry apply."
+}
+
+wait_for_https() {
+ local deadline
+ local endpoint_url="$1"
+ local next_notice
+ local now
+ local timeout_seconds
+
+ timeout_seconds="$(duration_seconds "$TIMEOUT")"
+ now="$(date +%s)"
+ deadline="$((now + timeout_seconds))"
+ next_notice="$((now + 30))"
+ log "Waiting for a trusted certificate and Manager response at '$endpoint_url'."
+
+ while [ "$now" -lt "$deadline" ]; do
+ if endpoint_is_ready "$endpoint_url" 2>/dev/null; then
+ log "Trusted HTTPS is ready."
+ return
+ fi
+ if [ "$now" -ge "$next_notice" ]; then
+ log "The trusted HTTPS endpoint is still pending."
+ next_notice="$((now + 30))"
+ fi
+ sleep 5
+ now="$(date +%s)"
+ done
+
+ if [ "$EXPOSURE" = "ingress" ]; then
+ die "Trusted HTTPS did not become ready at '$endpoint_url' within '$TIMEOUT'. Inspect ingress '$STACK_NAME/proxy', the shared ALB, and its ACM listener certificate."
+ fi
+ die "Trusted HTTPS did not become ready at '$endpoint_url' within '$TIMEOUT'. Inspect deployment/proxy logs and its persisted certificate state."
+}
+
+mqtts_endpoint_is_ready() {
+ local hostname
+ local probe_timeout="$1"
+
+ hostname="$HOSTNAME_VALUE"
+ if [ "$EXPOSURE" = ingress ]; then
+ hostname="$MQTTS_HOSTNAME_VALUE"
+ fi
+
+ # s_client can print "Verify return code: 0" without completing TLS, and SNI
+ # alone does not verify the hostname. Use the TLS API's verified handshake;
+ # no MQTT exchange is needed and peer shutdown after the handshake is harmless.
+ python3 - "$hostname" "$MQTTS_PORT" "$probe_timeout" <<'PY'
+import signal
+import socket
+import ssl
+import sys
+
+def timed_out(signum, frame):
+ raise TimeoutError("MQTTS TLS probe timed out")
+
+# Bound DNS resolution as well as connection/handshake on the Unix platforms
+# supported by these scripts, without requiring GNU timeout on macOS.
+signal.signal(signal.SIGALRM, timed_out)
+signal.alarm(int(sys.argv[3]))
+try:
+ # Pass resolved paths explicitly: macOS LibreSSL can ignore SSL_CERT_FILE
+ # when loading defaults, even though get_default_verify_paths resolves it.
+ trust = ssl.get_default_verify_paths()
+ context = ssl.create_default_context(cafile=trust.cafile, capath=trust.capath)
+ with socket.create_connection((sys.argv[1], int(sys.argv[2]))) as connection:
+ with context.wrap_socket(connection, server_hostname=sys.argv[1]):
+ pass
+except (OSError, ValueError):
+ sys.exit(1)
+finally:
+ signal.alarm(0)
+PY
+}
+
+wait_for_mqtts() {
+ local deadline
+ local next_notice
+ local now
+ local timeout_seconds
+ local probe_timeout
+ local hostname="$HOSTNAME_VALUE"
+ local service_name=proxy
+
+ if [ "$EXPOSURE" = ingress ]; then
+ hostname="$MQTTS_HOSTNAME_VALUE"
+ service_name=proxy-mqtts
+ fi
+
+ timeout_seconds="$(duration_seconds "$TIMEOUT")"
+ now="$(date +%s)"
+ deadline="$((now + timeout_seconds))"
+ next_notice="$((now + 30))"
+ log "Waiting for trusted MQTTS at '$hostname:$MQTTS_PORT'."
+
+ while [ "$now" -lt "$deadline" ]; do
+ probe_timeout="$((deadline - now))"
+ [ "$probe_timeout" -le 10 ] || probe_timeout=10
+ if mqtts_endpoint_is_ready "$probe_timeout"; then
+ log "Trusted MQTTS is ready."
+ return
+ fi
+ now="$(date +%s)"
+ [ "$now" -lt "$deadline" ] || break
+ if [ "$now" -ge "$next_notice" ]; then
+ log "The trusted MQTTS endpoint is still pending."
+ next_notice="$((now + 30))"
+ fi
+ probe_timeout="$((deadline - now))"
+ [ "$probe_timeout" -le 5 ] || probe_timeout=5
+ sleep "$probe_timeout"
+ now="$(date +%s)"
+ done
+
+ die "Trusted MQTTS did not become ready at '$hostname:$MQTTS_PORT' within '$TIMEOUT'. Inspect service '$STACK_NAME/$service_name' and deployment/proxy logs."
+}
+
+apply_stack() {
+ local endpoint_url
+
+ require_command helm
+ require_command dig
+ require_command curl
+ if [ "$MQTTS_ENABLED" = true ]; then
+ require_command python3
+ fi
+ [ -x "$OR_STACK_COMMAND" ] || die \
+ "Portable stack command '$OR_STACK_COMMAND' is missing or not executable."
+ if [ "$EXPOSURE" = "ingress" ]; then
+ require_command jq
+ fi
+
+ require_eks_target
+ if [ "$EXPOSURE" = "ingress" ]; then
+ resolve_ingress_certificate
+ resolve_ingress_mqtts_configuration
+ if [ "$CERTIFICATE_MODE" = managed ] || \
+ [ "$MQTTS_CERTIFICATE_MODE" = managed ]; then
+ require_command cksum
+ fi
+ if [ "$CERTIFICATE_MODE" = managed ]; then
+ [ "${#HOSTNAME_VALUE}" -le 64 ] || die \
+ "Managed ACM mode currently requires a hostname of at most 64 characters; use an existing or shared certificate for a longer name."
+ fi
+ if [ "$MQTTS_CERTIFICATE_MODE" = managed ]; then
+ [ "${#MQTTS_HOSTNAME_VALUE}" -le 64 ] || die \
+ "Managed MQTTS ACM mode currently requires a hostname of at most 64 characters; use an existing or shared certificate for a longer name."
+ fi
+ fi
+ require_managed_external_dns
+ require_stack_hostname_subdomain
+ if [ "$EXPOSURE" = ingress ] && [ "$MQTTS_ENABLED" = true ]; then
+ require_stack_hostname_subdomain "$MQTTS_HOSTNAME_VALUE"
+ fi
+ if [ "$EXPOSURE" = "ingress" ] && [ "$CERTIFICATE_MODE" != "managed" ]; then
+ require_selected_acm_certificate
+ fi
+ if [ "$EXPOSURE" = ingress ] && [ "$MQTTS_ENABLED" = true ] && \
+ [ "$MQTTS_CERTIFICATE_MODE" != managed ]; then
+ require_selected_mqtts_acm_certificate
+ fi
+ # Reserve before ACM requests as well as before Helm, so a losing concurrent
+ # apply cannot issue certificates or install a competing public endpoint.
+ prepare_portable_stack || die "Namespace preparation or hostname reservation failed; no workloads were installed."
+ if [ "$EXPOSURE" = "ingress" ]; then
+ record_initial_certificate_mode
+ record_initial_mqtts_configuration
+ fi
+ if [ "$CERTIFICATE_MODE" = "managed" ]; then
+ resolve_managed_certificate
+ wait_for_managed_certificate_validation_records
+ apply_managed_certificate_validation_records
+ wait_for_managed_certificate
+ fi
+ if [ "$MQTTS_CERTIFICATE_MODE" = managed ]; then
+ resolve_managed_mqtts_certificate
+ wait_for_managed_mqtts_certificate_validation_records
+ apply_managed_mqtts_certificate_validation_records
+ wait_for_managed_mqtts_certificate
+ fi
+ apply_portable_stack
+ if [ "$MQTTS_CERTIFICATE_MODE" != managed ]; then
+ record_mqtts_certificate_configuration
+ fi
+ if [ "$CERTIFICATE_MODE" != "managed" ]; then
+ record_certificate_configuration
+ fi
+
+ wait_for_load_balancer
+ wait_for_dns "$LOAD_BALANCER_ADDRESS"
+ if [ "$EXPOSURE" = ingress ] && [ "$MQTTS_ENABLED" = true ]; then
+ wait_for_mqtts_load_balancer
+ wait_for_dns "$MQTTS_LOAD_BALANCER_ADDRESS" "$MQTTS_HOSTNAME_VALUE"
+ fi
+ endpoint_url="$(public_manager_url)"
+
+ if [ "$EXPOSURE" = "haproxy" ]; then
+ trigger_acme
+ fi
+ wait_for_https "$endpoint_url"
+ if [ "$MQTTS_ENABLED" = true ]; then
+ wait_for_mqtts
+ fi
+
+ log "Stack '$STACK_NAME' is ready at:"
+ printf ' %s\n' "$endpoint_url"
+ if [ "$MQTTS_ENABLED" = true ]; then
+ if [ "$EXPOSURE" = ingress ]; then
+ printf ' mqtts://%s:%s\n' "$MQTTS_HOSTNAME_VALUE" "$MQTTS_PORT"
+ else
+ printf ' mqtts://%s:%s\n' "$HOSTNAME_VALUE" "$MQTTS_PORT"
+ fi
+ fi
+ log "Retrieve its Manager login at any time with:"
+ printf ' %q credentials --name %q --cluster %q --region %q' \
+ "$0" "$STACK_NAME" "$CLUSTER_NAME" "$AWS_REGION_NAME"
+ if [ -n "$AWS_PROFILE_NAME" ]; then
+ printf ' --profile %q' "$AWS_PROFILE_NAME"
+ fi
+ printf '\n'
+}
+
+show_certificate_configuration() {
+ local certificate_arn
+ local certificate_mode
+ local certificate_status
+ local mqtts_certificate_arn
+ local mqtts_certificate_mode
+ local mqtts_certificate_status
+ local mqtts_hostname
+
+ certificate_mode="$(stack_certificate_mode 2>/dev/null || true)"
+ certificate_arn="$(stack_certificate_arn 2>/dev/null || true)"
+
+ log "Certificate ownership"
+ printf 'Certificate mode: %s\n' "${certificate_mode:-unrecorded}"
+ if [ -n "$certificate_arn" ]; then
+ printf 'Certificate ARN: %s\n' "$certificate_arn"
+ fi
+
+ if { [ "$certificate_mode" = "existing" ] || \
+ [ "$certificate_mode" = "shared" ] || \
+ [ "$certificate_mode" = "managed" ]; } && \
+ [ -n "$certificate_arn" ]; then
+ if certificate_status="$(aws_acm describe-certificate \
+ --certificate-arn "$certificate_arn" \
+ --query 'Certificate.Status' \
+ --output text 2>/dev/null)"; then
+ printf 'Certificate status: %s\n' "$certificate_status"
+ else
+ warn "Unable to read the current status of $certificate_mode ACM certificate '$certificate_arn'."
+ fi
+ fi
+
+ mqtts_certificate_mode="$(stack_mqtts_certificate_mode 2>/dev/null || true)"
+ mqtts_certificate_arn="$(stack_mqtts_certificate_arn 2>/dev/null || true)"
+ mqtts_hostname="$(stack_mqtts_hostname 2>/dev/null || true)"
+ if [ -n "$mqtts_certificate_mode" ] || [ -n "$mqtts_certificate_arn" ] || \
+ [ -n "$mqtts_hostname" ]; then
+ printf 'MQTTS certificate mode: %s\n' "${mqtts_certificate_mode:-unrecorded}"
+ [ -z "$mqtts_hostname" ] || printf 'MQTTS hostname: %s\n' "$mqtts_hostname"
+ [ -z "$mqtts_certificate_arn" ] || printf 'MQTTS certificate ARN: %s\n' "$mqtts_certificate_arn"
+ if { [ "$mqtts_certificate_mode" = existing ] || \
+ [ "$mqtts_certificate_mode" = shared ] || \
+ [ "$mqtts_certificate_mode" = managed ]; } && \
+ [ -n "$mqtts_certificate_arn" ]; then
+ if mqtts_certificate_status="$(aws_acm describe-certificate \
+ --certificate-arn "$mqtts_certificate_arn" \
+ --query 'Certificate.Status' \
+ --output text 2>/dev/null)"; then
+ printf 'MQTTS certificate status: %s\n' "$mqtts_certificate_status"
+ else
+ warn "Unable to read the current status of MQTTS ACM certificate '$mqtts_certificate_arn'."
+ fi
+ fi
+ fi
+}
+
+prepare_mqtts_destroy() {
+ local mqtts_reservation="$1"
+ local recorded_arn
+ local recorded_hostname
+ local recorded_mode
+ local service_certificate_arn
+ local service_hostname
+ local service_managed_dns
+ local service_load_balancer_type
+ local service_port
+ local service_ssl_ports
+ local service_type
+ local validation_endpoint
+
+ [ -n "$mqtts_reservation" ] || return 0
+ [ "$(jq -r '.target + "/" + .dns' <<< "$mqtts_reservation")" = "eks/external-dns" ] || die \
+ "The MQTTS hostname reservation requires external cleanup and or-stack release-hostname."
+ [ "$(jq -r '.exposure' <<< "$mqtts_reservation")" = ingress ] && \
+ [ "$(jq -r '.endpoint' <<< "$mqtts_reservation")" = mqtts ] || die \
+ "The MQTTS hostname reservation is not an EKS Ingress MQTTS endpoint."
+
+ MQTTS_ENABLED=true
+ MQTTS_HOSTNAME_VALUE="$(jq -r '.hostname' <<< "$mqtts_reservation")"
+ normalize_and_validate_mqtts_hostname
+ [ "$MQTTS_HOSTNAME_VALUE" != "$HOSTNAME_VALUE" ] || die \
+ "The web and MQTTS reservations unexpectedly use the same hostname."
+ if [ "$STACK_NAMESPACE_EXISTS" != true ]; then
+ log "Stack namespace '$STACK_NAME' is already absent; recovering retained MQTTS DNS ownership."
+ return
+ fi
+
+ require_reservation_namespace_identity "$mqtts_reservation"
+ recorded_mode="$(stack_mqtts_certificate_mode)" || die \
+ "Unable to read MQTTS certificate mode from namespace '$STACK_NAME'."
+ recorded_arn="$(stack_mqtts_certificate_arn)" || die \
+ "Unable to read MQTTS certificate ARN from namespace '$STACK_NAME'."
+ recorded_hostname="$(stack_mqtts_hostname)" || die \
+ "Unable to read MQTTS hostname from namespace '$STACK_NAME'."
+
+ STACK_MQTTS_SERVICE_JSON="$(kubectl --context "$KUBE_CONTEXT" \
+ --namespace "$STACK_NAME" \
+ get service proxy-mqtts \
+ --ignore-not-found \
+ -o json)" || die \
+ "Unable to inspect Service '$STACK_NAME/proxy-mqtts' before destroy."
+ case "$recorded_mode" in
+ existing|shared|managed) ;;
+ "")
+ # Prepare reserves hostnames before the facade records certificate mode.
+ # An interruption there leaves a valid claim but no MQTTS resources. Allow
+ # cleanup of that state without adopting an unowned deployed endpoint.
+ [ -z "$STACK_MQTTS_SERVICE_JSON" ] && [ -z "$recorded_arn" ] || die \
+ "Stack '$STACK_NAME' has a low-level or unowned Ingress MQTTS endpoint; refusing automated cleanup without a supported openremote.io/mqtts-certificate-mode."
+ [ -z "$recorded_hostname" ] || [ "$recorded_hostname" = "$MQTTS_HOSTNAME_VALUE" ] || die \
+ "Namespace '$STACK_NAME' records MQTTS hostname '$recorded_hostname', but reserves '$MQTTS_HOSTNAME_VALUE'."
+ validation_endpoint="$(kubectl --context "$KUBE_CONTEXT" --namespace "$STACK_NAME" \
+ get dnsendpoint "$ACM_MQTTS_VALIDATION_DNS_ENDPOINT_NAME" \
+ --ignore-not-found -o json)" || die \
+ "Unable to inspect validation resources for the incomplete MQTTS configuration."
+ [ -z "$validation_endpoint" ] || die \
+ "Stack '$STACK_NAME' has MQTTS validation resources without a recorded certificate mode; refusing destructive cleanup."
+ # Normal tagged-certificate recovery and NLB/DNS cleanup still follow;
+ # neither missing mode nor missing Service proves external cleanup.
+ log "Stack '$STACK_NAME' has an incomplete MQTTS prepare; continuing verified cleanup."
+ return
+ ;;
+ *)
+ die "Stack '$STACK_NAME' has a low-level or unowned Ingress MQTTS endpoint; refusing automated cleanup without a supported openremote.io/mqtts-certificate-mode."
+ ;;
+ esac
+ [ "$recorded_hostname" = "$MQTTS_HOSTNAME_VALUE" ] || die \
+ "Namespace '$STACK_NAME' records MQTTS hostname '$recorded_hostname', but reserves '$MQTTS_HOSTNAME_VALUE'."
+ MQTTS_CERTIFICATE_MODE="$recorded_mode"
+ MQTTS_CERTIFICATE_ARN="$recorded_arn"
+
+ [ -n "$STACK_MQTTS_SERVICE_JSON" ] || return 0
+
+ service_managed_dns="$(jq -r '.metadata.labels["openremote.io/managed-dns"] // empty' \
+ <<< "$STACK_MQTTS_SERVICE_JSON")"
+ service_hostname="$(jq -r '.metadata.annotations["external-dns.kubernetes.io/hostname"] // empty' \
+ <<< "$STACK_MQTTS_SERVICE_JSON")"
+ service_certificate_arn="$(jq -r '.metadata.annotations["service.beta.kubernetes.io/aws-load-balancer-ssl-cert"] // empty' \
+ <<< "$STACK_MQTTS_SERVICE_JSON")"
+ service_load_balancer_type="$(jq -r '.metadata.annotations["service.beta.kubernetes.io/aws-load-balancer-type"] // empty' \
+ <<< "$STACK_MQTTS_SERVICE_JSON")"
+ service_ssl_ports="$(jq -r '.metadata.annotations["service.beta.kubernetes.io/aws-load-balancer-ssl-ports"] // empty' \
+ <<< "$STACK_MQTTS_SERVICE_JSON")"
+ service_type="$(jq -r '.spec.type // empty' <<< "$STACK_MQTTS_SERVICE_JSON")"
+ service_port="$(jq -r '.spec.ports[]? | select(.name == "mqtts") | .port' \
+ <<< "$STACK_MQTTS_SERVICE_JSON")"
+ [ "$service_managed_dns" = true ] || die \
+ "Service '$STACK_NAME/proxy-mqtts' is not marked for OpenRemote-managed DNS."
+ [ "$service_hostname" = "$MQTTS_HOSTNAME_VALUE" ] || die \
+ "Service '$STACK_NAME/proxy-mqtts' records hostname '$service_hostname', not '$MQTTS_HOSTNAME_VALUE'."
+ [ "$service_type" = LoadBalancer ] && [ -n "$service_port" ] || die \
+ "Service '$STACK_NAME/proxy-mqtts' is not the expected MQTTS LoadBalancer."
+ [ "$service_load_balancer_type" = external ] && [ "$service_ssl_ports" = mqtts ] || die \
+ "Service '$STACK_NAME/proxy-mqtts' is not configured as the expected ACM-terminated AWS NLB."
+ if [[ ! "$service_certificate_arn" =~ ^arn:[^:]+:acm:[a-z0-9-]+:[0-9]{12}:certificate/[A-Za-z0-9-]+$ ]]; then
+ die "Service '$STACK_NAME/proxy-mqtts' does not carry a valid ACM certificate ARN."
+ fi
+ if [ -n "$recorded_arn" ] && [ "$recorded_arn" != "$service_certificate_arn" ]; then
+ if [ "$recorded_mode" = managed ]; then
+ die "Service '$STACK_NAME/proxy-mqtts' uses certificate '$service_certificate_arn', not owned managed certificate '$recorded_arn'."
+ fi
+ warn "Service '$STACK_NAME/proxy-mqtts' uses certificate '$service_certificate_arn', while the namespace records '$recorded_arn'. This can follow an interrupted rotation; both externally owned certificates will be retained."
+ fi
+ MQTTS_CERTIFICATE_ARN="$service_certificate_arn"
+}
+
+mqtts_load_balancer_exists() {
+ local load_balancers
+ local load_balancer_arns
+ local arn
+ local tags
+ local matches
+
+ # Service status disappears with the namespace (or an earlier uninstall).
+ # Query the controller's AWS identity tags on every destroy attempt so an
+ # interrupted cleanup cannot release reservations while its NLB still exists.
+ load_balancers="$(aws_elbv2 describe-load-balancers --output json 2>&1)" || die \
+ "Unable to inspect MQTTS load balancers before releasing hostname reservations: $load_balancers"
+ load_balancer_arns="$(jq -er '
+ .LoadBalancers | if type != "array" then error("missing load balancers") else
+ [.[] | select(.Type == "network") | .LoadBalancerArn |
+ if type == "string" and length > 0 then . else error("missing NLB ARN") end] |
+ join("\n")
+ end' <<< "$load_balancers")" || die "Unable to parse the AWS load-balancer response."
+ while IFS= read -r arn; do
+ [ -n "$arn" ] || continue
+ if ! tags="$(aws_elbv2 describe-tags --resource-arns "$arn" --output json 2>&1)"; then
+ case "$tags" in
+ # The controller may delete an NLB between the list and tag queries.
+ *'(LoadBalancerNotFound)'*) continue ;;
+ *) die "Unable to inspect load-balancer ownership for '$arn': $tags" ;;
+ esac
+ fi
+ matches="$(jq -er --arg arn "$arn" --arg cluster "$CLUSTER_NAME" \
+ --arg service "$STACK_NAME/proxy-mqtts" '
+ .TagDescriptions |
+ if type != "array" or length != 1 then error("missing tag description") else
+ .[0] | if .ResourceArn != $arn or (.Tags | type) != "array" then
+ error("invalid tag description") else
+ .Tags | from_entries |
+ .["elbv2.k8s.aws/cluster"] == $cluster and
+ .["service.k8s.aws/stack"] == $service
+ end
+ end | tostring' <<< "$tags")" || die "Unable to parse AWS load-balancer ownership for '$arn'."
+ [ "$matches" != true ] || return 0
+ done <<< "$load_balancer_arns"
+ return 1
+}
+
+wait_for_mqtts_load_balancer_removed() {
+ local deadline
+ local next_notice
+ local now
+ local timeout_seconds
+
+ [ "$MQTTS_ENABLED" = true ] && [ "$EXPOSURE" = ingress ] || return 0
+ timeout_seconds="$(duration_seconds "$TIMEOUT")"
+ now="$(date +%s)"
+ deadline="$((now + timeout_seconds))"
+ next_notice="$((now + 30))"
+ log "Waiting for MQTTS NLBs for '$CLUSTER_NAME/$STACK_NAME/proxy-mqtts' to be deleted."
+
+ while [ "$now" -lt "$deadline" ]; do
+ if ! mqtts_load_balancer_exists; then
+ log "No MQTTS NLB remains for the stack."
+ return
+ fi
+ if [ "$now" -ge "$next_notice" ]; then
+ log "The MQTTS NLB is still present."
+ next_notice="$((now + 30))"
+ fi
+ sleep 5
+ now="$(date +%s)"
+ done
+ die "MQTTS NLB deletion for '$CLUSTER_NAME/$STACK_NAME/proxy-mqtts' was not confirmed within '$TIMEOUT'. Hostname reservations and certificate state were retained so destroy can be retried safely."
+}
+
+wait_for_public_hostname_removed() {
+ local description="$2"
+ local hostname="$1"
+ local records
+
+ select_managed_dns_domain "$hostname"
+ discover_authoritative_name_servers
+ records="$(printf '%s\t%s\n' "$hostname" A "$hostname" AAAA "$hostname" CNAME)"
+ wait_for_dns_records_removed "$description" "$records"
+}
+
+destroy_stack_with_managed_resources() {
+ local mqtts_reservation
+ local reservation
+
+ require_command dig
+ require_command helm
+ require_command jq
+ reservation="$(stack_hostname_reservation)" || die "Unable to inspect hostname reservations before destroy."
+ mqtts_reservation="$(stack_mqtts_hostname_reservation)" || die \
+ "Unable to inspect MQTTS hostname reservations before destroy."
+ # Every public EKS stack reserves its hostname before creating external
+ # resources. Never bypass DNS cleanup when that required state is missing.
+ [ -n "$reservation" ] || die \
+ "Stack '$STACK_NAME' has no hostname reservation; refusing EKS destruction without verifiable DNS cleanup state. Inspect the stack and hostname registry before retrying."
+ [ "$(jq -r '.target + "/" + .dns' <<< "$reservation")" = "eks/external-dns" ] || die \
+ "This reservation requires external cleanup and or-stack release-hostname."
+ HOSTNAME_VALUE="$(jq -r '.hostname' <<< "$reservation")"
+ EXPOSURE="$(jq -r '.exposure' <<< "$reservation")"
+ inspect_stack_namespace_for_destroy
+ if [ "$STACK_NAMESPACE_EXISTS" = true ]; then
+ require_reservation_namespace_identity "$reservation"
+ fi
+ MQTTS_ENABLED=false
+ if [ -n "$mqtts_reservation" ]; then
+ prepare_mqtts_destroy "$mqtts_reservation"
+ fi
+ capture_stack_ingress_for_destroy
+
+ resolve_managed_certificate_for_destroy
+ resolve_managed_mqtts_certificate_for_destroy
+ if [ "$MANAGED_MQTTS_CERTIFICATE_DESTROY" = true ] && \
+ [ -z "$mqtts_reservation" ]; then
+ die "Stack '$STACK_NAME' owns a managed MQTTS certificate without an MQTTS hostname reservation; refusing unverifiable DNS cleanup."
+ fi
+
+ if [ "$MANAGED_CERTIFICATE_DESTROY" = true ]; then
+ CERTIFICATE_MODE=managed
+ EXPOSURE=ingress
+ else
+ CERTIFICATE_MODE=""
+ fi
+ if [ "$MANAGED_MQTTS_CERTIFICATE_DESTROY" = true ]; then
+ MQTTS_CERTIFICATE_MODE=managed
+ fi
+ require_managed_external_dns
+ if [ "$MANAGED_CERTIFICATE_DESTROY" = true ]; then
+ validate_stack_ingress_for_managed_destroy
+ capture_validation_dns_records_for_destroy
+ fi
+ if [ "$MANAGED_MQTTS_CERTIFICATE_DESTROY" = true ]; then
+ capture_mqtts_validation_dns_records_for_destroy
+ fi
+
+ # The ALB Ingress can be removed independently, allowing its managed web
+ # certificate to detach before the namespace is deleted. The MQTTS
+ # certificate remains attached to the NLB until Service finalizer cleanup.
+ if [ "$MANAGED_CERTIFICATE_DESTROY" = true ]; then
+ remove_managed_stack_ingress
+ wait_for_public_hostname_removed "$HOSTNAME_VALUE" "stack hostname '$HOSTNAME_VALUE'"
+ wait_for_managed_certificate_detachment
+ fi
+
+ if [ "$STACK_NAMESPACE_EXISTS" = true ]; then
+ run_or_stack "$@"
+ STACK_NAMESPACE_EXISTS=false
+ elif [ "$MANAGED_CERTIFICATE_DESTROY" = true ] || \
+ [ "$MANAGED_MQTTS_CERTIFICATE_DESTROY" = true ]; then
+ log "Stack namespace '$STACK_NAME' is already absent; continuing external cleanup."
+ fi
+
+ wait_for_mqtts_load_balancer_removed
+ if [ "$MANAGED_CERTIFICATE_DESTROY" = false ]; then
+ wait_for_public_hostname_removed "$HOSTNAME_VALUE" "stack hostname '$HOSTNAME_VALUE'"
+ fi
+ if [ -n "$mqtts_reservation" ]; then
+ wait_for_public_hostname_removed "$MQTTS_HOSTNAME_VALUE" \
+ "MQTTS hostname '$MQTTS_HOSTNAME_VALUE'"
+ fi
+
+ # Managed mode assumes each certificate and its validation CNAMEs are
+ # exclusive to its endpoint. ACM can reuse CNAMEs for other certificates;
+ # ownership tags do not prove DNS-record exclusivity. Deleting the namespace
+ # lets ExternalDNS remove both validation DNSEndpoints before ACM deletion.
+ if [ "$MANAGED_MQTTS_CERTIFICATE_DESTROY" = true ]; then
+ wait_for_managed_mqtts_certificate_detachment
+ select_managed_dns_domain "$MQTTS_HOSTNAME_VALUE"
+ discover_authoritative_name_servers
+ wait_for_dns_records_removed "MQTTS ACM validation" \
+ "$MQTTS_VALIDATION_DNS_RECORDS"
+ delete_managed_mqtts_certificate
+ fi
+ if [ "$MANAGED_CERTIFICATE_DESTROY" = true ]; then
+ select_managed_dns_domain "$HOSTNAME_VALUE"
+ discover_authoritative_name_servers
+ wait_for_dns_records_removed "ACM validation" "$VALIDATION_DNS_RECORDS"
+ delete_managed_certificate
+ fi
+ if [ -n "$mqtts_reservation" ]; then
+ release_portable_mqtts_hostname
+ if [ "$MANAGED_MQTTS_CERTIFICATE_DESTROY" = false ] && [ -n "$MQTTS_CERTIFICATE_ARN" ]; then
+ log "Externally owned $MQTTS_CERTIFICATE_MODE MQTTS ACM certificate '$MQTTS_CERTIFICATE_ARN' was retained."
+ fi
+ fi
+ # Release the required web claim last. If its release fails, destroy can
+ # still be retried; releasing it first would strand a remaining optional
+ # MQTTS claim behind the web-reservation preflight.
+ release_portable_hostname
+ if [ "$MANAGED_CERTIFICATE_DESTROY" = true ] || \
+ [ "$MANAGED_MQTTS_CERTIFICATE_DESTROY" = true ]; then
+ log "Stack '$STACK_NAME' and its managed external certificate resources are destroyed."
+ fi
+}
+
+release_portable_hostname() {
+ run_or_stack release-hostname --name "$STACK_NAME" --kube-context "$KUBE_CONTEXT" \
+ --hostname "$HOSTNAME_VALUE" --confirm "$STACK_NAME"
+}
+
+release_portable_mqtts_hostname() {
+ run_or_stack release-hostname --name "$STACK_NAME" --kube-context "$KUBE_CONTEXT" \
+ --hostname "$MQTTS_HOSTNAME_VALUE" --confirm "$STACK_NAME"
+}
+
+delegate_stack_command() {
+ local stack_args=(
+ "$COMMAND"
+ --name "$STACK_NAME"
+ --kube-context "$KUBE_CONTEXT"
+ )
+
+ case "$COMMAND" in
+ uninstall)
+ stack_args+=(--timeout "$TIMEOUT")
+ ;;
+ destroy)
+ stack_args+=(--confirm "$CONFIRM_STACK_NAME" --timeout "$TIMEOUT")
+ ;;
+ esac
+
+ if [ "$COMMAND" = "destroy" ]; then
+ destroy_stack_with_managed_resources "${stack_args[@]}"
+ return
+ fi
+
+ run_or_stack "${stack_args[@]}"
+ if [ "$COMMAND" = "status" ]; then
+ show_certificate_configuration
+ fi
+}
+
+main() {
+ parse_arguments "$@"
+ validate_common_options
+ validate_command_options
+
+ case "$COMMAND" in
+ apply)
+ apply_stack
+ ;;
+ status|credentials|uninstall|destroy)
+ [ -x "$OR_STACK_COMMAND" ] || die \
+ "Portable stack command '$OR_STACK_COMMAND' is missing or not executable."
+ require_eks_target
+ delegate_stack_command
+ ;;
+ esac
+}
+
+if [ "${BASH_SOURCE[0]}" = "$0" ]; then
+ main "$@"
+fi
diff --git a/kubernetes/or-setup/Chart.yaml b/kubernetes/or-setup/Chart.yaml
index 596b9fd41..81534acd7 100644
--- a/kubernetes/or-setup/Chart.yaml
+++ b/kubernetes/or-setup/Chart.yaml
@@ -4,4 +4,4 @@ description: A Helm chart for setting up additional elements required for an OR
type: application
-version: 0.1.0
+version: 0.2.0
diff --git a/kubernetes/or-setup/templates/network-policy.yaml b/kubernetes/or-setup/templates/network-policy.yaml
new file mode 100644
index 000000000..cf8382e26
--- /dev/null
+++ b/kubernetes/or-setup/templates/network-policy.yaml
@@ -0,0 +1,18 @@
+{{- if .Values.networkPolicy.enabled }}
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+ name: openremote-stack-ingress
+spec:
+ podSelector: {}
+ policyTypes:
+ - Ingress
+ ingress:
+ - from:
+ # A podSelector without a namespaceSelector selects only Pods in this
+ # NetworkPolicy's namespace.
+ - podSelector: {}
+ {{- with .Values.networkPolicy.additionalIngressFrom }}
+ {{- toYaml . | nindent 8 }}
+ {{- end }}
+{{- end }}
diff --git a/kubernetes/or-setup/templates/openremote-manager-pv.yaml b/kubernetes/or-setup/templates/openremote-manager-pv.yaml
deleted file mode 100644
index 1e1c0f6b9..000000000
--- a/kubernetes/or-setup/templates/openremote-manager-pv.yaml
+++ /dev/null
@@ -1,23 +0,0 @@
-apiVersion: v1
-kind: PersistentVolume
-metadata:
- name: manager-data-pv
-{{- if .Values.aws.enabled }}
- labels:
- type: amazonEBS
-{{- end }}
-spec:
- capacity:
- storage: {{ .Values.managerVolumeSize }}
- accessModes:
- - ReadWriteOnce
- {{- if .Values.aws.enabled }}
- awsElasticBlockStore:
- volumeID: {{ .Values.aws.managerVolumeId }}
- fsType: ext4
- {{- else }}
- hostPath:
- path: "{{ .Values.basePath}}/manager-data"
- {{- end }}
- persistentVolumeReclaimPolicy: Retain
- storageClassName: manual
diff --git a/kubernetes/or-setup/templates/openremote-psql-pv.yaml b/kubernetes/or-setup/templates/openremote-psql-pv.yaml
deleted file mode 100644
index 63e70c6b8..000000000
--- a/kubernetes/or-setup/templates/openremote-psql-pv.yaml
+++ /dev/null
@@ -1,24 +0,0 @@
-apiVersion: v1
-kind: PersistentVolume
-metadata:
- name: postgresql-data-pv
- {{- if .Values.aws.enabled }}
- labels:
- type: amazonEBS
- {{- end }}
-
-spec:
- capacity:
- storage: {{ .Values.psqlVolumeSize }}
- accessModes:
- - ReadWriteOnce
- {{- if .Values.aws.enabled }}
- awsElasticBlockStore:
- volumeID: {{ .Values.aws.psqlVolumeId }}
- fsType: ext4
- {{- else }}
- hostPath:
- path: "{{ .Values.basePath}}/postgresql-data"
- {{- end }}
- persistentVolumeReclaimPolicy: Retain
- storageClassName: manual
diff --git a/kubernetes/or-setup/templates/openremote-secret.yaml b/kubernetes/or-setup/templates/openremote-secret.yaml
index 8cb29c8b0..054fab205 100644
--- a/kubernetes/or-setup/templates/openremote-secret.yaml
+++ b/kubernetes/or-setup/templates/openremote-secret.yaml
@@ -1,9 +1,44 @@
+{{- $secretName := "openremote-secret" -}}
+{{- $existingSecret := lookup "v1" "Secret" .Release.Namespace $secretName -}}
+{{- $adminPassword := default "" .Values.credentials.adminPassword -}}
+{{- $postgresqlUsername := default "" .Values.credentials.postgresqlUsername -}}
+{{- $postgresqlPassword := default "" .Values.credentials.postgresqlPassword -}}
+{{- if $existingSecret -}}
+ {{- $existingData := required "existing openremote-secret has no data" $existingSecret.data -}}
+ {{- $existingAdminPassword := required "existing openremote-secret is missing admin-password" (index $existingData "admin-password") -}}
+ {{- $existingPostgresqlUsername := required "existing openremote-secret is missing postgresql-username" (index $existingData "postgresql-username") -}}
+ {{- $existingPostgresqlPassword := required "existing openremote-secret is missing postgresql-password" (index $existingData "postgresql-password") -}}
+ {{- if and $adminPassword (ne $adminPassword ($existingAdminPassword | b64dec)) -}}
+ {{- fail "credentials.adminPassword differs from the existing Secret; credential rotation is not supported by apply" -}}
+ {{- end -}}
+ {{- if and $postgresqlUsername (ne $postgresqlUsername ($existingPostgresqlUsername | b64dec)) -}}
+ {{- fail "credentials.postgresqlUsername differs from the existing Secret; database credential rotation is not supported by apply" -}}
+ {{- end -}}
+ {{- if and $postgresqlPassword (ne $postgresqlPassword ($existingPostgresqlPassword | b64dec)) -}}
+ {{- fail "credentials.postgresqlPassword differs from the existing Secret; database credential rotation is not supported by apply" -}}
+ {{- end -}}
+ {{- $adminPassword = $existingAdminPassword | b64dec -}}
+ {{- $postgresqlUsername = $existingPostgresqlUsername | b64dec -}}
+ {{- $postgresqlPassword = $existingPostgresqlPassword | b64dec -}}
+{{- else -}}
+ {{- if not $adminPassword -}}
+ {{- $adminPassword = randAlphaNum 32 -}}
+ {{- end -}}
+ {{- if not $postgresqlUsername -}}
+ {{- $postgresqlUsername = "postgres" -}}
+ {{- end -}}
+ {{- if not $postgresqlPassword -}}
+ {{- $postgresqlPassword = randAlphaNum 32 -}}
+ {{- end -}}
+{{- end -}}
apiVersion: v1
kind: Secret
metadata:
- name: openremote-secret
+ name: {{ $secretName }}
+ annotations:
+ helm.sh/resource-policy: keep
type: Opaque
data:
- admin-password: c2VjcmV0
- postgresql-username: cG9zdGdyZXM=
- postgresql-password: cG9zdGdyZXM=
+ admin-password: {{ $adminPassword | b64enc | quote }}
+ postgresql-username: {{ $postgresqlUsername | b64enc | quote }}
+ postgresql-password: {{ $postgresqlPassword | b64enc | quote }}
diff --git a/kubernetes/or-setup/templates/proxy-network-policy.yaml b/kubernetes/or-setup/templates/proxy-network-policy.yaml
new file mode 100644
index 000000000..cd13c2a37
--- /dev/null
+++ b/kubernetes/or-setup/templates/proxy-network-policy.yaml
@@ -0,0 +1,20 @@
+{{- if and .Values.networkPolicy.enabled .Values.networkPolicy.proxy.enabled }}
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+ name: openremote-proxy-ingress
+spec:
+ podSelector:
+ {{- toYaml .Values.networkPolicy.proxy.podSelector | nindent 4 }}
+ policyTypes:
+ - Ingress
+ ingress:
+ # Omitting `from` permits load-balancer and local host traffic while this
+ # rule remains limited to HAProxy's enabled public container ports.
+ - ports:
+ {{- toYaml .Values.networkPolicy.proxy.ports | nindent 8 }}
+ {{- if .Values.networkPolicy.proxy.mqtts.enabled }}
+ - protocol: TCP
+ port: {{ .Values.networkPolicy.proxy.mqtts.port }}
+ {{- end }}
+{{- end }}
diff --git a/kubernetes/or-setup/values.yaml b/kubernetes/or-setup/values.yaml
index 1aa37572e..8a2d60109 100644
--- a/kubernetes/or-setup/values.yaml
+++ b/kubernetes/or-setup/values.yaml
@@ -1,14 +1,37 @@
-managerVolumeSize: "1Gi"
-psqlVolumeSize: "1Gi"
+# OpenRemote component charts own their namespaced PersistentVolumeClaims.
+# This chart provides the namespaced shared secret and optional network policies.
-# Configuration when deploying under EKS in AWS
-aws:
- # Set enabled to true if deployed under AWS
- enabled: false
- # Id of the EBS volume used for manager persistent storage
- managerVolumeId: vol-xxx
- # Id of the EBS volume used for postgreSQL persistent storage
- psqlVolumeId: vol-xxx
+credentials:
+ # Empty passwords are generated when the Secret is first created. Existing
+ # Secret values are always retained; apply does not rotate live credentials.
+ adminPassword: ""
+ # Empty defaults to postgres when the Secret is first created.
+ postgresqlUsername: ""
+ postgresqlPassword: ""
-# Root path for hostpath storage when deployed locally. On macOS, this needs to be under user's home folder
-basePath: "/Users/username/openremote"
+networkPolicy:
+ # or-stack target profiles enable this. It remains opt-in for standalone and
+ # legacy chart installations, whose external routing may require extra peers.
+ # The cluster CNI must implement NetworkPolicy for it to be enforced.
+ enabled: false
+ # Optional NetworkPolicyPeer entries for trusted ingress sources outside the
+ # stack namespace.
+ additionalIngressFrom: []
+ # Allow external traffic only to the per-stack proxy Pod. In HAProxy mode it
+ # is the public TLS endpoint; in Ingress mode it is the internal gateway
+ # targeted by the Ingress controller. Manager and Keycloak remain private.
+ proxy:
+ enabled: false
+ podSelector:
+ matchLabels:
+ app.kubernetes.io/name: proxy
+ ports:
+ - protocol: TCP
+ port: 8080
+ - protocol: TCP
+ port: 8443
+ # HAProxy MQTTS is opt-in. This is the proxy container listener, not the
+ # configurable public Service port.
+ mqtts:
+ enabled: false
+ port: 8883
diff --git a/kubernetes/or-stack b/kubernetes/or-stack
new file mode 100755
index 000000000..b02a93c7b
--- /dev/null
+++ b/kubernetes/or-stack
@@ -0,0 +1,1165 @@
+#!/usr/bin/env bash
+
+# OpenRemote stack lifecycle only; cluster infrastructure is managed separately.
+
+set -Eeuo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+KUBERNETES_DIR="$SCRIPT_DIR"
+TARGETS_DIR="$KUBERNETES_DIR/stack/targets"
+EXPOSURES_DIR="$KUBERNETES_DIR/stack/exposures"
+source "$SCRIPT_DIR/lib/hostname-reservations"
+
+DEFAULT_TIMEOUT="10m"
+PROXY_MQTTS_TLS_PORT="8883"
+PROXY_MQTT_EDGE_PORT="1883"
+
+COMMAND=""
+STACK_NAME="${OR_STACK_NAME:-}"
+KUBE_CONTEXT="${OR_STACK_KUBE_CONTEXT:-}"
+TARGET="${OR_STACK_TARGET:-}"
+EXPOSURE="${OR_STACK_EXPOSURE:-haproxy}"
+DNS_MODE="${OR_STACK_DNS:-none}"
+HOSTNAME_VALUE="${OR_STACK_HOSTNAME:-}"
+CERTIFICATE_ARN="${OR_STACK_CERTIFICATE_ARN:-}"
+VALUES_DIR="${OR_STACK_VALUES_DIR:-}"
+TIMEOUT="${OR_STACK_TIMEOUT:-$DEFAULT_TIMEOUT}"
+HTTP_PORT="${OR_STACK_HTTP_PORT:-80}"
+HTTPS_PORT="${OR_STACK_HTTPS_PORT:-443}"
+MQTTS_ENABLED="${OR_STACK_MQTTS:-false}"
+MQTTS_PORT="${OR_STACK_MQTTS_PORT:-8883}"
+MQTTS_HOSTNAME_VALUE="${OR_STACK_MQTTS_HOSTNAME:-}"
+MQTTS_CERTIFICATE_ARN="${OR_STACK_MQTTS_CERTIFICATE_ARN:-}"
+HTTP_PORT_CONFIGURED=false
+HTTPS_PORT_CONFIGURED=false
+MQTTS_PORT_CONFIGURED=false
+CLUSTER_DNS=""
+CONFIRM_STACK_NAME=""
+
+if [ "${OR_STACK_HTTP_PORT+x}" = "x" ]; then
+ HTTP_PORT_CONFIGURED=true
+fi
+if [ "${OR_STACK_HTTPS_PORT+x}" = "x" ]; then
+ HTTPS_PORT_CONFIGURED=true
+fi
+if [ "${OR_STACK_MQTTS_PORT+x}" = "x" ]; then
+ MQTTS_PORT_CONFIGURED=true
+fi
+
+usage() {
+ cat <<'EOF'
+Manage an OpenRemote stack in its own Kubernetes namespace.
+
+Usage:
+ or-stack prepare --name <name> --kube-context <context> --target <local|eks> [options]
+ or-stack apply --name <name> --kube-context <context> --target <local|eks> [options]
+ or-stack status --name <name> --kube-context <context>
+ or-stack credentials --name <name> --kube-context <context>
+ or-stack uninstall --name <name> --kube-context <context> [options]
+ or-stack destroy --name <name> --kube-context <context> --confirm <name> [options]
+ or-stack release-hostname --name <name> --kube-context <context> --hostname <host> --confirm <name>
+
+Commands:
+ prepare Create or verify the owned stack namespace without
+ installing any workloads.
+ apply Create the stack namespace and install or upgrade the
+ internal OpenRemote components.
+ status Show releases and namespaced stack resources.
+ credentials Show the Manager administrator login credentials.
+ uninstall Uninstall stack releases while retaining the namespace,
+ credentials, PVCs, and data.
+ destroy Uninstall stack releases and delete its namespace,
+ credentials, PVCs, and data after exact-name confirmation.
+ release-hostname Release a retained hostname after namespace deletion.
+ Exact-name confirmation asserts external cleanup is complete.
+
+Options:
+ --name <name> Stack name and Kubernetes namespace. Env: OR_STACK_NAME.
+ --kube-context <c> Kubernetes context to target. Env: OR_STACK_KUBE_CONTEXT.
+ --target <target> Target profile: local or eks. Env: OR_STACK_TARGET.
+ Required by prepare and apply.
+ --exposure <mode> Public exposure: haproxy, ingress, or none. Env:
+ OR_STACK_EXPOSURE. Default: haproxy for every target.
+ --dns <mode> DNS ownership: none or external-dns. Env: OR_STACK_DNS.
+ Default: none.
+ --hostname <host> Public hostname passed to Manager and Keycloak.
+ Env: OR_STACK_HOSTNAME. Required for public EKS
+ exposure; otherwise defaults to localhost.
+ --certificate-arn <arn>
+ ACM certificate ARN for an EKS ALB listener. Env:
+ OR_STACK_CERTIFICATE_ARN. Required only for EKS with
+ ingress exposure.
+ --http-port <port> Public HAProxy Service HTTP port. Env:
+ OR_STACK_HTTP_PORT. Default: 80.
+ --https-port <port> Public HAProxy Service HTTPS port and forwarded port.
+ Env: OR_STACK_HTTPS_PORT. Default: 443.
+ --mqtts Expose MQTTS through the stack proxy gateway. HAProxy
+ exposure reuses the web hostname and certificate. EKS
+ Ingress exposure requires the two MQTTS options below.
+ Env: OR_STACK_MQTTS. Default: false. Plaintext MQTT is
+ not exposed publicly.
+ --mqtts-port <port> Public MQTTS Service port. Env:
+ OR_STACK_MQTTS_PORT. Default: 8883. Requires --mqtts.
+ --mqtts-hostname <host>
+ Dedicated hostname for the per-stack MQTTS NLB. Env:
+ OR_STACK_MQTTS_HOSTNAME. Required only for EKS Ingress.
+ --mqtts-certificate-arn <arn>
+ ACM certificate ARN for the per-stack MQTTS NLB TLS
+ listener. Env: OR_STACK_MQTTS_CERTIFICATE_ARN. Required
+ only for EKS Ingress.
+ --values-dir <path> Optional directory containing per-component overrides:
+ or-setup.yaml, postgresql.yaml, keycloak.yaml, and
+ manager.yaml, plus proxy.yaml for either public
+ exposure.
+ Env: OR_STACK_VALUES_DIR.
+ --timeout <duration> Helm wait timeout. Env: OR_STACK_TIMEOUT. Default: 10m.
+ --confirm <name> Required exact stack name for destroy or release-hostname.
+ -h, --help Show this help.
+
+The stack name is also its namespace. Helm release names remain stable within
+that namespace so component service discovery continues to use the names
+postgresql, keycloak, and manager.
+
+The prepare command establishes namespace ownership, records the target,
+exposure and DNS choices, and reserves public hostnames. It refuses to adopt an
+unrelated namespace. The uninstall command retains chart-managed credentials
+and data and leaves the stack namespace in place. The destroy command is the
+explicit data-purge operation and deletes every resource in the stack
+namespace. Lifecycle and credential commands refuse namespaces without the
+matching openremote.io/stack label.
+
+HAProxy is the default exposure for local and EKS targets. On EKS it creates a
+per-stack internet-facing NLB; locally it uses the cluster's LoadBalancer
+support. Select ingress explicitly to route through a private per-stack HAProxy
+gateway (the EKS profile exposes it through the cluster's shared OpenRemote
+ALB), or none for an internal-only stack. With --dns external-dns, this command
+labels and annotates the stack's public routing resource for a separately
+installed ExternalDNS controller. It never calls a DNS provider API itself and
+does not manage ACM certificates.
+
+Public EKS, Ingress, and ExternalDNS hostnames are exclusively reserved in
+ConfigMap kube-system/openremote-hostnames before certificates or workloads.
+This prevents cross-stack routing conflicts, including concurrent applies.
+Requires jq and permission to get/create/update this ConfigMap and inspect
+namespaces, Ingresses, and Services cluster-wide. Hostname changes are rejected.
+Reservations survive uninstall and failed operations. EKS destroy through the
+facade releases them after external cleanup. Portable destroy retains them for
+explicit release, except local Ingress without managed DNS. Local HAProxy with
+unmanaged DNS can still share localhost on different ports; new internal stacks
+reserve nothing. Disabling ExternalDNS does not clear a retained reservation.
+EOF
+}
+
+log() {
+ printf '[or-stack] %s\n' "$*"
+}
+
+die() {
+ printf '[or-stack] ERROR: %s\n' "$*" >&2
+ exit 1
+}
+
+require_command() {
+ command -v "$1" >/dev/null 2>&1 || die "Required command '$1' was not found in PATH."
+}
+
+parse_arguments() {
+ if [ "$#" -eq 0 ]; then
+ usage >&2
+ exit 1
+ fi
+
+ case "$1" in
+ -h|--help)
+ usage
+ exit 0
+ ;;
+ prepare|apply|status|credentials|uninstall|destroy|release-hostname)
+ COMMAND="$1"
+ shift
+ ;;
+ *)
+ die "Unknown command '$1'. Run with --help for usage."
+ ;;
+ esac
+
+ while [ "$#" -gt 0 ]; do
+ case "$1" in
+ --name)
+ [ "$#" -ge 2 ] || die "Option --name requires a value."
+ STACK_NAME="$2"
+ shift 2
+ ;;
+ --kube-context)
+ [ "$#" -ge 2 ] || die "Option --kube-context requires a value."
+ KUBE_CONTEXT="$2"
+ shift 2
+ ;;
+ --target)
+ [ "$#" -ge 2 ] || die "Option --target requires a value."
+ TARGET="$2"
+ shift 2
+ ;;
+ --exposure)
+ [ "$#" -ge 2 ] || die "Option --exposure requires a value."
+ EXPOSURE="$2"
+ shift 2
+ ;;
+ --dns)
+ [ "$#" -ge 2 ] || die "Option --dns requires a value."
+ DNS_MODE="$2"
+ shift 2
+ ;;
+ --hostname)
+ [ "$#" -ge 2 ] || die "Option --hostname requires a value."
+ HOSTNAME_VALUE="$2"
+ shift 2
+ ;;
+ --certificate-arn)
+ [ "$#" -ge 2 ] || die "Option --certificate-arn requires a value."
+ CERTIFICATE_ARN="$2"
+ shift 2
+ ;;
+ --http-port)
+ [ "$#" -ge 2 ] || die "Option --http-port requires a value."
+ HTTP_PORT="$2"
+ HTTP_PORT_CONFIGURED=true
+ shift 2
+ ;;
+ --https-port)
+ [ "$#" -ge 2 ] || die "Option --https-port requires a value."
+ HTTPS_PORT="$2"
+ HTTPS_PORT_CONFIGURED=true
+ shift 2
+ ;;
+ --mqtts)
+ MQTTS_ENABLED=true
+ shift
+ ;;
+ --mqtts-port)
+ [ "$#" -ge 2 ] || die "Option --mqtts-port requires a value."
+ MQTTS_PORT="$2"
+ MQTTS_PORT_CONFIGURED=true
+ shift 2
+ ;;
+ --mqtts-hostname)
+ [ "$#" -ge 2 ] || die "Option --mqtts-hostname requires a value."
+ MQTTS_HOSTNAME_VALUE="$2"
+ shift 2
+ ;;
+ --mqtts-certificate-arn)
+ [ "$#" -ge 2 ] || die "Option --mqtts-certificate-arn requires a value."
+ MQTTS_CERTIFICATE_ARN="$2"
+ shift 2
+ ;;
+ --values-dir)
+ [ "$#" -ge 2 ] || die "Option --values-dir requires a value."
+ VALUES_DIR="$2"
+ shift 2
+ ;;
+ --timeout)
+ [ "$#" -ge 2 ] || die "Option --timeout requires a value."
+ TIMEOUT="$2"
+ shift 2
+ ;;
+ --confirm)
+ [ "$#" -ge 2 ] || die "Option --confirm requires a value."
+ CONFIRM_STACK_NAME="$2"
+ shift 2
+ ;;
+ -h|--help)
+ usage
+ exit 0
+ ;;
+ *)
+ die "Unknown option '$1'. Run with --help for usage."
+ ;;
+ esac
+ done
+}
+
+validate_stack_name() {
+ [ -n "$STACK_NAME" ] || die "A stack name is required; use --name."
+ [ "${#STACK_NAME}" -le 63 ] || die "Stack name must contain at most 63 characters."
+
+ if [[ ! "$STACK_NAME" =~ ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ ]]; then
+ die "Stack name must be a lowercase Kubernetes namespace name."
+ fi
+
+ case "$STACK_NAME" in
+ default|kube-system|kube-public|kube-node-lease)
+ die "Namespace '$STACK_NAME' is reserved and cannot be used as a stack name."
+ ;;
+ esac
+}
+
+validate_common_options() {
+ validate_stack_name
+ [ -n "$KUBE_CONTEXT" ] || die "A Kubernetes context is required; use --kube-context."
+}
+
+validate_apply_options() {
+ validate_target
+ validate_exposure
+ validate_dns_mode
+
+ validate_hostname_option
+
+ case "$MQTTS_ENABLED" in
+ true|false)
+ ;;
+ *)
+ die "OR_STACK_MQTTS must be true or false."
+ ;;
+ esac
+
+ if [ "$MQTTS_ENABLED" != true ] && [ "$MQTTS_PORT_CONFIGURED" = true ]; then
+ die "Option --mqtts-port requires --mqtts."
+ fi
+ if [ "$MQTTS_ENABLED" != true ] && { [ -n "$MQTTS_HOSTNAME_VALUE" ] || [ -n "$MQTTS_CERTIFICATE_ARN" ]; }; then
+ die "Options --mqtts-hostname and --mqtts-certificate-arn require --mqtts."
+ fi
+
+ if [ "$TARGET" = "eks" ] && [ "$EXPOSURE" = "ingress" ]; then
+ [ -n "$CERTIFICATE_ARN" ] || die \
+ "An ACM certificate ARN is required for EKS ingress exposure; use --certificate-arn or OR_STACK_CERTIFICATE_ARN."
+ validate_acm_certificate_arn "$CERTIFICATE_ARN" "Certificate"
+ elif [ -n "$CERTIFICATE_ARN" ]; then
+ die "Option --certificate-arn is only valid with --target eks --exposure ingress."
+ fi
+
+ if [ "$EXPOSURE" = "haproxy" ]; then
+ [ -z "$MQTTS_HOSTNAME_VALUE" ] || die \
+ "Option --mqtts-hostname is only valid with EKS Ingress exposure."
+ [ -z "$MQTTS_CERTIFICATE_ARN" ] || die \
+ "Option --mqtts-certificate-arn is only valid with EKS Ingress exposure."
+ validate_port "$HTTP_PORT" "HTTP"
+ validate_port "$HTTPS_PORT" "HTTPS"
+ [ "$HTTP_PORT" != "$HTTPS_PORT" ] || die "HAProxy HTTP and HTTPS ports must be different."
+ if [ "$MQTTS_ENABLED" = true ]; then
+ validate_port "$MQTTS_PORT" "MQTTS"
+ [ "$MQTTS_PORT" != "$HTTP_PORT" ] && [ "$MQTTS_PORT" != "$HTTPS_PORT" ] || die \
+ "HAProxy HTTP, HTTPS, and MQTTS ports must be different."
+ fi
+ elif [ "$EXPOSURE" = "ingress" ]; then
+ if [ "$HTTP_PORT_CONFIGURED" = true ] || [ "$HTTPS_PORT_CONFIGURED" = true ]; then
+ die "Options --http-port and --https-port are only valid with HAProxy exposure."
+ fi
+ if [ "$MQTTS_ENABLED" = true ]; then
+ [ "$TARGET" = "eks" ] || die "Ingress MQTTS is supported only with the EKS target."
+ validate_mqtts_hostname_option
+ [ "$MQTTS_HOSTNAME_VALUE" != "$HOSTNAME_VALUE" ] || die \
+ "Ingress web and MQTTS hostnames must be different because they target different load balancers."
+ [ -n "$MQTTS_CERTIFICATE_ARN" ] || die \
+ "An ACM certificate ARN is required for EKS Ingress MQTTS; use --mqtts-certificate-arn or OR_STACK_MQTTS_CERTIFICATE_ARN."
+ validate_acm_certificate_arn "$MQTTS_CERTIFICATE_ARN" "MQTTS certificate"
+ validate_port "$MQTTS_PORT" "MQTTS"
+ fi
+ else
+ [ "$MQTTS_ENABLED" != true ] || die "Internal-only exposure cannot enable MQTTS."
+ [ -z "$MQTTS_HOSTNAME_VALUE" ] || die \
+ "Option --mqtts-hostname is only valid with EKS Ingress exposure."
+ [ -z "$MQTTS_CERTIFICATE_ARN" ] || die \
+ "Option --mqtts-certificate-arn is only valid with EKS Ingress exposure."
+ if [ "$HTTP_PORT_CONFIGURED" = true ] || [ "$HTTPS_PORT_CONFIGURED" = true ]; then
+ die "Options --http-port and --https-port are only valid with HAProxy exposure."
+ fi
+ fi
+
+ if [ "$DNS_MODE" = "external-dns" ]; then
+ [ "$EXPOSURE" != "none" ] || die \
+ "ExternalDNS ownership requires haproxy or ingress exposure."
+ [ "$HOSTNAME_VALUE" != "localhost" ] && [[ "$HOSTNAME_VALUE" == *.* ]] || die \
+ "ExternalDNS ownership requires a non-local, fully qualified hostname."
+ fi
+ validate_timeout
+
+ if [ -n "$VALUES_DIR" ] && [ ! -d "$VALUES_DIR" ]; then
+ die "Values directory '$VALUES_DIR' does not exist."
+ fi
+}
+
+validate_hostname_option() {
+ HOSTNAME_VALUE="$(printf '%s' "$HOSTNAME_VALUE" | tr '[:upper:]' '[:lower:]')"
+ HOSTNAME_VALUE="${HOSTNAME_VALUE%.}"
+ if [ -z "$HOSTNAME_VALUE" ]; then
+ if [ "$TARGET" = "local" ] || [ "$EXPOSURE" = "none" ]; then
+ HOSTNAME_VALUE=localhost
+ else
+ die "A public hostname is required for public EKS exposure; use --hostname or OR_STACK_HOSTNAME."
+ fi
+ fi
+
+ [ -n "$HOSTNAME_VALUE" ] || die "Hostname cannot be empty."
+ [ "${#HOSTNAME_VALUE}" -le 253 ] || die "Hostname must contain at most 253 characters."
+ if [[ ! "$HOSTNAME_VALUE" =~ ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ ]]; then
+ die "Hostname must be a lowercase DNS name."
+ fi
+}
+
+validate_mqtts_hostname_option() {
+ MQTTS_HOSTNAME_VALUE="$(printf '%s' "$MQTTS_HOSTNAME_VALUE" | tr '[:upper:]' '[:lower:]')"
+ MQTTS_HOSTNAME_VALUE="${MQTTS_HOSTNAME_VALUE%.}"
+ [ -n "$MQTTS_HOSTNAME_VALUE" ] || die \
+ "A dedicated MQTTS hostname is required for EKS Ingress exposure; use --mqtts-hostname or OR_STACK_MQTTS_HOSTNAME."
+ [ "${#MQTTS_HOSTNAME_VALUE}" -le 253 ] || die \
+ "MQTTS hostname must contain at most 253 characters."
+ if [[ ! "$MQTTS_HOSTNAME_VALUE" =~ ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)+$ ]]; then
+ die "MQTTS hostname must be a lowercase, fully qualified DNS name."
+ fi
+}
+
+validate_acm_certificate_arn() {
+ local arn="$1"
+ local label="$2"
+ if [[ ! "$arn" =~ ^arn:[^:]+:acm:[a-z0-9-]+:[0-9]{12}:certificate/[A-Za-z0-9-]+$ ]]; then
+ die "$label ARN must be a valid ACM certificate ARN."
+ fi
+}
+
+validate_prepare_options() {
+ validate_target
+ validate_exposure
+ validate_dns_mode
+ validate_hostname_option
+
+ case "$MQTTS_ENABLED" in
+ true|false)
+ ;;
+ *)
+ die "OR_STACK_MQTTS must be true or false."
+ ;;
+ esac
+ if [ "$MQTTS_ENABLED" = true ] && [ "$EXPOSURE" = "ingress" ]; then
+ [ "$TARGET" = "eks" ] || die "Ingress MQTTS is supported only with the EKS target."
+ validate_mqtts_hostname_option
+ [ "$MQTTS_HOSTNAME_VALUE" != "$HOSTNAME_VALUE" ] || die \
+ "Ingress web and MQTTS hostnames must be different because they target different load balancers."
+ elif [ -n "$MQTTS_HOSTNAME_VALUE" ] || [ -n "$MQTTS_CERTIFICATE_ARN" ]; then
+ die "Options --mqtts-hostname and --mqtts-certificate-arn are only valid with EKS Ingress MQTTS."
+ elif [ "$MQTTS_ENABLED" = true ] && [ "$EXPOSURE" = "none" ]; then
+ die "Internal-only exposure cannot enable MQTTS."
+ fi
+
+ if [ "$DNS_MODE" = "external-dns" ] && [ "$EXPOSURE" = "none" ]; then
+ die "ExternalDNS ownership requires haproxy or ingress exposure."
+ fi
+ if [ "$DNS_MODE" = "external-dns" ]; then
+ [ "$HOSTNAME_VALUE" != localhost ] && [[ "$HOSTNAME_VALUE" == *.* ]] || die \
+ "ExternalDNS ownership requires a non-local, fully qualified hostname."
+ fi
+}
+
+validate_port() {
+ local port="$1"
+ local name="$2"
+
+ [[ "$port" =~ ^[0-9]+$ ]] || die "$name port must be a number from 1 to 65535."
+ [ "$port" -ge 1 ] && [ "$port" -le 65535 ] || die \
+ "$name port must be a number from 1 to 65535."
+}
+
+validate_timeout() {
+ [ -n "$TIMEOUT" ] || die "Timeout cannot be empty."
+}
+
+validate_destroy_options() {
+ validate_timeout
+ [ "$CONFIRM_STACK_NAME" = "$STACK_NAME" ] || die \
+ "Destroy requires --confirm '$STACK_NAME'."
+}
+
+validate_target() {
+ [ -n "$TARGET" ] || die "A target is required for prepare and apply; use --target local or --target eks."
+
+ case "$TARGET" in
+ local|eks)
+ ;;
+ *)
+ die "Unsupported target '$TARGET'; expected local or eks."
+ ;;
+ esac
+}
+
+validate_exposure() {
+ case "$EXPOSURE" in
+ haproxy|ingress|none)
+ ;;
+ *)
+ die "Unsupported exposure '$EXPOSURE'; expected haproxy, ingress, or none."
+ ;;
+ esac
+}
+
+validate_dns_mode() {
+ case "$DNS_MODE" in
+ none|external-dns)
+ ;;
+ *)
+ die "Unsupported DNS mode '$DNS_MODE'; expected none or external-dns."
+ ;;
+ esac
+}
+
+require_kube_context() {
+ local configured_context
+
+ configured_context="$(kubectl config get-contexts "$KUBE_CONTEXT" -o name 2>/dev/null || true)"
+ [ "$configured_context" = "$KUBE_CONTEXT" ] || die \
+ "Kubernetes context '$KUBE_CONTEXT' does not exist."
+}
+
+stack_namespace_label() {
+ kubectl --context "$KUBE_CONTEXT" get namespace "$STACK_NAME" \
+ -o 'jsonpath={.metadata.labels.openremote\.io/stack}'
+}
+
+stack_namespace_target() {
+ kubectl --context "$KUBE_CONTEXT" get namespace "$STACK_NAME" \
+ -o 'jsonpath={.metadata.labels.openremote\.io/target}'
+}
+
+stack_namespace_exposure() {
+ kubectl --context "$KUBE_CONTEXT" get namespace "$STACK_NAME" \
+ -o 'jsonpath={.metadata.labels.openremote\.io/exposure}'
+}
+
+stack_namespace_dns() {
+ kubectl --context "$KUBE_CONTEXT" get namespace "$STACK_NAME" \
+ -o 'jsonpath={.metadata.labels.openremote\.io/dns}'
+}
+
+stack_namespace_hostname() {
+ kubectl --context "$KUBE_CONTEXT" get namespace "$STACK_NAME" \
+ -o 'jsonpath={.metadata.annotations.openremote\.io/hostname}'
+}
+
+stack_namespace_mqtts_hostname() {
+ kubectl --context "$KUBE_CONTEXT" get namespace "$STACK_NAME" \
+ -o 'jsonpath={.metadata.annotations.openremote\.io/mqtts-hostname}'
+}
+
+ensure_namespace() {
+ local stack_label
+ local stored_exposure
+ local stored_target
+
+ if stack_label="$(stack_namespace_label 2>/dev/null)"; then
+ [ "$stack_label" = "$STACK_NAME" ] || die \
+ "Namespace '$STACK_NAME' already exists without the matching openremote.io/stack label; refusing to adopt it."
+
+ stored_target="$(stack_namespace_target)"
+ [ -n "$stored_target" ] || die "Stack '$STACK_NAME' has no recorded target; inspect its namespace metadata before continuing."
+ if [ "$stored_target" != "$TARGET" ]; then
+ die "Stack '$STACK_NAME' uses target '$stored_target'; refusing to change it to '$TARGET'."
+ fi
+
+ stored_exposure="$(stack_namespace_exposure)"
+ [ -n "$stored_exposure" ] || die "Stack '$STACK_NAME' has no recorded exposure; inspect its namespace metadata before continuing."
+ if [ "$stored_exposure" != "$EXPOSURE" ]; then
+ die "Stack '$STACK_NAME' uses exposure '$stored_exposure'; refusing to change it to '$EXPOSURE'."
+ fi
+ log "Using namespace '$STACK_NAME'."
+ else
+ log "Creating namespace '$STACK_NAME'."
+ kubectl --context "$KUBE_CONTEXT" create namespace "$STACK_NAME"
+ fi
+
+ kubectl --context "$KUBE_CONTEXT" label namespace "$STACK_NAME" \
+ app.kubernetes.io/part-of=openremote \
+ "openremote.io/stack=$STACK_NAME" \
+ "openremote.io/target=$TARGET" \
+ "openremote.io/exposure=$EXPOSURE" \
+ "openremote.io/dns=$DNS_MODE" \
+ --overwrite >/dev/null
+}
+
+record_stack_hostname() {
+ kubectl --context "$KUBE_CONTEXT" annotate namespace "$STACK_NAME" \
+ "openremote.io/hostname=$HOSTNAME_VALUE" \
+ --overwrite >/dev/null
+}
+
+record_stack_mqtts_hostname() {
+ kubectl --context "$KUBE_CONTEXT" annotate namespace "$STACK_NAME" \
+ "openremote.io/mqtts-hostname=$MQTTS_HOSTNAME_VALUE" \
+ --overwrite >/dev/null
+}
+
+prepare_stack() {
+ require_command kubectl
+ require_kube_context
+ ensure_namespace
+ reserve_stack_hostname
+ record_stack_hostname
+ if [ "$EXPOSURE" = "ingress" ] && [ "$MQTTS_ENABLED" = true ]; then
+ reserve_stack_mqtts_hostname
+ record_stack_mqtts_hostname
+ fi
+
+ log "Stack namespace '$STACK_NAME' is prepared in context '$KUBE_CONTEXT'."
+ log "No Helm releases or workloads were installed."
+}
+
+require_stack_namespace() {
+ local stack_label
+
+ if ! stack_label="$(stack_namespace_label 2>/dev/null)"; then
+ die "Stack namespace '$STACK_NAME' does not exist in context '$KUBE_CONTEXT'."
+ fi
+
+ [ "$stack_label" = "$STACK_NAME" ] || die \
+ "Namespace '$STACK_NAME' does not have the matching openremote.io/stack label; refusing to modify it."
+}
+
+validate_target_cluster() {
+ local ssl_redirect_port
+ local vpc_cni_containers
+
+ if [ "$TARGET" = "eks" ]; then
+ kubectl --context "$KUBE_CONTEXT" get storageclass openremote-ebs >/dev/null 2>&1 || die \
+ "StorageClass 'openremote-ebs' is not installed in context '$KUBE_CONTEXT'."
+ kubectl --context "$KUBE_CONTEXT" get csidriver ebs.csi.aws.com >/dev/null 2>&1 || die \
+ "EBS CSI driver 'ebs.csi.aws.com' is not installed in context '$KUBE_CONTEXT'."
+ kubectl --context "$KUBE_CONTEXT" get \
+ customresourcedefinition/policyendpoints.networking.k8s.aws >/dev/null 2>&1 || die \
+ "Amazon VPC CNI NetworkPolicy enforcement is not enabled in context '$KUBE_CONTEXT'; run or-eks-cluster apply first."
+ if [ "$EXPOSURE" != "none" ]; then
+ kubectl --context "$KUBE_CONTEXT" --namespace kube-system get \
+ deployment aws-load-balancer-controller >/dev/null 2>&1 || die \
+ "AWS Load Balancer Controller is not installed in context '$KUBE_CONTEXT'; run or-eks-cluster apply first."
+ fi
+ if [ "$EXPOSURE" = "ingress" ]; then
+ kubectl --context "$KUBE_CONTEXT" get ingressclass openremote-alb >/dev/null 2>&1 || die \
+ "Shared IngressClass 'openremote-alb' is not installed in context '$KUBE_CONTEXT'; run or-eks-cluster apply first."
+ if ! ssl_redirect_port="$(kubectl --context "$KUBE_CONTEXT" get \
+ ingressclassparams.elbv2.k8s.aws openremote-alb \
+ -o 'jsonpath={.spec.sslRedirectPort}' 2>/dev/null)"; then
+ die "Shared IngressClassParams 'openremote-alb' is not installed in context '$KUBE_CONTEXT'; run or-eks-cluster apply first."
+ fi
+ [ "$ssl_redirect_port" = "443" ] || die \
+ "Shared IngressClassParams 'openremote-alb' does not enforce HTTPS redirection; run or-eks-cluster apply first."
+ fi
+ vpc_cni_containers="$(kubectl --context "$KUBE_CONTEXT" --namespace kube-system \
+ get daemonset aws-node \
+ -o 'jsonpath={.spec.template.spec.containers[*].name}')" || die \
+ "Unable to inspect the Amazon VPC CNI DaemonSet in context '$KUBE_CONTEXT'."
+ case " $vpc_cni_containers " in
+ *" aws-eks-nodeagent "*)
+ ;;
+ *)
+ die "Amazon VPC CNI NetworkPolicy enforcement is not active in context '$KUBE_CONTEXT'; run or-eks-cluster apply first."
+ ;;
+ esac
+ fi
+}
+
+discover_cluster_dns() {
+ if ! CLUSTER_DNS="$(kubectl --context "$KUBE_CONTEXT" --namespace kube-system \
+ get service kube-dns \
+ -o 'jsonpath={.spec.clusterIP}:{.spec.ports[?(@.name=="dns")].port}' 2>/dev/null)"; then
+ die "Unable to discover the kube-dns Service in context '$KUBE_CONTEXT'."
+ fi
+ [ -n "$CLUSTER_DNS" ] && [[ "$CLUSTER_DNS" == *:* ]] || die \
+ "The kube-dns Service in context '$KUBE_CONTEXT' has no usable DNS endpoint."
+}
+
+helm_apply() {
+ local component="$1"
+ shift
+ local chart="$KUBERNETES_DIR/$component"
+ local target_values="$TARGETS_DIR/$TARGET/$component.yaml"
+ local exposure_values="$EXPOSURES_DIR/$EXPOSURE/$component.yaml"
+ local target_exposure_values="$EXPOSURES_DIR/$EXPOSURE/$TARGET/$component.yaml"
+ local stack_values=""
+ local helm_args=(
+ upgrade --install "$component" "$chart"
+ --kube-context "$KUBE_CONTEXT"
+ --namespace "$STACK_NAME"
+ )
+
+ if [ -f "$target_values" ]; then
+ helm_args+=(--values "$target_values")
+ fi
+
+ if [ -f "$exposure_values" ]; then
+ helm_args+=(--values "$exposure_values")
+ fi
+
+ if [ -f "$target_exposure_values" ]; then
+ helm_args+=(--values "$target_exposure_values")
+ fi
+
+ if [ -n "$VALUES_DIR" ]; then
+ stack_values="$VALUES_DIR/$component.yaml"
+ if [ -f "$stack_values" ]; then
+ helm_args+=(--values "$stack_values")
+ fi
+ fi
+
+ helm_args+=("$@" --wait --timeout "$TIMEOUT")
+
+ log "Applying Helm release '$component'."
+ helm "${helm_args[@]}"
+}
+
+apply_stack() {
+ local manager_args=(
+ --set-string "or.hostname=$HOSTNAME_VALUE"
+ )
+ local keycloak_hostname="$HOSTNAME_VALUE"
+ local proxy_args=()
+ local setup_args=(
+ --set "networkPolicy.proxy.mqtts.enabled=$MQTTS_ENABLED"
+ --set-string "networkPolicy.proxy.mqtts.port=$PROXY_MQTTS_TLS_PORT"
+ )
+
+ if [ "$EXPOSURE" != "none" ]; then
+ # Manager discovers the token issuer through Keycloak's internal HTTP URL.
+ # Pin the public URL so discovery and proxied browser requests return the
+ # same issuer, including any non-default HTTPS port.
+ keycloak_hostname="https://$HOSTNAME_VALUE"
+ if [ "$EXPOSURE" = "haproxy" ] && [ "$HTTPS_PORT" != "443" ]; then
+ keycloak_hostname="$keycloak_hostname:$HTTPS_PORT"
+ fi
+ # Manager's default CORS origins use the hostname without the public port.
+ # Allow this stack's exact HTTPS origin so browser writes are not rejected.
+ manager_args+=(--set-string "or.webserver.allowedOrigins=$keycloak_hostname")
+ keycloak_hostname="$keycloak_hostname/auth"
+ fi
+
+ require_command helm
+ require_command kubectl
+ require_kube_context
+ validate_target_cluster
+ if [ "$EXPOSURE" != "none" ]; then
+ discover_cluster_dns
+ fi
+ ensure_namespace
+ reserve_stack_hostname
+ record_stack_hostname
+ if [ "$EXPOSURE" = "ingress" ] && [ "$MQTTS_ENABLED" = true ]; then
+ setup_args=(
+ --set "networkPolicy.proxy.mqtts.enabled=true"
+ --set-string "networkPolicy.proxy.mqtts.port=$PROXY_MQTT_EDGE_PORT"
+ )
+ reserve_stack_mqtts_hostname
+ record_stack_mqtts_hostname
+ fi
+
+ helm_apply or-setup "${setup_args[@]}"
+ helm_apply postgresql
+ helm_apply keycloak --set-string "or.hostname=$keycloak_hostname"
+ helm_apply manager "${manager_args[@]}"
+ if [ "$EXPOSURE" != "none" ]; then
+ proxy_args=(
+ --set-string "or.hostname=$HOSTNAME_VALUE"
+ --set-string "or.managerHost=manager.$STACK_NAME.svc.cluster.local"
+ --set-string "or.keycloakHost=keycloak.$STACK_NAME.svc.cluster.local"
+ --set-string "or.nameserver=$CLUSTER_DNS"
+ --set dns.enabled=false
+ --set "service.mqtts.enabled=$MQTTS_ENABLED"
+ )
+ if [ "$DNS_MODE" = "external-dns" ]; then
+ proxy_args+=(
+ --set dns.enabled=true
+ --set-string "dns.hostname=$HOSTNAME_VALUE"
+ )
+ fi
+ if [ "$EXPOSURE" = "haproxy" ]; then
+ proxy_args+=(
+ --set-string "or.httpsForwardedPort=$HTTPS_PORT"
+ --set-string "service.http.port=$HTTP_PORT"
+ --set-string "service.https.port=$HTTPS_PORT"
+ )
+ if [ "$MQTTS_ENABLED" = true ]; then
+ proxy_args+=(
+ --set-string "service.mqtts.port=$MQTTS_PORT"
+ --set-string "or.mqttPort=$PROXY_MQTTS_TLS_PORT"
+ )
+ fi
+ else
+ proxy_args+=(--set-string "ingress.hostname=$HOSTNAME_VALUE")
+ if [ "$TARGET" = "eks" ]; then
+ proxy_args+=(
+ --set-string "ingress.annotations.alb\\.ingress\\.kubernetes\\.io/certificate-arn=$CERTIFICATE_ARN"
+ )
+ if [ "$MQTTS_ENABLED" = true ]; then
+ proxy_args+=(
+ --set-string "or.mqttPort=$PROXY_MQTT_EDGE_PORT"
+ --set-string "service.mqtts.port=$MQTTS_PORT"
+ --set-string "service.mqtts.annotations.service\\.beta\\.kubernetes\\.io/aws-load-balancer-ssl-cert=$MQTTS_CERTIFICATE_ARN"
+ )
+ if [ "$DNS_MODE" = "external-dns" ]; then
+ proxy_args+=(
+ --set service.mqtts.dns.enabled=true
+ --set-string "service.mqtts.dns.hostname=$MQTTS_HOSTNAME_VALUE"
+ )
+ fi
+ fi
+ else
+ proxy_args+=(--set-string "ingress.tls[0].hosts[0]=$HOSTNAME_VALUE")
+ fi
+ fi
+ helm_apply proxy "${proxy_args[@]}"
+ fi
+
+ log "Stack '$STACK_NAME' is applied in context '$KUBE_CONTEXT'."
+ log "NetworkPolicy enforcement depends on the cluster CNI; verify cross-stack isolation before relying on it."
+ log "Retrieve its Manager login at any time with:"
+ printf ' %q credentials --name %q --kube-context %q\n' \
+ "$0" "$STACK_NAME" "$KUBE_CONTEXT"
+ if [ "$EXPOSURE" = "ingress" ]; then
+ if [ "$TARGET" = "eks" ]; then
+ log "HTTPS routing is requested through the cluster's shared OpenRemote ALB and this stack's private HAProxy gateway:"
+ printf ' https://%s/manager\n' "$HOSTNAME_VALUE"
+ if [ "$DNS_MODE" = "external-dns" ]; then
+ log "The proxy Ingress declares '$HOSTNAME_VALUE' for the cluster's ExternalDNS controller."
+ else
+ log "DNS records and certificate lifecycle remain external to this command."
+ fi
+ if [ "$MQTTS_ENABLED" = true ]; then
+ log "MQTTS is exposed through the stack's dedicated NLB with TLS terminated by ACM:"
+ printf ' mqtts://%s:%s\n' "$MQTTS_HOSTNAME_VALUE" "$MQTTS_PORT"
+ log "Inspect its external address with:"
+ printf ' kubectl --context %q --namespace %q get service proxy-mqtts\n' \
+ "$KUBE_CONTEXT" "$STACK_NAME"
+ if [ "$DNS_MODE" = "external-dns" ]; then
+ log "The proxy-mqtts Service declares '$MQTTS_HOSTNAME_VALUE' for the cluster's ExternalDNS controller."
+ else
+ log "The MQTTS DNS record and certificate lifecycle remain external to this command."
+ fi
+ fi
+ else
+ log "Routing is requested through the cluster's configured Ingress controller and this stack's private HAProxy gateway:"
+ printf ' Host: %s, path: /manager\n' "$HOSTNAME_VALUE"
+ if [ "$DNS_MODE" = "external-dns" ]; then
+ log "The proxy Ingress declares '$HOSTNAME_VALUE' for the cluster's ExternalDNS controller."
+ else
+ log "Ingress controller, DNS, and TLS configuration remain external to this command."
+ fi
+ fi
+ elif [ "$EXPOSURE" = "haproxy" ]; then
+ log "HAProxy is exposed through the stack's 'proxy' LoadBalancer Service."
+ printf ' https://%s' "$HOSTNAME_VALUE"
+ if [ "$HTTPS_PORT" != "443" ]; then
+ printf ':%s' "$HTTPS_PORT"
+ fi
+ printf '/manager\n'
+ if [ "$MQTTS_ENABLED" = true ]; then
+ log "MQTTS is exposed through the same HAProxy hostname and certificate:"
+ printf ' mqtts://%s:%s\n' "$HOSTNAME_VALUE" "$MQTTS_PORT"
+ fi
+ log "Inspect its external address with:"
+ printf ' kubectl --context %q --namespace %q get service proxy\n' \
+ "$KUBE_CONTEXT" "$STACK_NAME"
+ if [ "$DNS_MODE" = "external-dns" ]; then
+ log "The proxy LoadBalancer Service declares '$HOSTNAME_VALUE' for the cluster's ExternalDNS controller."
+ else
+ log "DNS records and production certificate issuance remain external to this command."
+ fi
+ else
+ log "External routing is disabled for this stack."
+ fi
+}
+
+verify_retained_secret() {
+ local resource_policy
+
+ if ! resource_policy="$(kubectl --context "$KUBE_CONTEXT" --namespace "$STACK_NAME" \
+ get secret openremote-secret \
+ -o 'jsonpath={.metadata.annotations.helm\.sh/resource-policy}' 2>/dev/null)"; then
+ die "Secret '$STACK_NAME/openremote-secret' does not exist; refusing data-retaining uninstall."
+ fi
+
+ [ "$resource_policy" = "keep" ] || die \
+ "Secret '$STACK_NAME/openremote-secret' does not have the Helm keep policy; run apply before data-retaining uninstall."
+}
+
+verify_retained_pvcs() {
+ local pvc_metadata
+ local claim_name
+ local release_owner
+ local resource_policy
+
+ pvc_metadata="$(kubectl --context "$KUBE_CONTEXT" --namespace "$STACK_NAME" \
+ get persistentvolumeclaims \
+ -o 'jsonpath={range .items[*]}{.metadata.name}{"|"}{.metadata.annotations.meta\.helm\.sh/release-name}{"|"}{.metadata.annotations.helm\.sh/resource-policy}{"\n"}{end}')"
+
+ while IFS='|' read -r claim_name release_owner resource_policy; do
+ if [ -z "$claim_name" ]; then
+ continue
+ fi
+
+ case "$release_owner" in
+ manager|postgresql|proxy)
+ [ "$resource_policy" = "keep" ] || die \
+ "PVC '$STACK_NAME/$claim_name' is managed by Helm without the keep policy; refusing data-retaining uninstall."
+ ;;
+ esac
+ done <<< "$pvc_metadata"
+}
+
+verify_retained_state() {
+ verify_retained_secret
+ verify_retained_pvcs
+}
+
+uninstall_release() {
+ local release_name="$1"
+
+ if helm status "$release_name" \
+ --kube-context "$KUBE_CONTEXT" \
+ --namespace "$STACK_NAME" >/dev/null 2>&1; then
+ log "Uninstalling Helm release '$release_name'."
+ helm uninstall "$release_name" \
+ --kube-context "$KUBE_CONTEXT" \
+ --namespace "$STACK_NAME" \
+ --wait \
+ --timeout "$TIMEOUT"
+ else
+ log "Helm release '$release_name' is not installed; skipping."
+ fi
+}
+
+uninstall_releases() {
+ uninstall_release proxy
+ uninstall_release manager
+ uninstall_release keycloak
+ uninstall_release postgresql
+ uninstall_release or-setup
+}
+
+uninstall_stack() {
+ require_command helm
+ require_command kubectl
+ require_kube_context
+ require_stack_namespace
+ verify_retained_state
+
+ uninstall_releases
+
+ log "Stack '$STACK_NAME' is uninstalled. Its namespace, credentials, PVCs, and data are retained."
+ log "Run apply with the same name and context to restore its workloads."
+}
+
+show_stack_credentials() {
+ local admin_password
+
+ require_command kubectl
+ require_kube_context
+ require_stack_namespace
+
+ if ! admin_password="$(kubectl --context "$KUBE_CONTEXT" --namespace "$STACK_NAME" \
+ get secret openremote-secret \
+ -o 'go-template={{index .data "admin-password" | base64decode}}' 2>/dev/null)"; then
+ die "Unable to read admin-password from Secret '$STACK_NAME/openremote-secret'."
+ fi
+
+ [ -n "$admin_password" ] || die \
+ "Secret '$STACK_NAME/openremote-secret' contains an empty admin-password."
+
+ printf 'Manager username: admin\n'
+ printf 'Manager password: %s\n' "$admin_password"
+}
+
+destroy_stack() {
+ local pv_names
+ local pv_name
+ local pv_cleanup_failed=false
+ local reservation
+ local reservations
+ local reservation_endpoint
+
+ require_command helm
+ require_command kubectl
+ require_kube_context
+ require_stack_namespace
+
+ reservations="$(stack_hostname_reservations)" || die "Unable to inspect hostname reservations before destroy."
+ while IFS= read -r reservation; do
+ [ -n "$reservation" ] || continue
+ require_reservation_namespace_identity "$reservation"
+ done <<< "$reservations"
+
+ pv_names="$(kubectl --context "$KUBE_CONTEXT" --namespace "$STACK_NAME" \
+ get persistentvolumeclaims \
+ -o 'jsonpath={range .items[*]}{.spec.volumeName}{"\n"}{end}')"
+
+ uninstall_releases
+
+ log "Deleting stack namespace '$STACK_NAME', including all PVCs and namespaced data."
+ kubectl --context "$KUBE_CONTEXT" delete namespace "$STACK_NAME" \
+ --wait=true \
+ --timeout "$TIMEOUT"
+
+ for pv_name in $pv_names; do
+ if kubectl --context "$KUBE_CONTEXT" get "persistentvolume/$pv_name" >/dev/null 2>&1; then
+ log "Waiting for PersistentVolume '$pv_name' to be deleted."
+ if ! kubectl --context "$KUBE_CONTEXT" wait \
+ --for=delete "persistentvolume/$pv_name" \
+ --timeout "$TIMEOUT"; then
+ pv_cleanup_failed=true
+ fi
+ fi
+ done
+
+ if [ "$pv_cleanup_failed" = true ]; then
+ die "Stack namespace was deleted, but one or more PersistentVolumes remain; inspect their reclaim policy and backing storage."
+ fi
+
+ while IFS= read -r reservation; do
+ [ -n "$reservation" ] || continue
+ HOSTNAME_VALUE="$(jq -r '.hostname' <<< "$reservation")"
+ if [ "$(jq -r '.target + "/" + .dns' <<< "$reservation")" = "local/none" ]; then
+ release_stack_hostname
+ else
+ reservation_endpoint="$(jq -r '.endpoint' <<< "$reservation")"
+ log "$(printf '%s' "$reservation_endpoint" | tr '[:lower:]' '[:upper:]') hostname '$HOSTNAME_VALUE' remains reserved until external cleanup is confirmed."
+ if [ "$reservation_endpoint" = web ]; then
+ log "Use or-eks-stack destroy for automated cleanup, or after verifying external cleanup run:"
+ else
+ log "Use or-eks-stack destroy to verify EKS DNS cleanup and release the retained reservation, or after verifying external cleanup run:"
+ fi
+ printf ' %q release-hostname --name %q --kube-context %q --hostname %q --confirm %q\n' \
+ "$0" "$STACK_NAME" "$KUBE_CONTEXT" "$HOSTNAME_VALUE" "$STACK_NAME"
+ fi
+ done <<< "$reservations"
+ log "Stack '$STACK_NAME' and its namespaced data are destroyed."
+}
+
+show_stack_status() {
+ local dns_mode
+ local exposure
+ local proxy_address
+ local proxy_certificate
+ local proxy_hostname
+ local proxy_mqtts_address
+ local proxy_mqtts_hostname
+ local proxy_mqtts_port
+ local target
+
+ require_command helm
+ require_command kubectl
+ require_kube_context
+ require_stack_namespace
+
+ target="$(stack_namespace_target)"
+ exposure="$(stack_namespace_exposure)"
+ dns_mode="$(stack_namespace_dns)"
+ log "Configuration"
+ printf 'Target: %s\n' "${target:-unknown}"
+ printf 'Exposure: %s\n' "${exposure:-unknown}"
+ printf 'DNS: %s\n' "${dns_mode:-none}"
+
+ log "Helm releases"
+ helm list --kube-context "$KUBE_CONTEXT" --namespace "$STACK_NAME"
+
+ log "Workloads"
+ kubectl --context "$KUBE_CONTEXT" --namespace "$STACK_NAME" \
+ get deployments,statefulsets,pods -o wide
+
+ log "Networking"
+ kubectl --context "$KUBE_CONTEXT" --namespace "$STACK_NAME" \
+ get services,ingresses,networkpolicies
+
+ log "Persistent storage"
+ kubectl --context "$KUBE_CONTEXT" --namespace "$STACK_NAME" \
+ get persistentvolumeclaims
+
+ if [ "$exposure" = "haproxy" ]; then
+ proxy_address=""
+ if proxy_address="$(kubectl --context "$KUBE_CONTEXT" --namespace "$STACK_NAME" \
+ get service proxy \
+ -o 'jsonpath={.status.loadBalancer.ingress[0].ip}{.status.loadBalancer.ingress[0].hostname}' \
+ 2>/dev/null)" && [ -n "$proxy_address" ]; then
+ printf 'HAProxy external address: %s\n' "$proxy_address"
+ else
+ log "WARNING: HAProxy LoadBalancer external address is pending or unavailable."
+ fi
+
+ proxy_mqtts_port="$(kubectl --context "$KUBE_CONTEXT" --namespace "$STACK_NAME" \
+ get service proxy \
+ -o 'jsonpath={.spec.ports[?(@.name=="mqtts")].port}' \
+ 2>/dev/null || true)"
+ if [ -n "$proxy_mqtts_port" ]; then
+ proxy_hostname="$(stack_namespace_hostname 2>/dev/null || true)"
+ printf 'MQTTS: enabled\n'
+ printf 'MQTTS endpoint: mqtts://%s:%s\n' \
+ "${proxy_hostname:-${proxy_address:-pending}}" "$proxy_mqtts_port"
+ else
+ printf 'MQTTS: disabled\n'
+ fi
+
+ if proxy_certificate="$(kubectl --context "$KUBE_CONTEXT" --namespace "$STACK_NAME" \
+ exec deployment/proxy -- sh -c \
+ 'for cert in /deployment/certs/* /data/proxy/certs/*; do [ -f "$cert" ] || continue; openssl x509 -in "$cert" -noout -issuer -enddate && exit 0; done; exit 1' \
+ 2>/dev/null)"; then
+ log "HAProxy certificate"
+ printf '%s\n' "$proxy_certificate"
+ else
+ log "WARNING: HAProxy has no readable managed or custom certificate yet; its fallback self-signed certificate may be in use."
+ fi
+ elif [ "$exposure" = "ingress" ]; then
+ proxy_mqtts_port="$(kubectl --context "$KUBE_CONTEXT" --namespace "$STACK_NAME" \
+ get service proxy-mqtts \
+ -o 'jsonpath={.spec.ports[?(@.name=="mqtts")].port}' \
+ 2>/dev/null || true)"
+ if [ -n "$proxy_mqtts_port" ]; then
+ proxy_mqtts_address="$(kubectl --context "$KUBE_CONTEXT" --namespace "$STACK_NAME" \
+ get service proxy-mqtts \
+ -o 'jsonpath={.status.loadBalancer.ingress[0].ip}{.status.loadBalancer.ingress[0].hostname}' \
+ 2>/dev/null || true)"
+ proxy_mqtts_hostname="$(stack_namespace_mqtts_hostname 2>/dev/null || true)"
+ printf 'MQTTS: enabled\n'
+ printf 'MQTTS endpoint: mqtts://%s:%s\n' \
+ "${proxy_mqtts_hostname:-${proxy_mqtts_address:-pending}}" "$proxy_mqtts_port"
+ else
+ printf 'MQTTS: disabled\n'
+ fi
+ fi
+}
+
+main() {
+ parse_arguments "$@"
+ validate_common_options
+
+ if [ "$COMMAND" != "destroy" ] && [ "$COMMAND" != "release-hostname" ] && [ -n "$CONFIRM_STACK_NAME" ]; then
+ die "Option --confirm is only valid with destroy or release-hostname."
+ fi
+
+ case "$COMMAND" in
+ prepare)
+ validate_prepare_options
+ prepare_stack
+ ;;
+ apply)
+ validate_apply_options
+ apply_stack
+ ;;
+ status)
+ show_stack_status
+ ;;
+ credentials)
+ show_stack_credentials
+ ;;
+ uninstall)
+ validate_timeout
+ uninstall_stack
+ ;;
+ release-hostname)
+ [ "$CONFIRM_STACK_NAME" = "$STACK_NAME" ] || die "Release requires --confirm '$STACK_NAME'."
+ [ -n "$HOSTNAME_VALUE" ] || die "Release requires --hostname."
+ validate_hostname_option
+ require_command kubectl
+ require_kube_context
+ release_stack_hostname
+ ;;
+ destroy)
+ validate_destroy_options
+ destroy_stack
+ ;;
+ esac
+}
+
+main "$@"
diff --git a/kubernetes/postgresql/Chart.yaml b/kubernetes/postgresql/Chart.yaml
index 0ff06a079..5f7f28f04 100644
--- a/kubernetes/postgresql/Chart.yaml
+++ b/kubernetes/postgresql/Chart.yaml
@@ -4,6 +4,6 @@ description: A Helm chart for deploying OpenRemote PostgreSQL inKubernetes
type: application
-version: 0.1.0
+version: 0.2.0
appVersion: "17.9.0.1-slim"
diff --git a/kubernetes/postgresql/templates/_helpers.tpl b/kubernetes/postgresql/templates/_helpers.tpl
index 30c6288f4..e629262f4 100644
--- a/kubernetes/postgresql/templates/_helpers.tpl
+++ b/kubernetes/postgresql/templates/_helpers.tpl
@@ -60,3 +60,10 @@ Create the name of the service account to use
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}
+
+{{/*
+Return the PVC used for PostgreSQL data.
+*/}}
+{{- define "postgresql.persistence.claimName" -}}
+{{- default (include "postgresql.fullname" .) .Values.persistence.existingClaim -}}
+{{- end }}
diff --git a/kubernetes/postgresql/templates/pvc.yaml b/kubernetes/postgresql/templates/pvc.yaml
index e1f79c6f2..8eeff7e35 100644
--- a/kubernetes/postgresql/templates/pvc.yaml
+++ b/kubernetes/postgresql/templates/pvc.yaml
@@ -1,12 +1,29 @@
+{{- if not .Values.persistence.existingClaim }}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: {{ include "postgresql.fullname" . }}
+ labels:
+ {{- include "postgresql.labels" . | nindent 4 }}
+ {{- if or .Values.persistence.retain .Values.persistence.annotations }}
+ annotations:
+ {{- if .Values.persistence.retain }}
+ helm.sh/resource-policy: keep
+ {{- end }}
+ {{- with .Values.persistence.annotations }}
+ {{- toYaml . | nindent 4 }}
+ {{- end }}
+ {{- end }}
spec:
accessModes:
- - ReadWriteOnce
+ {{- toYaml .Values.persistence.accessModes | nindent 4 }}
resources:
requests:
- storage: {{ .Values.volumeSize }}
- volumeName: postgresql-data-pv
- storageClassName: manual
+ storage: {{ .Values.persistence.size }}
+ {{- with .Values.persistence.storageClass }}
+ storageClassName: {{ . | quote }}
+ {{- end }}
+ {{- with .Values.persistence.volumeName }}
+ volumeName: {{ . | quote }}
+ {{- end }}
+{{- end }}
diff --git a/kubernetes/postgresql/templates/statefulset.yaml b/kubernetes/postgresql/templates/statefulset.yaml
index decec1900..49cffac65 100644
--- a/kubernetes/postgresql/templates/statefulset.yaml
+++ b/kubernetes/postgresql/templates/statefulset.yaml
@@ -92,7 +92,7 @@ spec:
volumes:
- name: postgresql-data
persistentVolumeClaim:
- claimName: {{ include "postgresql.fullname" . }}
+ claimName: {{ include "postgresql.persistence.claimName" . }}
{{- with .Values.volumes }}
{{- toYaml . | nindent 8 }}
{{- end }}
diff --git a/kubernetes/postgresql/values-eks.yaml b/kubernetes/postgresql/values-eks.yaml
index effbd7369..e5c58cc09 100644
--- a/kubernetes/postgresql/values-eks.yaml
+++ b/kubernetes/postgresql/values-eks.yaml
@@ -1,12 +1,5 @@
requiresPermissionsFix: true
useSubPath: true
-affinity:
- nodeAffinity:
- requiredDuringSchedulingIgnoredDuringExecution:
- nodeSelectorTerms:
- - matchExpressions:
- - key: topology.kubernetes.io/zone
- operator: In
- values:
- - eu-west-1a
+persistence:
+ storageClass: openremote-ebs
diff --git a/kubernetes/postgresql/values.yaml b/kubernetes/postgresql/values.yaml
index 75a998789..93a15541e 100644
--- a/kubernetes/postgresql/values.yaml
+++ b/kubernetes/postgresql/values.yaml
@@ -81,8 +81,20 @@ readinessProbe:
command:
- pg_isready
-# Size of volume requested in PVC, must be aligned with available PV
-volumeSize: "1Gi"
+# Persistent PostgreSQL data. When storageClass is empty, the cluster's default
+# StorageClass dynamically provisions the volume. Set existingClaim to use a
+# PVC managed outside this chart instead.
+persistence:
+ size: "1Gi"
+ storageClass: ""
+ existingClaim: ""
+ # Optionally bind the chart-managed claim to a pre-provisioned PV.
+ volumeName: ""
+ accessModes:
+ - ReadWriteOnce
+ # Preserve the chart-managed PVC when Helm uninstalls the release.
+ retain: true
+ annotations: {}
# Additional volumes on the output Deployment definition.
volumes: []
diff --git a/kubernetes/proxy/Chart.yaml b/kubernetes/proxy/Chart.yaml
index d689c61e2..524e2e6db 100644
--- a/kubernetes/proxy/Chart.yaml
+++ b/kubernetes/proxy/Chart.yaml
@@ -4,6 +4,6 @@ description: A Helm chart for deploying OR HAProxy in Kubernetes
type: application
-version: 0.1.0
+version: 0.2.0
appVersion: "3.2.19.0"
diff --git a/kubernetes/proxy/templates/deployment.yaml b/kubernetes/proxy/templates/deployment.yaml
index 3d171b49a..001db3742 100644
--- a/kubernetes/proxy/templates/deployment.yaml
+++ b/kubernetes/proxy/templates/deployment.yaml
@@ -6,6 +6,8 @@ metadata:
{{- include "proxy.labels" . | nindent 4 }}
spec:
replicas: {{ .Values.replicaCount }}
+ strategy:
+ {{- toYaml .Values.strategy | nindent 4 }}
selector:
matchLabels:
{{- include "proxy.selectorLabels" . | nindent 6 }}
@@ -34,10 +36,23 @@ spec:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
-# ports:
-# - name: http
-# containerPort: {{ .Values.service.port }}
-# protocol: TCP
+ ports:
+ - name: http
+ containerPort: {{ .Values.or.httpPort }}
+ protocol: TCP
+ {{- if .Values.service.https.enabled }}
+ - name: https
+ containerPort: {{ .Values.or.httpsPort }}
+ protocol: TCP
+ {{- end }}
+ {{- if .Values.service.mqtts.enabled }}
+ - name: mqtts
+ containerPort: {{ .Values.or.mqttPort }}
+ protocol: TCP
+ {{- end }}
+ - name: metrics
+ containerPort: 8404
+ protocol: TCP
livenessProbe:
{{- toYaml .Values.livenessProbe | nindent 12 }}
readinessProbe:
@@ -57,6 +72,10 @@ spec:
value: "{{ .Values.or.additionalHostnames }}"
- name: HAPROXY_CONFIG
value: "{{ .Values.or.haproxyConfig }}"
+ {{- if or .Values.or.disableAcme .Values.certificate.existingSecret }}
+ - name: DISABLE_ACME
+ value: "true"
+ {{- end }}
- name: HTTP_PORT
value: "{{ .Values.or.httpPort }}"
- name: HTTPS_PORT
@@ -71,6 +90,10 @@ spec:
value: "{{ .Values.or.leExtraArgs }}"
- name: AWS_ROUTE53_ROLE
value: "{{ .Values.or.awsRoute53Role }}"
+ {{- if .Values.certificate.existingSecret }}
+ - name: CUSTOM_CERT_DIR
+ value: "/data/proxy/certs"
+ {{- end }}
- name: MANAGER_HOST
value: "{{ .Values.or.managerHost }}"
- name: MANAGER_WEB_PORT
@@ -92,14 +115,37 @@ spec:
{{- with .Values.or.env }}
{{- toYaml . | nindent 12 }}
{{- end }}
- {{- with .Values.volumeMounts }}
+ {{- if or .Values.persistence.enabled .Values.certificate.existingSecret .Values.volumeMounts }}
volumeMounts:
+ {{- if .Values.persistence.enabled }}
+ - name: proxy-data
+ mountPath: /deployment
+ {{- end }}
+ {{- if .Values.certificate.existingSecret }}
+ - name: custom-certificates
+ mountPath: /data/proxy/certs
+ readOnly: true
+ {{- end }}
+ {{- with .Values.volumeMounts }}
{{- toYaml . | nindent 12 }}
{{- end }}
- {{- with .Values.volumes }}
+ {{- end }}
+ {{- if or .Values.persistence.enabled .Values.certificate.existingSecret .Values.volumes }}
volumes:
+ {{- if .Values.persistence.enabled }}
+ - name: proxy-data
+ persistentVolumeClaim:
+ claimName: {{ .Values.persistence.existingClaim | default (include "proxy.fullname" .) }}
+ {{- end }}
+ {{- if .Values.certificate.existingSecret }}
+ - name: custom-certificates
+ secret:
+ secretName: {{ .Values.certificate.existingSecret }}
+ {{- end }}
+ {{- with .Values.volumes }}
{{- toYaml . | nindent 8 }}
{{- end }}
+ {{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
diff --git a/kubernetes/proxy/templates/ingress.yaml b/kubernetes/proxy/templates/ingress.yaml
new file mode 100644
index 000000000..7bd2cff62
--- /dev/null
+++ b/kubernetes/proxy/templates/ingress.yaml
@@ -0,0 +1,39 @@
+{{- if .Values.ingress.enabled }}
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ name: {{ include "proxy.fullname" . }}
+ labels:
+ {{- include "proxy.labels" . | nindent 4 }}
+ {{- if .Values.dns.enabled }}
+ openremote.io/managed-dns: "true"
+ {{- end }}
+ {{- if or .Values.ingress.annotations .Values.dns.enabled }}
+ annotations:
+ {{- with .Values.ingress.annotations }}
+ {{- toYaml . | nindent 4 }}
+ {{- end }}
+ {{- if .Values.dns.enabled }}
+ external-dns.kubernetes.io/hostname: {{ required "dns.hostname is required when dns.enabled is true" .Values.dns.hostname | quote }}
+ {{- end }}
+ {{- end }}
+spec:
+ {{- with .Values.ingress.className }}
+ ingressClassName: {{ . }}
+ {{- end }}
+ {{- with .Values.ingress.tls }}
+ tls:
+ {{- toYaml . | nindent 4 }}
+ {{- end }}
+ rules:
+ - host: {{ .Values.ingress.hostname | quote }}
+ http:
+ paths:
+ - path: {{ .Values.ingress.path }}
+ pathType: {{ .Values.ingress.pathType }}
+ backend:
+ service:
+ name: {{ include "proxy.fullname" . }}
+ port:
+ number: {{ .Values.service.http.port }}
+{{- end }}
diff --git a/kubernetes/proxy/templates/pvc.yaml b/kubernetes/proxy/templates/pvc.yaml
new file mode 100644
index 000000000..4ba0fe2ea
--- /dev/null
+++ b/kubernetes/proxy/templates/pvc.yaml
@@ -0,0 +1,29 @@
+{{- if and .Values.persistence.enabled (not .Values.persistence.existingClaim) }}
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+ name: {{ include "proxy.fullname" . }}
+ labels:
+ {{- include "proxy.labels" . | nindent 4 }}
+ {{- if or .Values.persistence.retain .Values.persistence.annotations }}
+ annotations:
+ {{- if .Values.persistence.retain }}
+ helm.sh/resource-policy: keep
+ {{- end }}
+ {{- with .Values.persistence.annotations }}
+ {{- toYaml . | nindent 4 }}
+ {{- end }}
+ {{- end }}
+spec:
+ accessModes:
+ {{- toYaml .Values.persistence.accessModes | nindent 4 }}
+ resources:
+ requests:
+ storage: {{ .Values.persistence.size }}
+ {{- with .Values.persistence.storageClass }}
+ storageClassName: {{ . | quote }}
+ {{- end }}
+ {{- with .Values.persistence.volumeName }}
+ volumeName: {{ . | quote }}
+ {{- end }}
+{{- end }}
diff --git a/kubernetes/proxy/templates/service.yaml b/kubernetes/proxy/templates/service.yaml
index aed7dc5b3..c6b211ea6 100644
--- a/kubernetes/proxy/templates/service.yaml
+++ b/kubernetes/proxy/templates/service.yaml
@@ -4,9 +4,17 @@ metadata:
name: {{ include "proxy.fullname" . }}
labels:
{{- include "proxy.labels" . | nindent 4 }}
- {{- with .Values.service.http.annotations }}
+ {{- if and .Values.dns.enabled (eq .Values.service.http.type "LoadBalancer") }}
+ openremote.io/managed-dns: "true"
+ {{- end }}
+ {{- if or .Values.service.http.annotations (and .Values.dns.enabled (eq .Values.service.http.type "LoadBalancer")) }}
annotations:
+ {{- with .Values.service.http.annotations }}
{{- toYaml . | nindent 4 }}
+ {{- end }}
+ {{- if and .Values.dns.enabled (eq .Values.service.http.type "LoadBalancer") }}
+ external-dns.kubernetes.io/hostname: {{ required "dns.hostname is required when dns.enabled is true" .Values.dns.hostname | quote }}
+ {{- end }}
{{- end }}
spec:
type: {{ .Values.service.http.type }}
@@ -15,14 +23,18 @@ spec:
targetPort: {{ .Values.or.httpPort }}
protocol: TCP
name: http
- - port: 443
+ {{- if .Values.service.https.enabled }}
+ - port: {{ .Values.service.https.port }}
targetPort: {{ .Values.or.httpsPort }}
protocol: TCP
name: https
- - port: 8883
+ {{- end }}
+ {{- if and .Values.service.mqtts.enabled (not .Values.service.mqtts.separate) }}
+ - port: {{ .Values.service.mqtts.port }}
targetPort: {{ .Values.or.mqttPort }}
protocol: TCP
name: mqtts
+ {{- end }}
selector:
{{- include "proxy.selectorLabels" . | nindent 4 }}
---
@@ -45,3 +57,33 @@ spec:
selector:
{{- include "proxy.selectorLabels" . | nindent 4 }}
{{- end }}
+---
+{{- if and .Values.service.mqtts.enabled .Values.service.mqtts.separate }}
+apiVersion: v1
+kind: Service
+metadata:
+ name: {{ include "proxy.fullname" . }}-mqtts
+ labels:
+ {{- include "proxy.labels" . | nindent 4 }}
+ {{- if .Values.service.mqtts.dns.enabled }}
+ openremote.io/managed-dns: "true"
+ {{- end }}
+ {{- if or .Values.service.mqtts.annotations .Values.service.mqtts.dns.enabled }}
+ annotations:
+ {{- with .Values.service.mqtts.annotations }}
+ {{- toYaml . | nindent 4 }}
+ {{- end }}
+ {{- if .Values.service.mqtts.dns.enabled }}
+ external-dns.kubernetes.io/hostname: {{ required "service.mqtts.dns.hostname is required when service.mqtts.dns.enabled is true" .Values.service.mqtts.dns.hostname | quote }}
+ {{- end }}
+ {{- end }}
+spec:
+ type: {{ .Values.service.mqtts.type }}
+ ports:
+ - port: {{ .Values.service.mqtts.port }}
+ targetPort: {{ .Values.or.mqttPort }}
+ protocol: TCP
+ name: mqtts
+ selector:
+ {{- include "proxy.selectorLabels" . | nindent 4 }}
+{{- end }}
diff --git a/kubernetes/proxy/templates/tests/test-connection.yaml b/kubernetes/proxy/templates/tests/test-connection.yaml
index 79f35f2da..3407a857a 100644
--- a/kubernetes/proxy/templates/tests/test-connection.yaml
+++ b/kubernetes/proxy/templates/tests/test-connection.yaml
@@ -11,5 +11,5 @@ spec:
- name: wget
image: busybox
command: ['wget']
- args: ['{{ include "proxy.fullname" . }}:{{ .Values.service.port }}']
+ args: ['{{ include "proxy.fullname" . }}:{{ .Values.service.http.port }}']
restartPolicy: Never
diff --git a/kubernetes/proxy/values-eks.yaml b/kubernetes/proxy/values-eks.yaml
index edab6dceb..b8d5ebecc 100644
--- a/kubernetes/proxy/values-eks.yaml
+++ b/kubernetes/proxy/values-eks.yaml
@@ -1,3 +1,6 @@
+persistence:
+ storageClass: openremote-ebs
+
service:
http:
annotations:
diff --git a/kubernetes/proxy/values.yaml b/kubernetes/proxy/values.yaml
index 3619170eb..dedacb688 100644
--- a/kubernetes/proxy/values.yaml
+++ b/kubernetes/proxy/values.yaml
@@ -5,6 +5,11 @@
# This will set the replicaset count more information can be found here: https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/
replicaCount: 1
+# The proxy writes certificate state to a ReadWriteOnce volume. Recreate avoids
+# a rolling update trying to mount that volume on old and new Pods at once.
+strategy:
+ type: Recreate
+
# This sets the container image more information can be found here: https://kubernetes.io/docs/concepts/containers/images/
image:
repository: openremote/proxy
@@ -57,6 +62,22 @@ service:
# This sets the ports more information can be found here: https://kubernetes.io/docs/concepts/services-networking/service/#field-spec-ports
port: 80
annotations: {}
+ https:
+ enabled: true
+ port: 443
+ mqtts:
+ # MQTTS is opt-in. In HAProxy exposure mode this port is added to the main
+ # Service and HAProxy terminates TLS. In EKS Ingress mode, separate creates
+ # a dedicated NLB Service whose listener terminates TLS before forwarding
+ # plaintext MQTT to the namespace-local proxy gateway.
+ enabled: false
+ separate: false
+ type: LoadBalancer
+ port: 8883
+ annotations: {}
+ dns:
+ enabled: false
+ hostname: ""
metrics:
enabled: true
# Service type, typically use ClusterIP for local deployment and LoadBalancer for cloud deployments
@@ -64,6 +85,45 @@ service:
port: 8404
annotations: {}
+# Route HTTP(S) traffic to the proxy through a Kubernetes Ingress. In this
+# mode, the Ingress controller terminates TLS and the proxy provides the
+# namespace-local gateway to Manager and Keycloak.
+ingress:
+ enabled: false
+ className: ""
+ hostname: localhost
+ annotations: {}
+ path: /
+ pathType: Prefix
+ tls: []
+
+# Declare the public hostname to an ExternalDNS controller. or-stack enables
+# this only on the single public resource for the selected exposure mode:
+# the Ingress in ingress mode or the LoadBalancer Service in HAProxy mode.
+dns:
+ enabled: false
+ hostname: ""
+
+# Persist the proxy's /deployment directory, which contains the Certbot account,
+# generated certificates, and renewal state. Set existingClaim to use a PVC
+# managed outside this chart.
+persistence:
+ enabled: true
+ size: "1Gi"
+ storageClass: ""
+ existingClaim: ""
+ volumeName: ""
+ accessModes:
+ - ReadWriteOnce
+ # Preserve the chart-managed PVC when Helm uninstalls the release.
+ retain: true
+ annotations: {}
+
+certificate:
+ # Optional Secret containing one or more full-chain PEM files, including
+ # private keys. Its keys are mounted as files in CUSTOM_CERT_DIR.
+ existingSecret: ""
+
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
@@ -133,6 +193,10 @@ or:
# USE A CUSTOM PROXY CONFIG - COPY FROM https://raw.githubusercontent.com/openremote/proxy/main/haproxy.cfg
haproxyConfig: "/etc/haproxy/haproxy.cfg"
+ # Disable proxy-managed ACME certificates, for example when TLS terminates
+ # at an Ingress controller.
+ disableAcme: false
+
# Email address for SSL certificate issuance and expiration notifications.
leEmail: ""
diff --git a/kubernetes/stack/exposures/haproxy/eks/proxy.yaml b/kubernetes/stack/exposures/haproxy/eks/proxy.yaml
new file mode 100644
index 000000000..5a54d9812
--- /dev/null
+++ b/kubernetes/stack/exposures/haproxy/eks/proxy.yaml
@@ -0,0 +1,10 @@
+persistence:
+ storageClass: openremote-ebs
+
+service:
+ http:
+ type: LoadBalancer
+ annotations:
+ service.beta.kubernetes.io/aws-load-balancer-type: "external"
+ service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
+ service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
diff --git a/kubernetes/stack/exposures/haproxy/local/proxy.yaml b/kubernetes/stack/exposures/haproxy/local/proxy.yaml
new file mode 100644
index 000000000..e32397dac
--- /dev/null
+++ b/kubernetes/stack/exposures/haproxy/local/proxy.yaml
@@ -0,0 +1,5 @@
+# Docker Desktop provides local LoadBalancer support. For clusters without it,
+# override service.http.type to ClusterIP and use kubectl port-forward.
+service:
+ http:
+ type: LoadBalancer
diff --git a/kubernetes/stack/exposures/haproxy/or-setup.yaml b/kubernetes/stack/exposures/haproxy/or-setup.yaml
new file mode 100644
index 000000000..2abc48139
--- /dev/null
+++ b/kubernetes/stack/exposures/haproxy/or-setup.yaml
@@ -0,0 +1,3 @@
+networkPolicy:
+ proxy:
+ enabled: true
diff --git a/kubernetes/stack/exposures/haproxy/proxy.yaml b/kubernetes/stack/exposures/haproxy/proxy.yaml
new file mode 100644
index 000000000..bd49eac7b
--- /dev/null
+++ b/kubernetes/stack/exposures/haproxy/proxy.yaml
@@ -0,0 +1,8 @@
+# Backends use namespace-qualified names supplied by or-stack. Certificate and
+# Certbot state are retained on the proxy PVC across uninstall/reapply.
+persistence:
+ enabled: true
+
+service:
+ mqtts:
+ enabled: false
diff --git a/kubernetes/stack/exposures/ingress/eks/proxy.yaml b/kubernetes/stack/exposures/ingress/eks/proxy.yaml
new file mode 100644
index 000000000..3ba8e04d0
--- /dev/null
+++ b/kubernetes/stack/exposures/ingress/eks/proxy.yaml
@@ -0,0 +1,18 @@
+ingress:
+ className: openremote-alb
+ annotations:
+ alb.ingress.kubernetes.io/backend-protocol: HTTP
+ alb.ingress.kubernetes.io/healthcheck-path: /docker-health
+ alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80},{"HTTPS":443}]'
+
+service:
+ mqtts:
+ separate: true
+ type: LoadBalancer
+ annotations:
+ service.beta.kubernetes.io/aws-load-balancer-type: "external"
+ service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
+ service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
+ service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "mqtts"
+ service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
+ service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: "tcp"
diff --git a/kubernetes/stack/exposures/ingress/or-setup.yaml b/kubernetes/stack/exposures/ingress/or-setup.yaml
new file mode 100644
index 000000000..1f946fdbd
--- /dev/null
+++ b/kubernetes/stack/exposures/ingress/or-setup.yaml
@@ -0,0 +1,6 @@
+networkPolicy:
+ proxy:
+ enabled: true
+ ports:
+ - protocol: TCP
+ port: 8080
diff --git a/kubernetes/stack/exposures/ingress/proxy.yaml b/kubernetes/stack/exposures/ingress/proxy.yaml
new file mode 100644
index 000000000..f1ade867b
--- /dev/null
+++ b/kubernetes/stack/exposures/ingress/proxy.yaml
@@ -0,0 +1,20 @@
+# The Ingress controller terminates TLS. This private HAProxy gateway gives the
+# external data plane one namespaced target without exposing Manager or
+# Keycloak directly to other namespaces.
+persistence:
+ enabled: false
+
+service:
+ http:
+ type: ClusterIP
+ https:
+ enabled: false
+ mqtts:
+ enabled: false
+
+ingress:
+ enabled: true
+
+or:
+ haproxyConfig: /etc/haproxy/haproxy-edge-terminated-tls.cfg
+ disableAcme: true
diff --git a/kubernetes/stack/exposures/none/or-setup.yaml b/kubernetes/stack/exposures/none/or-setup.yaml
new file mode 100644
index 000000000..9dce0703d
--- /dev/null
+++ b/kubernetes/stack/exposures/none/or-setup.yaml
@@ -0,0 +1 @@
+# Internal-only exposure adds no public ingress policy.
diff --git a/kubernetes/stack/targets/eks/manager.yaml b/kubernetes/stack/targets/eks/manager.yaml
new file mode 100644
index 000000000..3256ecc61
--- /dev/null
+++ b/kubernetes/stack/targets/eks/manager.yaml
@@ -0,0 +1,2 @@
+persistence:
+ storageClass: openremote-ebs
diff --git a/kubernetes/stack/targets/eks/or-setup.yaml b/kubernetes/stack/targets/eks/or-setup.yaml
new file mode 100644
index 000000000..12e4a94ac
--- /dev/null
+++ b/kubernetes/stack/targets/eks/or-setup.yaml
@@ -0,0 +1,2 @@
+networkPolicy:
+ enabled: true
diff --git a/kubernetes/stack/targets/eks/postgresql.yaml b/kubernetes/stack/targets/eks/postgresql.yaml
new file mode 100644
index 000000000..e5c58cc09
--- /dev/null
+++ b/kubernetes/stack/targets/eks/postgresql.yaml
@@ -0,0 +1,5 @@
+requiresPermissionsFix: true
+useSubPath: true
+
+persistence:
+ storageClass: openremote-ebs
diff --git a/kubernetes/stack/targets/local/or-setup.yaml b/kubernetes/stack/targets/local/or-setup.yaml
new file mode 100644
index 000000000..12e4a94ac
--- /dev/null
+++ b/kubernetes/stack/targets/local/or-setup.yaml
@@ -0,0 +1,2 @@
+networkPolicy:
+ enabled: true
diff --git a/kubernetes/stack/targets/local/postgresql.yaml b/kubernetes/stack/targets/local/postgresql.yaml
new file mode 100644
index 000000000..216ac99e8
--- /dev/null
+++ b/kubernetes/stack/targets/local/postgresql.yaml
@@ -0,0 +1,3 @@
+# Docker Desktop's dynamically provisioned hostpath volumes are initially
+# root-owned and require the PostgreSQL ownership init container.
+requiresPermissionsFix: true
diff --git a/kubernetes/test/aws-profile-test b/kubernetes/test/aws-profile-test
new file mode 100755
index 000000000..9c5e28e0a
--- /dev/null
+++ b/kubernetes/test/aws-profile-test
@@ -0,0 +1,99 @@
+#!/usr/bin/env bash
+
+# Verify profile selection at the AWS/eksctl boundary without calling AWS.
+set -Eeuo pipefail
+
+TEST_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+KUBERNETES_DIR="$(cd "$TEST_DIR/.." && pwd)"
+TEST_TEMP_DIR="$(mktemp -d "${TMPDIR:-/tmp}/aws-profile-test.XXXXXX")"
+CALL_LOG="$TEST_TEMP_DIR/calls.log"
+trap 'rm -rf "$TEST_TEMP_DIR"' EXIT
+
+fail() {
+ printf 'FAIL: %s\n' "$*" >&2
+ exit 1
+}
+
+log_call() {
+ local argument
+ printf '%s' "$1" >> "$CALL_LOG"
+ shift
+ for argument in "$@"; do
+ printf '\t%s' "$argument" >> "$CALL_LOG"
+ done
+ printf '\n' >> "$CALL_LOG"
+}
+
+check_profile() (
+ local cli="$1"
+ local environment="$2"
+ local selection="$3"
+ local expected_profile=""
+ local expected_calls=2
+ local args=(status --name test-stack --cluster test-cluster)
+
+ unset AWS_PROFILE AWS_DEFAULT_PROFILE OR_EKS_AWS_PROFILE OR_EKS_STACK_AWS_PROFILE
+ if [ "$environment" = configured ]; then
+ # These must not be promoted to --profile, which changes credential
+ # precedence. Standard AWS settings should remain available to the tools.
+ export AWS_PROFILE=ambient-profile
+ export AWS_DEFAULT_PROFILE=ambient-default
+ export OR_EKS_AWS_PROFILE=old-cluster-profile
+ export OR_EKS_STACK_AWS_PROFILE=old-stack-profile
+ fi
+
+ source "$KUBERNETES_DIR/$cli"
+ aws() { log_call aws "$@"; }
+ eksctl() { log_call eksctl "$@"; }
+
+ if [ "$cli" = or-eks-cluster ]; then
+ args=(status --name test-cluster)
+ expected_calls=3
+ fi
+ case "$selection" in
+ explicit)
+ args+=(--profile selected-profile)
+ expected_profile=selected-profile
+ ;;
+ empty)
+ args+=(--profile "")
+ ;;
+ esac
+ parse_arguments "${args[@]}"
+ AWS_REGION_NAME=eu-west-1
+ KUBE_CONTEXT=test-cluster@eu-west-1
+
+ : > "$CALL_LOG"
+ aws_eks describe-cluster --name test-cluster
+ if [ "$cli" = or-eks-cluster ]; then
+ eksctl_with_profile get cluster --name test-cluster
+ configure_kube_context >/dev/null
+ grep -F $'aws\teks\tupdate-kubeconfig' "$CALL_LOG" >/dev/null || fail \
+ "Kubeconfig refresh did not call AWS"
+ else
+ aws_cli sts get-caller-identity
+ fi
+
+ [ "$(wc -l < "$CALL_LOG" | tr -d ' ')" = "$expected_calls" ] || fail \
+ "Unexpected call count for $cli"
+ if [ -n "$expected_profile" ]; then
+ [ "$(grep -cF -- $'--profile\t'"$expected_profile" "$CALL_LOG")" = "$expected_calls" ] || fail \
+ "$cli did not pass the explicitly selected profile to every call"
+ elif grep -F -- '--profile' "$CALL_LOG" >/dev/null; then
+ fail "$cli injected a profile for environment=$environment selection=$selection"
+ fi
+ if [ "$environment" = configured ]; then
+ [ "$AWS_PROFILE" = ambient-profile ] && [ "$AWS_DEFAULT_PROFILE" = ambient-default ] || fail \
+ "$cli changed standard AWS environment settings"
+ fi
+)
+
+for cli in or-eks-cluster or-eks-stack; do
+ for environment in unset configured; do
+ for selection in omitted explicit empty; do
+ check_profile "$cli" "$environment" "$selection"
+ done
+ done
+done
+
+printf 'AWS profile tests passed\n'
diff --git a/kubernetes/test/command-stubs/helm b/kubernetes/test/command-stubs/helm
new file mode 100755
index 000000000..c38e5957e
--- /dev/null
+++ b/kubernetes/test/command-stubs/helm
@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+
+set -eu
+
+printf 'helm' >> "$OR_STACK_TEST_LOG"
+for argument in "$@"; do
+ printf '\t%s' "$argument" >> "$OR_STACK_TEST_LOG"
+done
+printf '\n' >> "$OR_STACK_TEST_LOG"
+
+if [ "${1:-}" = upgrade ] && [ "${3:-}" = "${OR_STACK_TEST_HELM_FAIL_RELEASE:-}" ]; then
+ exit 1
+fi
diff --git a/kubernetes/test/command-stubs/hostname-registry.py b/kubernetes/test/command-stubs/hostname-registry.py
new file mode 100644
index 000000000..8b54382b1
--- /dev/null
+++ b/kubernetes/test/command-stubs/hostname-registry.py
@@ -0,0 +1,41 @@
+"""Small API store for testing create/replace contention across real processes.
+
+The lock models API-server atomic writes, not a lock available to the scripts.
+Unlike a canned-success kubectl stub, this rejects stale resourceVersions and
+duplicate creates, and preserves reservations across separate lifecycle calls.
+"""
+import fcntl
+import json
+import os
+from pathlib import Path
+import sys
+
+args = sys.argv[1:]
+operation = next(word for word in args if word in ("get", "create", "replace"))
+manifest = json.load(sys.stdin) if operation != "get" else None
+state_path = os.environ.get("OR_STACK_TEST_REGISTRY_FILE")
+if os.environ.get("OR_STACK_TEST_REGISTRY_ERROR"):
+ print("Error from server (Forbidden): hostname registry access denied", file=sys.stderr)
+ sys.exit(1)
+if not state_path:
+ sys.exit(0)
+
+path = Path(state_path)
+with Path(state_path + ".lock").open("w") as lock:
+ fcntl.flock(lock, fcntl.LOCK_EX)
+ existing = json.loads(path.read_text()) if path.exists() else None
+ if operation == "get":
+ if existing:
+ print(json.dumps(existing))
+ sys.exit(0)
+ if operation == "create" and existing:
+ print("Error from server (AlreadyExists): configmap already exists", file=sys.stderr)
+ sys.exit(1)
+ if operation == "replace" and (
+ not existing or manifest["metadata"]["resourceVersion"] != existing["metadata"]["resourceVersion"]
+ ):
+ print("Error from server (Conflict): object has been modified", file=sys.stderr)
+ sys.exit(1)
+ version = int(existing["metadata"]["resourceVersion"]) + 1 if existing else 1
+ manifest["metadata"]["resourceVersion"] = str(version)
+ path.write_text(json.dumps(manifest))
diff --git a/kubernetes/test/command-stubs/kubectl b/kubernetes/test/command-stubs/kubectl
new file mode 100755
index 000000000..59fa42531
--- /dev/null
+++ b/kubernetes/test/command-stubs/kubectl
@@ -0,0 +1,184 @@
+#!/usr/bin/env bash
+
+set -eu
+
+printf 'kubectl' >> "$OR_STACK_TEST_LOG"
+for argument in "$@"; do
+ printf '\t%s' "$argument" >> "$OR_STACK_TEST_LOG"
+done
+printf '\n' >> "$OR_STACK_TEST_LOG"
+
+if [ "${1:-}" = "config" ] && [ "${2:-}" = "get-contexts" ]; then
+ if [ "${3:-}" = "${OR_STACK_TEST_CONTEXT:-test-context}" ]; then
+ printf '%s\n' "$3"
+ fi
+ exit 0
+fi
+
+EMPTY_RESOURCE_LIST='{"items":[]}'
+arguments=" $* "
+case "$arguments" in
+ *" get configmap openremote-hostnames "*|*" --namespace kube-system create -f - "*|*" --namespace kube-system replace -f - "*)
+ exec python3 "$(dirname "$0")/hostname-registry.py" "$@"
+ ;;
+ *" get namespaces "*)
+ printf '%s\n' "${OR_STACK_TEST_NAMESPACES_JSON:-$EMPTY_RESOURCE_LIST}"
+ exit 0
+ ;;
+ *" get ingress,services "*)
+ [ "${OR_STACK_TEST_ROUTES_READ_SUCCESS:-true}" = true ] || exit 1
+ printf '%s\n' "${OR_STACK_TEST_ROUTES_JSON:-$EMPTY_RESOURCE_LIST}"
+ exit 0
+ ;;
+ *" get namespace "*" -o json "*)
+ if [ -n "${OR_STACK_TEST_NAMESPACE_JSON:-}" ]; then
+ printf '%s\n' "$OR_STACK_TEST_NAMESPACE_JSON"
+ exit 0
+ fi
+ previous_argument=""
+ for argument in "$@"; do
+ if [ "$previous_argument" = namespace ]; then
+ namespace_name="$argument"
+ break
+ fi
+ previous_argument="$argument"
+ done
+ jq -cn --arg name "$namespace_name" --arg uid "${OR_STACK_TEST_NAMESPACE_UID:-uid-$namespace_name}" \
+ --arg reserved "${OR_STACK_TEST_RESERVED_HOSTNAME:-}" \
+ --arg reserved_mqtts "${OR_STACK_TEST_RESERVED_MQTTS_HOSTNAME:-}" \
+ '{metadata:{name:$name,uid:$uid,labels:{"openremote.io/stack":$name}}} |
+ if $reserved != "" then .metadata.annotations["openremote.io/reserved-hostname"]=$reserved else . end |
+ if $reserved_mqtts != "" then .metadata.annotations["openremote.io/reserved-mqtts-hostname"]=$reserved_mqtts else . end'
+ exit 0
+ ;;
+ *" get namespace "*" --ignore-not-found -o name "*)
+ if [ "${OR_STACK_TEST_NAMESPACE_EXISTS:-false}" = true ] && \
+ [ ! -f "${OR_STACK_TEST_NAMESPACE_DELETED_FILE:-/nonexistent}" ]; then
+ printf 'namespace/stack-a\n'
+ fi
+ exit 0
+ ;;
+ *" delete namespace "*)
+ if [ -n "${OR_STACK_TEST_NAMESPACE_DELETED_FILE:-}" ]; then
+ : > "$OR_STACK_TEST_NAMESPACE_DELETED_FILE"
+ fi
+ ;;
+ *" get namespace "*)
+ if [ "${OR_STACK_TEST_NAMESPACE_EXISTS:-false}" = "true" ]; then
+ case "$arguments" in
+ *" jsonpath="*)
+ namespace_name=""
+ previous_argument=""
+ for argument in "$@"; do
+ if [ "$previous_argument" = "namespace" ]; then
+ namespace_name="$argument"
+ break
+ fi
+ previous_argument="$argument"
+ done
+
+ case "$arguments" in
+ *"openremote\\.io/mqtts-hostname"*)
+ printf '%s' "${OR_STACK_TEST_MQTTS_HOSTNAME:-mqtt-stack-a.localhost}"
+ ;;
+ *"openremote\\.io/hostname"*)
+ printf '%s' "${OR_STACK_TEST_HOSTNAME:-stack-a.localhost}"
+ ;;
+ *"openremote\\.io/target"*)
+ printf '%s' "${OR_STACK_TEST_TARGET_LABEL-local}"
+ ;;
+ *"openremote\\.io/exposure"*)
+ printf '%s' "${OR_STACK_TEST_EXPOSURE_LABEL-haproxy}"
+ ;;
+ *"openremote\\.io/dns"*)
+ printf '%s' "${OR_STACK_TEST_DNS_LABEL-none}"
+ ;;
+ *)
+ if [ "${OR_STACK_TEST_NAMESPACE_LABEL+x}" = "x" ]; then
+ printf '%s' "${OR_STACK_TEST_NAMESPACE_LABEL:-}"
+ else
+ printf '%s' "$namespace_name"
+ fi
+ ;;
+ esac
+ ;;
+ esac
+ exit 0
+ fi
+ exit 1
+ ;;
+ *" get storageclass openremote-ebs "*)
+ [ "${OR_STACK_TEST_EKS_STORAGE:-true}" = "true" ]
+ ;;
+ *" get csidriver ebs.csi.aws.com "*)
+ [ "${OR_STACK_TEST_EKS_CSI:-true}" = "true" ]
+ ;;
+ *" get customresourcedefinition/policyendpoints.networking.k8s.aws "*)
+ [ "${OR_STACK_TEST_EKS_NETWORK_POLICY:-true}" = "true" ]
+ ;;
+ *" get deployment aws-load-balancer-controller "*)
+ [ "${OR_STACK_TEST_EKS_LOAD_BALANCER_CONTROLLER:-true}" = "true" ]
+ ;;
+ *" get ingressclass openremote-alb "*)
+ [ "${OR_STACK_TEST_EKS_INGRESS_CLASS:-true}" = "true" ]
+ ;;
+ *" get ingressclassparams.elbv2.k8s.aws openremote-alb "*)
+ [ "${OR_STACK_TEST_EKS_INGRESS_CLASS_PARAMS:-true}" = "true" ] || exit 1
+ printf '%s' "${OR_STACK_TEST_EKS_SSL_REDIRECT_PORT:-443}"
+ ;;
+ *" get daemonset aws-node "*)
+ printf '%s' "${OR_STACK_TEST_EKS_CNI_CONTAINERS:-aws-node aws-eks-nodeagent}"
+ ;;
+ *" get service kube-dns "*)
+ [ "${OR_STACK_TEST_KUBE_DNS_EXISTS:-true}" = "true" ] || exit 1
+ printf '%s' "${OR_STACK_TEST_KUBE_DNS:-10.96.0.10:53}"
+ ;;
+ *" get service proxy "*"spec.ports"*"mqtts"*)
+ printf '%s' "${OR_STACK_TEST_PROXY_MQTTS_PORT:-}"
+ ;;
+ *" get service proxy-mqtts "*"spec.ports"*"mqtts"*)
+ printf '%s' "${OR_STACK_TEST_PROXY_MQTTS_PORT:-}"
+ ;;
+ *" get service proxy-mqtts "*)
+ [ "${OR_STACK_TEST_PROXY_MQTTS_SERVICE_EXISTS:-true}" = "true" ] || exit 1
+ printf '%s' "${OR_STACK_TEST_PROXY_MQTTS_ADDRESS:-localhost}"
+ ;;
+ *" get service proxy "*)
+ [ "${OR_STACK_TEST_PROXY_SERVICE_EXISTS:-true}" = "true" ] || exit 1
+ printf '%s' "${OR_STACK_TEST_PROXY_ADDRESS:-localhost}"
+ ;;
+ *" exec deployment/proxy "*)
+ [ "${OR_STACK_TEST_PROXY_CERTIFICATE_EXISTS:-true}" = "true" ] || exit 1
+ printf '%s\n' "${OR_STACK_TEST_PROXY_CERTIFICATE:-issuer=CN=OpenRemote Test CA
+notAfter=Sep 3 12:00:00 2027 GMT}"
+ ;;
+ *" get secret openremote-secret "*)
+ [ "${OR_STACK_TEST_SECRET_EXISTS:-true}" = "true" ] || exit 1
+ case "$arguments" in
+ *"helm\\.sh/resource-policy"*)
+ printf '%s' "${OR_STACK_TEST_SECRET_POLICY-keep}"
+ ;;
+ *"base64decode"*)
+ printf '%s' "${OR_STACK_TEST_ADMIN_PASSWORD:-generated-admin-password}"
+ ;;
+ esac
+ ;;
+ *" get persistentvolumeclaims "*"metadata.name"*)
+ printf '%s\n' "${OR_STACK_TEST_PVC_METADATA:-manager|manager|keep
+postgresql|postgresql|keep
+proxy|proxy|keep}"
+ ;;
+ *" get persistentvolumeclaims "*"spec.volumeName"*)
+ printf '%s\n' "${OR_STACK_TEST_PV_NAMES:-pvc-manager
+pvc-postgresql
+pvc-proxy}"
+ ;;
+ *" get persistentvolume/"*)
+ [ "${OR_STACK_TEST_PV_EXISTS:-true}" = "true" ]
+ ;;
+ *" wait --for=delete persistentvolume/"*)
+ [ "${OR_STACK_TEST_PV_WAIT_SUCCESS:-true}" = "true" ]
+ ;;
+esac
+
+exit 0
diff --git a/kubernetes/test/hostname-reservations-test b/kubernetes/test/hostname-reservations-test
new file mode 100644
index 000000000..f59f5aab4
--- /dev/null
+++ b/kubernetes/test/hostname-reservations-test
@@ -0,0 +1,223 @@
+#!/usr/bin/env python3
+"""Exercise portable lifecycle commands against a persistent, atomic API stub."""
+import concurrent.futures
+import json
+import os
+from pathlib import Path
+import subprocess
+import tempfile
+import unittest
+
+KUBERNETES = Path(__file__).resolve().parent.parent
+
+
+class HostnameReservationsTest(unittest.TestCase):
+ def setUp(self):
+ self.directory = tempfile.TemporaryDirectory(prefix="or-hostname-test-")
+ self.addCleanup(self.directory.cleanup)
+ root = Path(self.directory.name)
+ self.registry = root / "registry.json"
+ self.log = root / "calls.log"
+ self.env = dict(os.environ)
+ self.env.update(
+ PATH=str(KUBERNETES / "test/command-stubs") + os.pathsep + self.env["PATH"],
+ OR_STACK_TEST_LOG=str(self.log),
+ OR_STACK_TEST_REGISTRY_FILE=str(self.registry),
+ OR_STACK_TEST_NAMESPACE_DELETED_FILE=str(root / "namespace-deleted"),
+ )
+
+ def command(self, *args, **env):
+ return subprocess.run(
+ [str(KUBERNETES / "or-stack"), *args, "--kube-context", "test-context"],
+ env=self.env | env, text=True, capture_output=True,
+ )
+
+ def apply(self, stack="stack-a", host="a.example.com", target="eks", exposure="haproxy", **env):
+ args = ["apply", "--name", stack, "--hostname", host, "--target", target, "--exposure", exposure]
+ if target == "eks" and exposure == "ingress":
+ args += ["--certificate-arn", "arn:aws:acm:eu-west-1:123456789012:certificate/test"]
+ return self.command(*args, **env)
+
+ def apply_ingress_mqtts(self, stack="stack-a", web_host="a.example.com",
+ mqtts_host="mqtt-a.example.com", **env):
+ return self.command(
+ "apply", "--name", stack, "--target", "eks", "--exposure", "ingress",
+ "--hostname", web_host,
+ "--certificate-arn", "arn:aws:acm:eu-west-1:123456789012:certificate/web",
+ "--mqtts", "--mqtts-hostname", mqtts_host,
+ "--mqtts-certificate-arn",
+ "arn:aws:acm:eu-west-1:123456789012:certificate/mqtts",
+ **env,
+ )
+
+ def success(self, result):
+ self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+
+ def failure(self, result, message):
+ self.assertNotEqual(result.returncode, 0, result.stdout)
+ self.assertIn(message, result.stderr)
+
+ def claims(self):
+ return {key: json.loads(value) for key, value in json.loads(self.registry.read_text())["data"].items()}
+
+ def test_prepare_without_namespace_annotations(self):
+ for annotations in ({}, {"annotations": {}}, {"annotations": None}):
+ with self.subTest(annotations=annotations):
+ namespace = {"metadata": {"name": "stack-a", "uid": "uid-stack-a",
+ "labels": {"openremote.io/stack": "stack-a"}, **annotations}}
+ self.success(self.command("prepare", "--name", "stack-a", "--target", "eks",
+ "--exposure", "ingress", "--dns", "external-dns",
+ "--hostname", "a.example.com",
+ OR_STACK_TEST_NAMESPACE_JSON=json.dumps(namespace)))
+ self.assertEqual(self.claims()["a.example.com"]["namespaceUID"], "uid-stack-a")
+ self.assertNotIn("helm\t", self.log.read_text())
+
+ def test_duplicate_and_immutable_hostname(self):
+ self.success(self.apply(host="A.EXAMPLE.COM."))
+ self.log.write_text("")
+ self.failure(self.apply("stack-b"), "conflicting reservation")
+ self.assertNotIn("helm\t", self.log.read_text())
+ self.failure(self.apply(host="other.example.com"), "conflicting reservation")
+ self.success(self.apply())
+ self.assertEqual(list(self.claims()), ["a.example.com"])
+
+ def test_concurrent_claims_have_one_winner(self):
+ with concurrent.futures.ThreadPoolExecutor() as pool:
+ results = list(pool.map(lambda name: self.apply(name), ["stack-a", "stack-b"]))
+ self.assertEqual(sorted(result.returncode for result in results), [0, 1])
+ self.assertEqual(len(self.claims()), 1)
+
+ def test_concurrent_different_hosts_for_one_stack_have_one_winner(self):
+ with concurrent.futures.ThreadPoolExecutor() as pool:
+ results = list(pool.map(lambda host: self.apply(host=host), ["a.example.com", "b.example.com"]))
+ self.assertEqual(sorted(result.returncode for result in results), [0, 1])
+ self.assertEqual(len(self.claims()), 1)
+
+ def test_concurrent_independent_stacks_preserve_both_claims(self):
+ with concurrent.futures.ThreadPoolExecutor() as pool:
+ results = list(pool.map(lambda n: self.apply("stack-" + n, n + ".example.com"), ["a", "b"]))
+ for result in results:
+ self.success(result)
+ self.assertEqual(len(self.claims()), 2)
+
+ def test_stack_reserves_independent_web_and_mqtts_hostnames(self):
+ self.success(self.apply_ingress_mqtts())
+ claims = self.claims()
+ self.assertEqual(set(claims), {"a.example.com", "mqtt-a.example.com"})
+ self.assertEqual(claims["a.example.com"]["endpoint"], "web")
+ self.assertEqual(claims["mqtt-a.example.com"]["endpoint"], "mqtts")
+ self.success(self.apply_ingress_mqtts())
+ self.failure(self.apply_ingress_mqtts(mqtts_host="other-mqtt.example.com"),
+ "conflicting reservation")
+ self.failure(self.apply("stack-b", host="mqtt-a.example.com"),
+ "conflicting reservation")
+
+ def test_destroy_retains_and_explicitly_releases_both_public_hostnames(self):
+ self.success(self.apply_ingress_mqtts())
+ self.success(self.command("destroy", "--name", "stack-a", "--confirm", "stack-a",
+ OR_STACK_TEST_NAMESPACE_EXISTS="true"))
+ self.assertEqual(len(self.claims()), 2)
+ for hostname in ("a.example.com", "mqtt-a.example.com"):
+ self.success(self.command("release-hostname", "--name", "stack-a",
+ "--hostname", hostname, "--confirm", "stack-a"))
+ self.assertEqual(self.claims(), {})
+
+ def test_failed_apply_and_uninstall_retain_claim(self):
+ self.assertNotEqual(self.apply(OR_STACK_TEST_HELM_FAIL_RELEASE="manager").returncode, 0)
+ self.failure(self.apply("stack-b"), "conflicting reservation")
+ self.success(self.apply())
+ self.success(self.command("uninstall", "--name", "stack-a", OR_STACK_TEST_NAMESPACE_EXISTS="true"))
+ self.assertEqual(len(self.claims()), 1)
+ self.success(self.apply())
+
+ def test_namespace_recreation_cannot_steal_claim(self):
+ self.success(self.apply())
+ self.failure(self.apply(OR_STACK_TEST_NAMESPACE_UID="new-uid"), "conflicting reservation")
+
+ def test_destroy_rejects_recreated_namespace(self):
+ self.success(self.apply())
+ self.log.write_text("")
+ self.failure(self.command("destroy", "--name", "stack-a", "--confirm", "stack-a",
+ OR_STACK_TEST_NAMESPACE_EXISTS="true",
+ OR_STACK_TEST_NAMESPACE_UID="new-uid"), "different namespace UID")
+ self.assertNotIn("helm\tuninstall", self.log.read_text())
+ self.assertNotIn("delete\tnamespace", self.log.read_text())
+ self.assertEqual(len(self.claims()), 1)
+
+ def test_explicit_release_guards_and_reuse(self):
+ self.success(self.apply())
+ args = ("release-hostname", "--name", "stack-a", "--hostname", "a.example.com")
+ self.failure(self.command(*args), "Release requires --confirm")
+ self.failure(self.command(*args, "--confirm", "stack-a", OR_STACK_TEST_NAMESPACE_EXISTS="true"), "still exists")
+ self.failure(self.command("release-hostname", "--name", "stack-b", "--hostname", "a.example.com", "--confirm", "stack-b"), "another stack")
+ self.success(self.command(*args, "--confirm", "stack-a"))
+ self.success(self.command(*args, "--confirm", "stack-a"))
+ self.success(self.apply("stack-b"))
+
+ def test_destroy_retains_public_eks_claim(self):
+ self.success(self.apply())
+ self.success(self.command("destroy", "--name", "stack-a", "--confirm", "stack-a", OR_STACK_TEST_NAMESPACE_EXISTS="true"))
+ self.assertEqual(len(self.claims()), 1)
+
+ def test_local_ingress_destroy_releases_claim(self):
+ self.success(self.apply(target="local", exposure="ingress"))
+ self.success(self.command("destroy", "--name", "stack-a", "--confirm", "stack-a", OR_STACK_TEST_NAMESPACE_EXISTS="true"))
+ self.assertEqual(self.claims(), {})
+
+ def test_local_haproxy_and_internal_stacks_do_not_claim(self):
+ self.success(self.apply(host="localhost", target="local"))
+ self.success(self.apply("stack-b", host="localhost", target="local"))
+ self.success(self.apply(exposure="none"))
+ self.assertFalse(self.registry.exists())
+
+ def test_disabling_external_dns_retains_cleanup_obligation(self):
+ args = ("prepare", "--name", "stack-a", "--target", "local", "--hostname", "a.example.com")
+ self.success(self.command(*args, "--dns", "external-dns"))
+ # The stub has no reserved-hostname annotation, modelling an interrupted
+ # apply after the registry write but before the annotation was recorded.
+ self.failure(self.apply(host="other.example.com", target="local"), "hostname changes")
+ self.success(self.command(*args, "--dns", "none"))
+ self.assertEqual(self.claims()["a.example.com"]["dns"], "external-dns")
+ self.success(self.command("destroy", "--name", "stack-a", "--confirm", "stack-a",
+ OR_STACK_TEST_NAMESPACE_EXISTS="true"))
+ self.assertEqual(len(self.claims()), 1)
+
+ def test_existing_routes_block_claims_and_release(self):
+ for kind, spec, annotations in [
+ ("Ingress", {"rules": [{"host": "a.example.com"}]}, {}),
+ ("Ingress", {"rules": [{"host": "*.example.com"}]}, {}),
+ ("Service", {}, {"external-dns.alpha.kubernetes.io/hostname": "other.example.com, A.EXAMPLE.COM."}),
+ ("Service", {}, {"external-dns.kubernetes.io/hostname": "a.example.com"}),
+ ]:
+ routes = json.dumps({"items": [{"kind": kind, "metadata": {"namespace": "another-workload", "name": "proxy", "annotations": annotations}, "spec": spec}]})
+ self.failure(self.apply(OR_STACK_TEST_ROUTES_JSON=routes), "existing routing")
+ self.assertFalse(self.registry.exists())
+ self.success(self.apply())
+ self.failure(self.command("release-hostname", "--name", "stack-a", "--hostname", "a.example.com", "--confirm", "stack-a", OR_STACK_TEST_ROUTES_JSON=routes), "existing routing")
+
+ def test_namespace_reservation_rejects_hostname_change(self):
+ self.failure(self.apply(host="b.example.com", OR_STACK_TEST_RESERVED_HOSTNAME="a.example.com"), "hostname changes")
+ self.assertFalse(self.registry.exists())
+
+ def test_unrelated_wildcard_does_not_block_claim(self):
+ routes = json.dumps({"items": [{"kind": "Ingress", "metadata": {"namespace": "another-workload", "name": "proxy"},
+ "spec": {"rules": [{"host": "*.other.example.com"}]}}]})
+ self.success(self.apply(OR_STACK_TEST_ROUTES_JSON=routes))
+
+ def test_another_namespaces_reservation_blocks_claim(self):
+ namespaces = json.dumps({"items": [{"metadata": {"name": "stack-b", "annotations": {"openremote.io/reserved-hostname": "a.example.com"}}}]})
+ self.failure(self.apply(OR_STACK_TEST_NAMESPACES_JSON=namespaces), "already recorded")
+
+ def test_api_errors_fail_closed_before_helm(self):
+ for env in [{"OR_STACK_TEST_REGISTRY_ERROR": "true"}, {"OR_STACK_TEST_ROUTES_READ_SUCCESS": "false"}]:
+ self.log.write_text("")
+ self.assertNotEqual(self.apply(**env).returncode, 0)
+ self.assertNotIn("helm\t", self.log.read_text())
+
+ def test_unowned_registry_is_not_adopted(self):
+ self.registry.write_text(json.dumps({"metadata": {"resourceVersion": "1"}, "data": {}}))
+ self.failure(self.apply(), "unowned or malformed")
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/kubernetes/test/keycloak-url-test b/kubernetes/test/keycloak-url-test
new file mode 100644
index 000000000..bf3686cd1
--- /dev/null
+++ b/kubernetes/test/keycloak-url-test
@@ -0,0 +1,36 @@
+#!/usr/bin/env python3
+"""Render Keycloak URLs for CLI and legacy Helm values without a cluster."""
+
+from pathlib import Path
+import re
+import subprocess
+import unittest
+
+
+CHART = Path(__file__).resolve().parents[1] / "keycloak"
+
+
+class KeycloakUrlTest(unittest.TestCase):
+ def test_public_urls(self):
+ cases = (
+ ("localhost", "https://localhost/auth"),
+ ("load2.openremote.app", "https://load2.openremote.app/auth"),
+ ("localhost:9443", "https://localhost:9443/auth"),
+ ("https://load2.openremote.app/auth", "https://load2.openremote.app/auth"),
+ ("https://localhost:8443/auth", "https://localhost:8443/auth"),
+ ("http://localhost:8080/auth", "http://localhost:8080/auth"),
+ ("https://login.example.com/custom", "https://login.example.com/custom"),
+ )
+ for hostname, expected in cases:
+ with self.subTest(hostname=hostname):
+ result = subprocess.run(
+ ["helm", "template", "keycloak", str(CHART),
+ "--set-string", f"or.hostname={hostname}"],
+ capture_output=True, text=True, check=True,
+ )
+ urls = re.findall(r'- name: KC_HOSTNAME\s+value: "([^"]+)"', result.stdout)
+ self.assertEqual(urls, [expected])
+
+
+if __name__ == "__main__":
+ unittest.main(verbosity=2)
diff --git a/kubernetes/test/legacy-haproxy-cleanup-test b/kubernetes/test/legacy-haproxy-cleanup-test
new file mode 100644
index 000000000..54f0a8502
--- /dev/null
+++ b/kubernetes/test/legacy-haproxy-cleanup-test
@@ -0,0 +1,116 @@
+#!/usr/bin/env bash
+
+# Exercise the legacy cleanup script with local command stubs only.
+set -Eeuo pipefail
+
+TEST_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+KUBERNETES_DIR="$(cd "$TEST_DIR/.." && pwd)"
+TEST_TEMP_DIR="$(mktemp -d "${TMPDIR:-/tmp}/legacy-haproxy-cleanup-test.XXXXXX")"
+export CALL_LOG="$TEST_TEMP_DIR/calls.log"
+trap 'rm -rf "$TEST_TEMP_DIR"' EXIT
+
+fail() {
+ printf 'FAIL: %s\n' "$*" >&2
+ exit 1
+}
+
+# The script sources ./eks-common.sh. Supply an isolated replacement so no AWS
+# configuration, credentials, or real cluster commands are touched.
+cat > "$TEST_TEMP_DIR/eks-common.sh" <<'SH'
+set -eo pipefail
+CLUSTER_NAME=test-cluster
+FQDN=test.example.com
+
+log_call() {
+ local argument
+ printf '%s' "$1" >> "$CALL_LOG"
+ shift
+ for argument in "$@"; do
+ printf '\t%s' "$argument" >> "$CALL_LOG"
+ done
+ printf '\n' >> "$CALL_LOG"
+}
+
+aws() {
+ log_call aws "$@"
+ case "$1 $2" in
+ "eks describe-cluster") printf 'vpc-test\n' ;;
+ "elbv2 describe-load-balancers") printf '[]\n' ;;
+ "ec2 describe-vpc-peering-connections"|"route53 change-resource-record-sets") ;;
+ *) return 99 ;;
+ esac
+}
+
+helm() { log_call helm "$@"; }
+eksctl() { log_call eksctl "$@"; }
+
+kubectl() {
+ log_call kubectl "$@"
+ case "$1 $2" in
+ "get pvc")
+ if [ "$3" = proxy ]; then
+ case "${PROXY_STATE:-bound}" in
+ missing) return 1 ;;
+ unbound) return 0 ;;
+ esac
+ fi
+ printf 'pvc-%s-test\n' "$3"
+ ;;
+ "get pv/"*) ;;
+ "delete pvc") ;;
+ "wait --for=delete")
+ if [ "$3" = pv/pvc-proxy-test ] && [ "${PROXY_WAIT_FAIL:-false}" = true ]; then
+ return 1
+ fi
+ ;;
+ *) return 99 ;;
+ esac
+}
+SH
+
+run_cleanup() (
+ cd "$TEST_TEMP_DIR"
+ bash "$KUBERNETES_DIR/eks-cleanup-haproxy.sh"
+)
+
+assert_before() {
+ local first_line
+ local second_line
+ first_line="$(grep -nF -- "$1" "$CALL_LOG" | head -n 1 | cut -d: -f1)" || fail \
+ "Missing command: $1"
+ second_line="$(grep -nF -- "$2" "$CALL_LOG" | head -n 1 | cut -d: -f1)" || fail \
+ "Missing command: $2"
+ [ "$first_line" -lt "$second_line" ] || fail \
+ "Expected '$1' before '$2'"
+}
+
+delete_claims=$'kubectl\tdelete\tpvc\tmanager\tpostgresql\tproxy\t--ignore-not-found\t--wait=true'
+delete_driver=$'eksctl\tdelete\taddon\t--cluster\ttest-cluster\t--name\taws-ebs-csi-driver'
+
+run_cleanup > "$TEST_TEMP_DIR/output.log" 2>&1 || fail "Cleanup failed"
+for component in manager postgresql proxy; do
+ assert_before $'kubectl\tget\tpvc\t'"$component"$'\t' $'helm\tuninstall\t'"$component"
+ assert_before $'helm\tuninstall\t'"$component" "$delete_claims"
+ assert_before "$delete_claims" $'kubectl\twait\t--for=delete\tpv/pvc-'"$component"$'-test\t'
+ assert_before $'kubectl\twait\t--for=delete\tpv/pvc-'"$component"$'-test\t' "$delete_driver"
+done
+
+for state in missing unbound; do
+ : > "$CALL_LOG"
+ PROXY_STATE="$state" run_cleanup > "$TEST_TEMP_DIR/output.log" 2>&1 || fail \
+ "Cleanup failed with a $state proxy claim"
+ assert_before "$delete_claims" "$delete_driver"
+ if grep -F -- $'kubectl\twait\t--for=delete\tpv/pvc-proxy-test' "$CALL_LOG" >/dev/null; then
+ fail "Cleanup waited for a volume belonging to a $state proxy claim"
+ fi
+done
+
+: > "$CALL_LOG"
+if PROXY_WAIT_FAIL=true run_cleanup > "$TEST_TEMP_DIR/output.log" 2>&1; then
+ fail "Cleanup succeeded despite the proxy volume deletion failing"
+fi
+if grep -F -- $'eksctl\tdelete' "$CALL_LOG" >/dev/null; then
+ fail "Cleanup removed cluster infrastructure before the proxy volume was deleted"
+fi
+
+printf 'Legacy HAProxy cleanup tests passed\n'
diff --git a/kubernetes/test/load-eks-test b/kubernetes/test/load-eks-test
new file mode 100644
index 000000000..ac1cf1ab7
--- /dev/null
+++ b/kubernetes/test/load-eks-test
@@ -0,0 +1,317 @@
+#!/usr/bin/env python3
+"""Offline load-profile checks using the real CLI and Helm, with kubectl stubbed.
+
+Requires Python 3, Bash, Helm, jq, and envsubst, like the existing CLI tests.
+No AWS credentials or Kubernetes cluster are used.
+"""
+
+import os
+import json
+from pathlib import Path
+import re
+import shutil
+import shlex
+import subprocess
+import tempfile
+
+
+ROOT = Path(__file__).resolve().parents[2]
+KUBERNETES = ROOT / "kubernetes"
+STUBS = KUBERNETES / "test/command-stubs"
+HELM = shutil.which("helm")
+REPOSITORY = "123456789012.dkr.ecr.eu-west-1.amazonaws.com/openremote/manager"
+CERTIFICATE = "arn:aws:acm:eu-west-1:123456789012:certificate/load-test"
+PROFILES = [ROOT / "test/load1-eks", *sorted((ROOT / "test/load2-eks/profiles").iterdir())]
+
+
+def run(args, **kwargs):
+ result = subprocess.run(args, text=True, capture_output=True, **kwargs)
+ if result.returncode:
+ raise AssertionError(f"Command failed: {args}\n{result.stdout}\n{result.stderr}")
+ return result.stdout
+
+
+def contains(text, expected):
+ assert expected in text, f"Missing {expected!r} in:\n{text}"
+
+
+def documents(text, kind):
+ return [doc for doc in text.split("\n---") if f"\nkind: {kind}\n" in doc]
+
+
+def helm_options(arguments):
+ """Reuse the exact values/set options chosen by or-stack, without a cluster."""
+ result = []
+ index = 0
+ while index < len(arguments):
+ arg = arguments[index]
+ if arg == "--wait":
+ index += 1
+ elif arg in ("--kube-context", "--timeout"):
+ index += 2
+ else:
+ assert arg in ("--namespace", "--values", "--set", "--set-string"), arg
+ result.extend(arguments[index:index + 2])
+ index += 2
+ return result
+
+
+def check_profile(profile, temporary):
+ load = "load1" if profile.name == "load1-eks" else "load2"
+ name = "load1" if load == "load1" else f"load2-{profile.name}"
+ values = temporary / name
+ env = {key: value for key, value in os.environ.items()
+ if not key.startswith(("OR_STACK_", "OR_EKS_"))}
+ env.update(LOAD_MANAGER_REPOSITORY=REPOSITORY, CLUSTER_NAME=f"{name}-cluster",
+ AWS_REGION="eu-west-1")
+ config = temporary / f"{name}.env"
+ settings = {"LOAD_TEST": load, "LOAD_MANAGER_REPOSITORY": REPOSITORY}
+ if load == "load2":
+ settings["LOAD_PROFILE"] = profile.name
+ config.write_text("".join(f"{key}={shlex.quote(value)}\n" for key, value in settings.items()))
+ preparation = [str(ROOT / "test/or-eks-load"), "prepare", "--config", str(config),
+ "--values-dir", str(values)]
+ run(preparation, env=env)
+ for component in ("manager", "postgresql", "keycloak", "proxy"):
+ assert "${" not in (values / f"{component}.yaml").read_text()
+ cluster = run(["envsubst", "${CLUSTER_NAME} ${AWS_REGION}"],
+ input=(profile / "cluster.yaml").read_text(), env=env)
+ contains(cluster, f"name: {name}-cluster")
+ contains(cluster, "region: eu-west-1")
+ contains(cluster, 'configurationValues: \'{"enableNetworkPolicy":"true"}\'')
+ contains(cluster, "name: aws-ebs-csi-driver")
+ contains(cluster, "withOIDC: true")
+ assert "${" not in cluster
+
+ for exposure in ("haproxy", "ingress", "none"):
+ namespace = f"{name}-{exposure}"
+ hostname = f"{namespace}.example.com"
+ call_log = temporary / f"{namespace}.log"
+ env.update(PATH=f"{STUBS}:{os.environ['PATH']}", OR_STACK_TEST_LOG=str(call_log),
+ OR_STACK_TEST_CONTEXT="test-context", OR_STACK_TEST_NAMESPACE_EXISTS="false")
+ args = [str(KUBERNETES / "or-stack"), "apply", "--name", namespace,
+ "--kube-context", "test-context", "--target", "eks",
+ "--exposure", exposure, "--values-dir", str(values), "--timeout", "90m"]
+ if exposure != "none":
+ args += ["--hostname", hostname, "--dns", "external-dns", "--mqtts"]
+ if exposure == "ingress":
+ args += ["--certificate-arn", CERTIFICATE, "--mqtts-certificate-arn", CERTIFICATE,
+ "--mqtts-hostname", f"mqtt-{hostname}"]
+ run(args, env=env)
+ releases = {}
+ for line in call_log.read_text().splitlines():
+ command = line.split("\t")
+ if command[:3] != ["helm", "upgrade", "--install"]:
+ continue
+ component, chart = command[3:5]
+ options = helm_options(command[5:])
+ assert options[options.index("--namespace") + 1] == namespace
+ if component != "or-setup":
+ assert str(values / f"{component}.yaml") in options
+ run([HELM, "lint", chart, "--strict", *options])
+ releases[component] = run([HELM, "template", component, chart, *options])
+
+ expected = {"or-setup", "postgresql", "keycloak", "manager"}
+ if exposure != "none":
+ expected.add("proxy")
+ assert releases.keys() == expected
+ all_manifests = "\n".join(releases.values())
+ assert not documents(all_manifests, "PersistentVolume")
+ assert "namespace: default" not in all_manifests
+ assert "volumeName:" not in all_manifests
+ assert "kind: NetworkPolicy" in releases["or-setup"]
+ for component in ("manager", "postgresql"):
+ size = "16Gi" if load == "load1" else ("32Gi" if component == "manager" else "256Gi")
+ pvc, = documents(releases[component], "PersistentVolumeClaim")
+ contains(pvc, f"storage: {size}")
+ contains(pvc, 'storageClassName: "openremote-ebs"')
+ contains(pvc, "helm.sh/resource-policy: keep")
+ for component in ("manager", "keycloak"):
+ deployment, = documents(releases[component], "Deployment")
+ contains(deployment, "type: Recreate")
+ # Preserve each profile's resource budget through CLI layering.
+ requests = (values / f"{component}.yaml").read_text().split(" requests:\n", 1)[1]
+ for resource, value in re.findall(r"^ (cpu|memory): (\S+)", requests, re.M):
+ contains(deployment, f"{resource}: {value}")
+ manager = releases["manager"]
+ contains(manager, f'image: "{REPOSITORY}:{load}"')
+ contains(manager, "imagePullPolicy: Always")
+ contains(manager, "name: OR_SETUP_REGULAR_USERS" if load == "load1" else "name: OR_SETUP_USERS")
+ assert re.search(r'name: OR_SETUP_RUN_ON_RESTART\s+value: "false"', manager)
+ if load == "load2":
+ contains(manager, "name: manager-jmx")
+ assert not documents(manager, "Ingress")
+ assert "type: LoadBalancer" not in manager
+ if exposure != "none":
+ contains(releases["keycloak"], f'value: "https://{hostname}/auth"')
+ assert re.search(r'name: OR_HOSTNAME\s+value: "' + re.escape(hostname) + '"', manager)
+ proxy = releases["proxy"]
+ contains(proxy, f"manager.{namespace}.svc.cluster.local")
+ contains(proxy, f"keycloak.{namespace}.svc.cluster.local")
+ contains(proxy, "port: 8883")
+ contains(proxy, f'external-dns.kubernetes.io/hostname: "{hostname}"')
+ if exposure == "haproxy":
+ assert not documents(proxy, "Ingress")
+ assert len(documents(proxy, "PersistentVolumeClaim")) == 1
+ assert "name: proxy-mqtts" not in proxy
+ else:
+ assert len(documents(proxy, "Ingress")) == 1
+ assert not documents(proxy, "PersistentVolumeClaim")
+ contains(proxy, "name: proxy-mqtts")
+ contains(proxy, f'external-dns.kubernetes.io/hostname: "mqtt-{hostname}"')
+ else:
+ assert not documents(all_manifests, "Ingress")
+ assert "type: LoadBalancer" not in all_manifests
+ print(f"PASS: {name} / {exposure}", flush=True)
+
+
+def check_legacy_scripts(temporary):
+ for folder in (ROOT / "test/load1-eks", ROOT / "test/load2-eks"):
+ for script in sorted(folder.glob("eks-*.sh")):
+ run(["bash", "-n", str(script)])
+ if folder.name == "load2-eks" and script.name != "eks-setup-load.sh":
+ continue
+ # No external commands are available: migration guidance must fail
+ # before attempting AWS, Kubernetes, or credential changes.
+ result = subprocess.run(["/bin/bash", str(script)], text=True,
+ capture_output=True, cwd="/", env={"PATH": "/nonexistent"})
+ assert result.returncode == 1
+ contains(result.stderr, "or-eks-cluster")
+ contains(result.stderr, "or-eks-stack")
+ assert "command not found" not in result.stderr
+
+ # Exercise the retained experiment with command stubs, then render its
+ # actual Helm arguments. Never invoke real AWS or Kubernetes commands.
+ binaries = temporary / "legacy-bin"
+ binaries.mkdir()
+ call_log = temporary / "legacy-calls.jsonl"
+ stub = binaries / "stub"
+ stub.write_text('''#!/usr/bin/env python3
+import json, os, sys
+from pathlib import Path
+command = Path(sys.argv[0]).name
+args = sys.argv[1:]
+with open(os.environ["LEGACY_CALL_LOG"], "a") as log:
+ log.write(json.dumps([command, *args]) + "\\n")
+if command == "eksctl" and "create" in args:
+ config = sys.stdin.read()
+ assert "name: legacy-test" in config and "${" not in config
+elif command == "kubectl":
+ if "kube-dns" in args:
+ print("10.100.0.10:53")
+ if "namespaces" in args and os.environ.get("CLI_STACK_PRESENT"):
+ print("namespace/another-stack")
+elif command == "dig":
+ print("192.0.2.10")
+elif command == "aws":
+ if "assume-role" in args:
+ print("fake-key\\tfake-secret\\tfake-token")
+ elif "describe-cluster" in args:
+ print("vpc-legacy")
+ elif "request-certificate" in args:
+ print("arn:aws:acm:eu-west-1:123456789012:certificate/legacy")
+ elif "describe-certificate" in args:
+ print("_validation.example.com")
+ elif "describe-load-balancers" in args:
+ query = args[args.index("--query") + 1]
+ if "CanonicalHostedZoneId" in query:
+ print('"Z12345678"')
+ elif "DNSName" in query:
+ print('"legacy.elb.amazonaws.com"')
+ else:
+ print('{"Code": "active"}')
+''')
+ stub.chmod(0o755)
+ for command in ("aws", "eksctl", "kubectl", "helm", "dig"):
+ (binaries / command).symlink_to(stub)
+ env = dict(PATH=f"{binaries}:{os.environ['PATH']}", LEGACY_CALL_LOG=str(call_log),
+ AWS_ACCOUNT_ID="123456789012", AWS_DEVELOPERS_ACCOUNT_ID="210987654321",
+ CLUSTER_NAME="legacy-test", AWS_REGION="eu-west-1", OR_PROFILE="large",
+ FQDN="legacy.example.com", K_CONTEXT="legacy-test@eu-west-1")
+ folder = ROOT / "test/load2-eks"
+ for profile in PROFILES:
+ if profile.name == "load1-eks":
+ continue
+ env["OR_PROFILE"] = profile.name
+ for script in ("eks-setup-load-acm.sh", "eks-deploy-load.sh"):
+ call_log.write_text("")
+ run(["bash", f"./{script}"], cwd=folder, env=env)
+ calls = [json.loads(line) for line in call_log.read_text().splitlines()]
+ setup = script == "eks-setup-load-acm.sh"
+ assert not any("create-volume" in call for call in calls)
+ if setup:
+ assert any("certificate-validated" in call for call in calls)
+ assert any("apply" in call and any("storage-class.yaml" in arg for arg in call) for call in calls)
+ rendered = {}
+ for call in calls:
+ if call[0] == "kubectl":
+ assert call[call.index("--context") + 1] == env["K_CONTEXT"]
+ if call[0] != "helm" or call[1] not in ("install", "upgrade"):
+ continue
+ # Both --kube-context=value and --kube-context value are used.
+ call = [part for arg in call for part in
+ (arg.split("=", 1) if arg.startswith("--kube-context=") else [arg])]
+ assert call[call.index("--kube-context") + 1] == env["K_CONTEXT"]
+ if "aws-load-balancer-controller" in call:
+ continue
+ assert call[call.index("--namespace") + 1] == "default"
+ component_index = next(i for i, arg in enumerate(call)
+ if arg in ("or-setup", "postgresql", "manager", "keycloak", "proxy"))
+ component, chart = call[component_index:component_index + 2]
+ options = call[component_index + 2:]
+ if component != "or-setup":
+ shared_values = f"profiles/{profile.name}/{component}.yaml"
+ assert shared_values in options
+ if component == "proxy":
+ assert options.index(shared_values) < options.index("values-proxy-acm-load.yaml")
+ run([HELM, "lint", chart, "--strict", *options], cwd=folder)
+ rendered[component] = run([HELM, "template", component, chart, *options], cwd=folder)
+ assert "${LOAD_MANAGER_REPOSITORY}" not in rendered[component]
+ if component != "or-setup":
+ requests = (profile / f"{component}.yaml").read_text().split(" requests:\n", 1)[1]
+ for resource, value in re.findall(r"^ (cpu|memory): (\S+)", requests, re.M):
+ contains(rendered[component], f"{resource}: {value}")
+ if component in ("manager", "keycloak"):
+ contains(rendered[component], "type: Recreate")
+ expected = {"postgresql", "manager", "keycloak"}
+ if setup:
+ expected |= {"or-setup", "proxy"}
+ proxy = rendered["proxy"]
+ assert not documents(proxy, "PersistentVolumeClaim")
+ assert not documents(proxy, "Ingress")
+ contains(proxy, 'aws-load-balancer-ssl-ports: 443,8883')
+ assert re.search(r"port: 443\s+targetPort: 8080", proxy)
+ assert re.search(r"port: 8883\s+targetPort: 1883", proxy)
+ contains(proxy, "haproxy-edge-terminated-tls.cfg")
+ assert proxy.count("name: DISABLE_ACME") == 1
+ assert rendered.keys() == expected
+ for component in ("manager", "postgresql"):
+ size = re.search(r'persistence:\n size: "([^"\n]+)"',
+ (profile / f"{component}.yaml").read_text()).group(1)
+ contains(rendered[component], f"storage: {size}")
+ contains(rendered["manager"], 'image: "210987654321.dkr.ecr.eu-west-1.amazonaws.com/openremote/manager:load2"')
+ contains(rendered["keycloak"], 'value: "https://legacy.example.com/auth"')
+ assert re.search(r'name: OR_HOSTNAME\s+value: "legacy.example.com"', rendered["manager"])
+ reset = "false" if setup else "true"
+ assert re.search(r'name: OR_SETUP_RUN_ON_RESTART\s+value: "' + reset + '"', rendered["manager"])
+ print(f"PASS: legacy {profile.name} / {script}", flush=True)
+
+ # A CLI stack must stop legacy cluster cleanup before AWS calls or changes.
+ call_log.write_text("")
+ env["CLI_STACK_PRESENT"] = "true"
+ result = subprocess.run(["bash", "./eks-cleanup-load.sh"], cwd=folder, env=env,
+ capture_output=True, text=True)
+ assert result.returncode == 1
+ contains(result.stderr, "Refusing legacy cleanup")
+ cleanup_calls = [json.loads(line) for line in call_log.read_text().splitlines()]
+ assert len(cleanup_calls) == 1 and cleanup_calls[0][0] == "kubectl"
+ print("PASS: legacy ACM/NLB deployment and shared-cluster cleanup guard", flush=True)
+
+
+if __name__ == "__main__":
+ assert HELM, "helm is required"
+ with tempfile.TemporaryDirectory(prefix="load-eks-test.") as directory:
+ for profile in PROFILES:
+ check_profile(profile, Path(directory))
+ check_legacy_scripts(Path(directory))
+ print("PASS: all CLI load profiles and the retained legacy ACM experiment")
diff --git a/kubernetes/test/load-helper-test b/kubernetes/test/load-helper-test
new file mode 100644
index 000000000..fdd14f36b
--- /dev/null
+++ b/kubernetes/test/load-helper-test
@@ -0,0 +1,288 @@
+#!/usr/bin/env python3
+"""Offline Bash helper orchestration tests; no real AWS or Kubernetes calls."""
+import json
+import os
+from pathlib import Path
+import shutil
+import subprocess
+import tempfile
+import unittest
+
+ROOT = Path(__file__).resolve().parents[2]
+REPOSITORY = '123456789012.dkr.ecr.eu-west-1.amazonaws.com/openremote/manager'
+SIZES = ('large', 'xlarge-minimal', 'xlarge', '2xlarge', '4xlarge', '8xlarge')
+STUB = r'''#!/usr/bin/env python3
+import json, os, sys
+from pathlib import Path
+name = Path(sys.argv[0]).name
+args = sys.argv[1:]
+def option(flag, default=None):
+ return args[args.index(flag)+1] if flag in args else default
+record = {'tool': name, 'args': args}
+assert not any(k.startswith(('OR_EKS_', 'OR_STACK_')) for k in os.environ)
+if '--values-dir' in args:
+ record['values'] = {p.name: p.read_text() for p in Path(option('--values-dir')).glob('*.yaml')}
+if name == 'aws':
+ record['access_key'] = os.getenv('AWS_ACCESS_KEY_ID')
+with open(os.environ['HELPER_TEST_LOG'], 'a') as f:
+ f.write(json.dumps(record)+'\n')
+account = '463235666115'
+role = 'arn:aws:iam::134517981306:role/test-dns-role'
+if name == 'aws':
+ if 'get-caller-identity' in args:
+ print('134517981306' if option('--profile') == 'dns-account' else account)
+ elif 'describe-cluster' in args:
+ if os.getenv('CLUSTER_ERROR'):
+ print('ExpiredTokenException', file=sys.stderr); sys.exit(1)
+ if os.getenv('HELPER_TEST_CLUSTER_EXISTS'):
+ print('ACTIVE')
+ else:
+ print('ResourceNotFoundException', file=sys.stderr); sys.exit(1)
+ elif 'list-hosted-zones' in args:
+ if os.getenv('DNS_READ_DENIED') and os.getenv('AWS_ACCESS_KEY_ID') == 'dns-key':
+ print('AccessDenied: Route53 read', file=sys.stderr); sys.exit(1)
+ zones = [{'Id': '/hostedzone/Z123', 'Name': 'example.com.', 'Config': {'PrivateZone': False}},
+ {'Id': '/hostedzone/ZPRIVATE', 'Name': 'example.com.', 'Config': {'PrivateZone': True}}]
+ if os.getenv('AMBIGUOUS_ZONE'):
+ zones.append({'Id': '/hostedzone/Z456', 'Name': 'example.com.', 'Config': {'PrivateZone': False}})
+ if os.getenv('SUBDOMAIN_ZONE'):
+ zones.append({'Id': '/hostedzone/ZSUB', 'Name': 'test.example.com.', 'Config': {'PrivateZone': False}})
+ print(json.dumps({'HostedZones': zones}))
+ elif 'describe-stacks' in args:
+ print(role if option('--profile') == 'dns-account' else role.replace('134517981306', account))
+ elif 'assume-role-with-web-identity' in args:
+ assert option('--web-identity-token').startswith('file://')
+ assert Path(option('--web-identity-token')[7:]).read_text().strip() == 'test-jwt'
+ assert os.getenv('AWS_ACCESS_KEY_ID') is None
+ print(json.dumps({'Credentials': {'AccessKeyId': 'irsa-key', 'SecretAccessKey': 'private-irsa-secret', 'SessionToken': 'private-irsa-session'}}))
+ elif 'assume-role' in args:
+ assert os.getenv('AWS_ACCESS_KEY_ID') == 'irsa-key'
+ if os.getenv('ASSUME_DENIED'):
+ print('AccessDenied: sts:AssumeRole', file=sys.stderr); sys.exit(1)
+ print(json.dumps({'Credentials': {'AccessKeyId': 'dns-key', 'SecretAccessKey': 'private-dns-secret', 'SessionToken': 'private-dns-session'}}))
+ elif 'list-resource-record-sets' in args:
+ assert os.getenv('AWS_ACCESS_KEY_ID') == 'dns-key'
+ print('{}')
+elif name == 'kubectl':
+ if 'serviceaccount' in args:
+ print('arn:aws:iam::463235666115:role/openremote-loadtest-eu-west-1-external-dns')
+ elif 'token' in args:
+ print('test-jwt')
+elif name == 'helm':
+ if args[0] == 'template':
+ hostname = option('--set-string').split('=',1)[1]
+ if os.getenv('BAD_RENDER'):
+ hostname = 'https://' + hostname + '/auth'
+ print('- name: KC_HOSTNAME\n value: "' + hostname + '"')
+ elif args[0] == 'list':
+ print('[{"name":"openremote-external-dns"}]' if os.getenv('HELPER_TEST_EXISTING_DNS') else '[]')
+ elif args[:2] == ['get','values']:
+ print(json.dumps({'txtOwnerId': 'preserve-me', 'domainFilters': ['example.com'], 'extraArgs': {
+ 'zone-id-filter': os.getenv('EXISTING_ZONE','Z123'), 'aws-assume-role': role,
+ 'aws-assume-role-external-id': 'existing-external-id', 'dry-run': bool(os.getenv('DNS_DRY_RUN'))}}))
+elif name == 'curl':
+ issuer = args[-1].removesuffix('/.well-known/openid-configuration')
+ if os.getenv('BAD_ISSUER'):
+ issuer = 'https://https://broken/auth/auth/realms/master'
+ print(json.dumps({'issuer': issuer, 'authorization_endpoint': issuer+'/protocol/openid-connect/auth'}))
+elif name == 'or-eks-stack' and args[0] == 'apply' and os.getenv('STACK_FAIL'):
+ sys.exit(37)
+'''
+
+class LoadHelperTest(unittest.TestCase):
+ def setUp(self):
+ self.temp = tempfile.TemporaryDirectory(prefix='load-helper-test-')
+ self.addCleanup(self.temp.cleanup)
+ self.root = Path(self.temp.name).resolve() / 'checkout with spaces'
+ (self.root / 'test').mkdir(parents=True)
+ (self.root / 'kubernetes').mkdir()
+ self.helper = self.root / 'test/or-eks-load'
+ shutil.copy2(ROOT / 'test/or-eks-load', self.helper)
+ for profile in [Path('test/load1-eks'), *(Path('test/load2-eks/profiles')/s for s in SIZES)]:
+ target = self.root/profile
+ target.mkdir(parents=True)
+ for component in ('manager', 'keycloak', 'postgresql', 'proxy', 'cluster'):
+ shutil.copy2(ROOT/profile/f'{component}.yaml', target)
+ binaries = self.root/'bin'
+ binaries.mkdir()
+ for name in ('aws', 'kubectl', 'helm', 'curl', 'sleep', 'eksctl', 'dig', 'or-eks-stack', 'or-eks-cluster'):
+ path = self.root/'kubernetes'/name if name.startswith('or-eks-') else binaries/name
+ path.write_text(STUB)
+ path.chmod(0o755)
+ self.config = self.root/'deployment.env'
+ self.settings = dict(LOAD_TEST='load2', LOAD_PROFILE='xlarge', CLUSTER_NAME='loadtest', STACK_NAME='load2',
+ AWS_REGION='eu-west-1', LOAD_HOSTNAME='load2.example.com',
+ LOAD_MANAGER_REPOSITORY=REPOSITORY, DNS_AWS_PROFILE='dns-account')
+ self.log = self.root/'calls.jsonl'
+ self.env = {k:v for k,v in os.environ.items() if not k.startswith(('LOAD_', 'OR_EKS_', 'OR_STACK_', 'AWS_', 'DNS_'))}
+ self.env.update(PATH=f'{binaries}:{os.environ["PATH"]}', HELPER_TEST_LOG=str(self.log))
+
+ def invoke(self, command, *options, expected=0, **env):
+ import shlex
+ self.config.write_text(''.join(f'{k}={shlex.quote(v)}\n' for k,v in self.settings.items()))
+ self.log.write_text('')
+ result = subprocess.run(['/bin/bash', str(self.helper), command, '--config', str(self.config), *options],
+ cwd=self.temp.name, env={**self.env, **env}, capture_output=True, text=True)
+ self.assertEqual(result.returncode, expected, result.stdout+result.stderr)
+ self.assertNotIn('private-irsa-secret', result.stdout+result.stderr)
+ self.assertNotIn('private-dns-secret', result.stdout+result.stderr)
+ self.calls = [json.loads(line) for line in self.log.read_text().splitlines()]
+ return result
+
+ def calls_to(self, tool, action):
+ return [c for c in self.calls if c['tool']==tool and action in c['args']]
+
+ def test_all_profiles_create_and_apply(self):
+ for load,size in [('load1',''), *(('load2',s) for s in SIZES)]:
+ with self.subTest(load=load,size=size):
+ self.settings.update(LOAD_TEST=load,LOAD_PROFILE=size)
+ self.invoke('up', AWS_ACCESS_KEY_ID='operator-key')
+ created, = self.calls_to('or-eks-cluster','create')
+ path = created['args'][created['args'].index('--config')+1]
+ expected_path = self.root/('test/load1-eks' if load=='load1' else f'test/load2-eks/profiles/{size}')
+ self.assertEqual(Path(path), expected_path/'cluster.yaml')
+ deployed, = self.calls_to('or-eks-stack','apply')
+ for component in ('manager','keycloak','postgresql','proxy'):
+ filename=f'{component}.yaml'
+ self.assertEqual(deployed['values'][filename], (expected_path/filename).read_text().replace('${LOAD_MANAGER_REPOSITORY}',REPOSITORY))
+ directory=Path(deployed['args'][deployed['args'].index('--values-dir')+1])
+ self.assertFalse(directory.exists())
+ token_call,=self.calls_to('aws','assume-role-with-web-identity')
+ token_path=Path(token_call['args'][token_call['args'].index('--web-identity-token')+1][7:])
+ self.assertFalse(token_path.exists())
+ read_call,=self.calls_to('aws','list-resource-record-sets')
+ self.assertLess(self.calls.index(read_call),self.calls.index(deployed))
+ self.assertIn('--mqtts',deployed['args'])
+ self.assertIn('90m',deployed['args'])
+
+ def test_resume_preserves_owner_and_stable_iam_configuration(self):
+ self.invoke('up',HELPER_TEST_CLUSTER_EXISTS='1',HELPER_TEST_EXISTING_DNS='1')
+ before,=self.calls_to('aws','deploy')
+ cluster,=self.calls_to('or-eks-cluster','apply')
+ self.assertIn('preserve-me',cluster['args'])
+ self.assertFalse(self.calls_to('or-eks-cluster','create'))
+ self.invoke('up',HELPER_TEST_CLUSTER_EXISTS='1',HELPER_TEST_EXISTING_DNS='1')
+ after,=self.calls_to('aws','deploy')
+ self.assertEqual(before['args'],after['args'])
+
+ def test_profile_selection_and_stale_environment(self):
+ self.settings['CLUSTER_AWS_PROFILE'] = 'cluster-account'
+ self.invoke('up', LOAD_PROFILE='invalid-old-export', OR_EKS_STACK_HOSTNAME='wrong.example.com',
+ OR_EKS_EXTERNAL_DNS_DRY_RUN='true')
+ identity = self.calls_to('aws','get-caller-identity')
+ self.assertIn('cluster-account', identity[0]['args'])
+ self.assertIn('dns-account', identity[1]['args'])
+ self.settings['DNS_AWS_PROFILE'] = ''
+ self.invoke('up')
+ for call in self.calls_to('aws','get-caller-identity'):
+ self.assertIn('cluster-account', call['args'])
+ role, = self.calls_to('aws','deploy')
+ self.assertIn('cluster-account', role['args'])
+
+ def test_invalid_hostnames_and_repositories_fail_before_aws(self):
+ for key, value in [('LOAD_HOSTNAME', 'https://load2.example.com/auth'),
+ ('STACK_NAME', 'default'),
+ ('LOAD_MANAGER_REPOSITORY', 'https://' + REPOSITORY),
+ ('LOAD_MANAGER_REPOSITORY', REPOSITORY + ':load2'),
+ ('LOAD_MANAGER_REPOSITORY', 'registry.example.com')]:
+ with self.subTest(key=key, value=value):
+ previous = self.settings[key]
+ self.settings[key] = value
+ self.invoke('up', expected=1)
+ self.assertFalse(self.calls)
+ self.settings[key] = previous
+
+ def test_zone_discovery_and_explicit_disambiguation(self):
+ result=self.invoke('up',expected=1,AMBIGUOUS_ZONE='1')
+ self.assertIn('Multiple matching zones',result.stderr)
+ self.assertFalse(self.calls_to('aws','deploy'))
+ self.settings['DNS_ZONE_ID']='Z456'
+ self.invoke('cluster-up',AMBIGUOUS_ZONE='1')
+ role,=self.calls_to('aws','deploy')
+ self.assertIn('HostedZoneId=Z456',role['args'])
+ self.assertFalse(self.calls_to('or-eks-stack','apply'))
+ del self.settings['DNS_ZONE_ID']
+ self.settings['LOAD_HOSTNAME']='load2.test.example.com'
+ self.invoke('cluster-up',SUBDOMAIN_ZONE='1')
+ role,=self.calls_to('aws','deploy')
+ self.assertIn('HostedZoneId=ZSUB',role['args'])
+
+ def test_iam_denials_stop_before_workloads(self):
+ for error in ('ASSUME_DENIED','DNS_READ_DENIED'):
+ with self.subTest(error=error):
+ result=self.invoke('up',expected=1,**{error:'1'})
+ self.assertIn('ExternalDNS cannot assume its roles or read Route 53',result.stderr)
+ self.assertFalse(self.calls_to('or-eks-stack','apply'))
+
+ def test_existing_cluster_zone_cannot_change(self):
+ result=self.invoke('up',expected=1,HELPER_TEST_CLUSTER_EXISTS='1',HELPER_TEST_EXISTING_DNS='1',EXISTING_ZONE='ZOTHER')
+ self.assertIn('different DNS zone',result.stderr)
+ self.assertFalse(self.calls_to('aws','deploy'))
+
+ def test_deploy_only_uses_existing_dns_configuration(self):
+ self.invoke('deploy',HELPER_TEST_CLUSTER_EXISTS='1',HELPER_TEST_EXISTING_DNS='1')
+ self.assertFalse(self.calls_to('aws','deploy'))
+ self.assertFalse(self.calls_to('or-eks-cluster','apply'))
+ assume,=self.calls_to('aws','assume-role')
+ self.assertIn('existing-external-id',assume['args'])
+ self.invoke('deploy',expected=1)
+ self.assertFalse(self.calls_to('or-eks-stack','apply'))
+ self.invoke('deploy',expected=1,HELPER_TEST_CLUSTER_EXISTS='1',HELPER_TEST_EXISTING_DNS='1',DNS_DRY_RUN='1')
+ self.assertFalse(self.calls_to('or-eks-stack','apply'))
+
+ def test_prepare_and_custom_values_never_overwritten(self):
+ output=self.root/'custom values'
+ self.invoke('prepare','--values-dir',str(output))
+ self.assertFalse(self.calls)
+ manager=output/'manager.yaml'
+ manager.write_text(manager.read_text().replace('tag: "load2"','tag: "custom"'))
+ before=manager.read_bytes()
+ self.invoke('prepare','--values-dir',str(output),expected=1)
+ self.assertEqual(before,manager.read_bytes())
+ self.settings['LOAD_VALUES_DIR']='custom values'
+ self.invoke('up')
+ deployed,=self.calls_to('or-eks-stack','apply')
+ self.assertEqual(deployed['values']['manager.yaml'],before.decode())
+ self.assertEqual(before,manager.read_bytes())
+
+ def test_lifecycle_commands_require_no_dns_credentials(self):
+ self.settings=dict(CLUSTER_NAME='loadtest',STACK_NAME='load2',CLUSTER_AWS_PROFILE='cluster-account')
+ for command in ('status','credentials','uninstall'):
+ self.invoke(command)
+ self.assertEqual(len(self.calls),1)
+ self.assertIn('cluster-account',self.calls[0]['args'])
+ for command,name,tool in (('destroy-stack','load2','or-eks-stack'),('destroy-cluster','loadtest','or-eks-cluster')):
+ self.invoke(command,expected=1)
+ self.assertFalse(self.calls)
+ self.invoke(command,'--confirm','wrong',expected=1)
+ self.assertFalse(self.calls)
+ self.invoke(command,'--confirm',name)
+ self.assertEqual(len(self.calls),1)
+ self.assertEqual(self.calls[0]['tool'],tool)
+
+ def test_bad_configuration_and_expired_credentials_fail_early(self):
+ result=self.invoke('up',expected=1,BAD_RENDER='1')
+ self.assertIn('invalid public URL',result.stderr)
+ self.assertFalse(self.calls_to('aws','deploy'))
+ self.invoke('up',expected=1,CLUSTER_ERROR='1')
+ self.assertFalse(self.calls_to('or-eks-cluster','create'))
+ self.settings['LOAD_PROFILE']='../invalid'
+ self.invoke('up',expected=1)
+ self.assertFalse(self.calls)
+
+ def test_stack_failure_and_bad_discovery(self):
+ self.invoke('up',expected=37,STACK_FAIL='1')
+ deployed,=self.calls_to('or-eks-stack','apply')
+ self.assertFalse(Path(deployed['args'][deployed['args'].index('--values-dir')+1]).exists())
+ result=self.invoke('up',expected=1,BAD_ISSUER='1')
+ self.assertIn('incorrect issuer',result.stderr)
+
+ def test_ingress_options(self):
+ self.settings.update(LOAD_EXPOSURE='ingress',LOAD_MQTTS_HOSTNAME='mqtt.example.com',LOAD_TIMEOUT='120m',CERTIFICATE_ARN='web-cert',MQTTS_CERTIFICATE_ARN='mqtt-cert')
+ self.invoke('up')
+ deployed,=self.calls_to('or-eks-stack','apply')
+ for value in ('ingress','mqtt.example.com','120m','web-cert','mqtt-cert', '--certificate-mode', '--mqtts-certificate-mode', 'existing'):
+ self.assertIn(value,deployed['args'])
+
+if __name__=='__main__':
+ unittest.main(verbosity=2)
diff --git a/kubernetes/test/mqtts-readiness-test b/kubernetes/test/mqtts-readiness-test
new file mode 100644
index 000000000..d95d43a3c
--- /dev/null
+++ b/kubernetes/test/mqtts-readiness-test
@@ -0,0 +1,157 @@
+#!/usr/bin/env python3
+"""Exercise the real EKS MQTTS probe against local TLS and non-TLS servers."""
+import contextlib
+import os
+from pathlib import Path
+import socket
+import ssl
+import subprocess
+import tempfile
+import threading
+import time
+import unittest
+
+KUBERNETES = Path(__file__).resolve().parent.parent
+
+
+class MqttsReadinessTest(unittest.TestCase):
+ @classmethod
+ def setUpClass(cls):
+ cls.directory = tempfile.TemporaryDirectory(prefix="or-mqtts-test-")
+ cls.addClassCleanup(cls.directory.cleanup)
+ cls.root = Path(cls.directory.name)
+ cls.ca = cls.root / "ca.pem"
+ cls.ca_key = cls.root / "ca.key"
+ config = cls.root / "ca.cnf"
+ config.write_text("""[req]
+prompt = no
+distinguished_name = dn
+x509_extensions = ca
+[dn]
+CN = OpenRemote local test CA
+[ca]
+basicConstraints = critical, CA:TRUE
+keyUsage = critical, keyCertSign, cRLSign
+subjectKeyIdentifier = hash
+""")
+ cls.openssl("req", "-x509", "-newkey", "rsa:2048", "-nodes", "-sha256",
+ "-days", "1", "-config", config, "-keyout", cls.ca_key, "-out", cls.ca)
+ cls.contexts = {}
+ for name, hostname in (("valid", "localhost"), ("wrong-host", "wrong.example.com")):
+ key, request, cert = (cls.root / (name + suffix) for suffix in (".key", ".csr", ".pem"))
+ extensions = cls.root / (name + ".cnf")
+ extensions.write_text(
+ "basicConstraints = critical, CA:FALSE\n"
+ "keyUsage = critical, digitalSignature, keyEncipherment\n"
+ "extendedKeyUsage = serverAuth\n"
+ "subjectKeyIdentifier = hash\n"
+ "authorityKeyIdentifier = keyid,issuer\n"
+ f"subjectAltName = DNS:{hostname}\n")
+ cls.openssl("req", "-new", "-newkey", "rsa:2048", "-nodes", "-sha256",
+ "-subj", f"/CN={hostname}", "-keyout", key, "-out", request)
+ cls.openssl("x509", "-req", "-in", request, "-CA", cls.ca, "-CAkey", cls.ca_key,
+ "-CAcreateserial", "-days", "1", "-sha256", "-extfile", extensions,
+ "-out", cert)
+ context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
+ context.load_cert_chain(cert, key)
+ cls.contexts[name] = context
+
+ @staticmethod
+ def openssl(*args):
+ subprocess.run(["openssl", *map(str, args)], check=True, capture_output=True, timeout=20)
+
+ @contextlib.contextmanager
+ def server(self, behavior):
+ listener = socket.socket()
+ try:
+ listener.bind(("127.0.0.1", 0))
+ listener.listen()
+ listener.settimeout(5)
+ except OSError:
+ listener.close()
+ raise
+ port = listener.getsockname()[1]
+ stop = threading.Event()
+
+ def serve():
+ try:
+ connection, _ = listener.accept()
+ with connection:
+ connection.settimeout(3)
+ if behavior in self.contexts:
+ with self.contexts[behavior].wrap_socket(connection, server_side=True) as tls:
+ tls.recv(1)
+ elif behavior == "plaintext":
+ connection.recv(2048)
+ connection.sendall(b"HTTP/1.1 400 Bad Request\r\n\r\n")
+ elif behavior == "stall":
+ stop.wait(5)
+ # Otherwise close the connection before any TLS handshake.
+ except OSError:
+ # Certificate rejection and early disconnects are expected.
+ pass
+
+ thread = threading.Thread(target=serve, daemon=True)
+ thread.start()
+ try:
+ yield port
+ finally:
+ stop.set()
+ listener.close()
+ thread.join(timeout=6)
+ self.assertFalse(thread.is_alive(), "Local test server did not stop")
+
+ def probe(self, behavior, *, exposure="haproxy", trusted=True, wait=False):
+ env = dict(os.environ)
+ env["SSL_CERT_FILE"] = str(self.ca if trusted else self.root / "missing-ca.pem")
+ env["SSL_CERT_DIR"] = str(self.root / "empty-ca-dir")
+ with self.server(behavior) as port:
+ command = """source "$1/or-eks-stack"
+EXPOSURE="$2"
+HOSTNAME_VALUE=localhost
+MQTTS_HOSTNAME_VALUE=localhost
+if [ "$EXPOSURE" = ingress ]; then HOSTNAME_VALUE=unused-web.example.com; fi
+MQTTS_PORT="$3"
+TIMEOUT=1s
+if [ "$4" = wait ]; then wait_for_mqtts; else mqtts_endpoint_is_ready 2; fi
+"""
+ started = time.monotonic()
+ result = subprocess.run(
+ ["bash", "-c", command, "bash", str(KUBERNETES), exposure, str(port),
+ "wait" if wait else "probe"],
+ env=env, text=True, capture_output=True, timeout=4,
+ )
+ return result, time.monotonic() - started
+
+ def test_accepts_verified_handshake_for_each_exposure(self):
+ for exposure in ("haproxy", "ingress"):
+ with self.subTest(exposure=exposure):
+ result, _ = self.probe("valid", exposure=exposure)
+ self.assertEqual(result.returncode, 0, result.stderr)
+
+ def test_rejects_trusted_certificate_for_wrong_hostname(self):
+ result, _ = self.probe("wrong-host")
+ self.assertEqual(result.returncode, 1, result.stderr)
+
+ def test_rejects_untrusted_certificate_for_correct_hostname(self):
+ result, _ = self.probe("valid", trusted=False)
+ self.assertEqual(result.returncode, 1, result.stderr)
+
+ def test_rejects_plaintext_server(self):
+ result, _ = self.probe("plaintext")
+ self.assertEqual(result.returncode, 1, result.stderr)
+
+ def test_rejects_connection_closed_before_handshake(self):
+ result, _ = self.probe("disconnect")
+ self.assertEqual(result.returncode, 1, result.stderr)
+
+ def test_stalled_handshake_respects_readiness_deadline(self):
+ result, elapsed = self.probe("stall", wait=True)
+ self.assertEqual(result.returncode, 1, result.stderr)
+ self.assertIn("Trusted MQTTS did not become ready", result.stderr)
+ self.assertNotIn("Trusted MQTTS is ready", result.stdout)
+ self.assertLess(elapsed, 3)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/kubernetes/test/or-eks-cluster-test b/kubernetes/test/or-eks-cluster-test
new file mode 100755
index 000000000..f0975d92b
--- /dev/null
+++ b/kubernetes/test/or-eks-cluster-test
@@ -0,0 +1,506 @@
+#!/usr/bin/env bash
+
+set -Eeuo pipefail
+
+TEST_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+KUBERNETES_DIR="$(cd "$TEST_DIR/.." && pwd)"
+TEST_TEMP_DIR="$(mktemp -d "${TMPDIR:-/tmp}/or-eks-cluster-test.XXXXXX")"
+CALL_LOG="$TEST_TEMP_DIR/calls.log"
+IAM_CONFIG_LOG="$TEST_TEMP_DIR/iam-config.log"
+TEST_CNI_ROLE_ARN="arn:aws:iam::123456789012:role/test-vpc-cni"
+TEST_SHARED_CERTIFICATE_ARN="arn:aws:acm:eu-west-1:123456789012:certificate/test-certificate"
+TEST_DEFAULT_VPC_CNI_CONFIGURATION='{}'
+TEST_EMPTY_RESOURCE_LIST='{"items":[]}'
+EXTERNAL_DNS_ROUTE53_ROLE_TEMPLATE="$KUBERNETES_DIR/cluster/eks/external-dns-route53-role.yaml"
+
+cleanup_test() {
+ cleanup
+ rm -rf "$TEST_TEMP_DIR"
+}
+
+# Source the command so its cluster operations can be replaced with deterministic
+# stubs. Its main function is guarded when sourced.
+source "$KUBERNETES_DIR/or-eks-cluster"
+trap cleanup_test EXIT
+
+fail() {
+ printf 'FAIL: %s\n' "$*" >&2
+ exit 1
+}
+
+log_call() {
+ local line="$1"
+ shift
+ for argument in "$@"; do
+ line+=$'\t'"$argument"
+ done
+ printf '%s\n' "$line" >> "$CALL_LOG"
+}
+
+require_command() {
+ :
+}
+
+require_kube_context() {
+ :
+}
+
+aws_eks() {
+ log_call aws-eks "$@"
+ case " $* " in
+ *" describe-addon "*)
+ jq -cn \
+ --arg configuration "${OR_EKS_TEST_VPC_CNI_CONFIGURATION-$TEST_DEFAULT_VPC_CNI_CONFIGURATION}" \
+ --arg role "${OR_EKS_TEST_ADDON_ROLE_ARN-$TEST_CNI_ROLE_ARN}" \
+ '{configurationValues: $configuration} +
+ (if $role == "" then {} else {serviceAccountRoleArn: $role} end)'
+ ;;
+ *" update-addon "*)
+ printf '%s\n' "${OR_EKS_TEST_UPDATE_ID:-test-update}"
+ ;;
+ *" wait addon-active "*)
+ [ "${OR_EKS_TEST_ADDON_WAIT_SUCCESS:-true}" = "true" ]
+ ;;
+ *" describe-update "*)
+ printf '%s\n' '{"Id":"test-update","Status":"Failed","Errors":[{"errorCode":"test-error"}]}'
+ ;;
+ esac
+}
+
+eksctl_with_profile() {
+ local argument
+ local config_file=""
+ local previous_argument=""
+
+ log_call eksctl "$@"
+ case " $* " in
+ *" get iamserviceaccount "*)
+ printf '%s\n' "${OR_EKS_TEST_IAM_SERVICE_ACCOUNTS:-[]}"
+ ;;
+ *" create iamserviceaccount "*|*" update iamserviceaccount "*)
+ for argument in "$@"; do
+ if [ "$previous_argument" = "--config-file" ]; then
+ config_file="$argument"
+ break
+ fi
+ previous_argument="$argument"
+ done
+ [ -n "$config_file" ] || fail "IAM service account command has no config file"
+ cp "$config_file" "$IAM_CONFIG_LOG"
+ ;;
+ esac
+}
+
+helm() {
+ log_call helm "$@"
+ case " $* " in
+ *" show crds "*)
+ printf '%s\n' 'apiVersion: apiextensions.k8s.io/v1' 'kind: CustomResourceDefinition'
+ ;;
+ esac
+}
+
+kubectl() {
+ log_call kubectl "$@"
+ if [ -n "${OR_EKS_TEST_FAILED_RESOURCE:-}" ] && \
+ [[ " $* " == *" get $OR_EKS_TEST_FAILED_RESOURCE "* ]]; then
+ return 1
+ fi
+ case " $* " in
+ *" get configmap openremote-hostnames "*)
+ printf '%s' "${OR_EKS_TEST_REGISTRY_JSON:-}"
+ ;;
+ *" apply -f - "*)
+ while IFS= read -r _; do
+ :
+ done
+ ;;
+ *" get ingressclass openremote-alb "*"go-template"*)
+ printf '%s' "${OR_EKS_TEST_SHARED_CERTIFICATE_ARN-$TEST_SHARED_CERTIFICATE_ARN}"
+ ;;
+ *" get serviceaccount external-dns "*)
+ printf '%s' "${OR_EKS_TEST_EXTERNAL_DNS_IRSA_ROLE_ARN:-arn:aws:iam::123456789012:role/openremote-test-cluster-eu-west-1-external-dns}"
+ ;;
+ *" get deployment openremote-external-dns "*)
+ printf '%s\n' "${OR_EKS_TEST_EXTERNAL_DNS_ARGS:---source=service
+--source=ingress
+--source=crd
+--txt-owner-id=test-cluster-eu-west-1
+--dry-run}"
+ ;;
+ *" get customresourcedefinition/dnsendpoints.externaldns.k8s.io "*)
+ if [ "${OR_EKS_TEST_DNS_ENDPOINT_CRD:-true}" = "true" ]; then
+ printf '%s\n' 'customresourcedefinition.apiextensions.k8s.io/dnsendpoints.externaldns.k8s.io'
+ elif [[ " $* " != *" --ignore-not-found "* ]]; then
+ return 1
+ fi
+ ;;
+ *" get ingress --all-namespaces "*)
+ printf '%s\n' "${OR_EKS_TEST_INGRESSES-$TEST_EMPTY_RESOURCE_LIST}"
+ ;;
+ *" get services --all-namespaces "*)
+ printf '%s\n' "${OR_EKS_TEST_SERVICES-$TEST_EMPTY_RESOURCE_LIST}"
+ ;;
+ *" get persistentvolumeclaims --all-namespaces "*)
+ printf '%s\n' "${OR_EKS_TEST_PVCS-$TEST_EMPTY_RESOURCE_LIST}"
+ ;;
+ *" get namespaces -o json "*)
+ printf '%s\n' "${OR_EKS_TEST_NAMESPACES-$TEST_EMPTY_RESOURCE_LIST}"
+ ;;
+ *" get deployments,statefulsets,daemonsets,replicasets,replicationcontrollers,jobs,cronjobs --all-namespaces "*)
+ printf '%s\n' "${OR_EKS_TEST_WORKLOAD_CONTROLLERS-$TEST_EMPTY_RESOURCE_LIST}"
+ ;;
+ *" get persistentvolumes -o json "*)
+ printf '%s\n' "${OR_EKS_TEST_PVS-$TEST_EMPTY_RESOURCE_LIST}"
+ ;;
+ *" get pods --all-namespaces "*)
+ printf '%s\n' "${OR_EKS_TEST_PODS-$TEST_EMPTY_RESOURCE_LIST}"
+ ;;
+ *" get dnsendpoints --all-namespaces "*)
+ printf '%s\n' "${OR_EKS_TEST_DNS_ENDPOINTS-$TEST_EMPTY_RESOURCE_LIST}"
+ ;;
+ *" get serviceaccount aws-node "*)
+ printf '%s' "${OR_EKS_TEST_SERVICE_ACCOUNT_ROLE_ARN-$TEST_CNI_ROLE_ARN}"
+ ;;
+ *" get daemonset aws-node "*)
+ printf '%s' "${OR_EKS_TEST_CNI_CONTAINERS:-aws-node aws-eks-nodeagent}"
+ ;;
+ *" get customresourcedefinition/policyendpoints.networking.k8s.aws "*)
+ [ "${OR_EKS_TEST_POLICY_CRD:-true}" = "true" ]
+ ;;
+ esac
+}
+
+assert_contains() {
+ local expected="$1"
+ grep -F -- "$expected" "$CALL_LOG" >/dev/null || fail \
+ "Expected command log to contain: $expected"
+}
+
+assert_not_contains() {
+ local unexpected="$1"
+ if grep -F -- "$unexpected" "$CALL_LOG" >/dev/null; then
+ fail "Expected command log not to contain: $unexpected"
+ fi
+}
+
+set_external_dns_test_configuration() {
+ MANAGE_EXTERNAL_DNS=true
+ EXTERNAL_DNS_HOSTED_ZONE_ID=Z0123456789
+ EXTERNAL_DNS_DOMAIN=example.com
+ EXTERNAL_DNS_ASSUME_ROLE_ARN=arn:aws:iam::210987654321:role/openremote-route53
+ EXTERNAL_DNS_ASSUME_ROLE_EXTERNAL_ID=test-external-id
+ EXTERNAL_DNS_OWNER_ID=test-cluster-eu-west-1
+ EXTERNAL_DNS_IRSA_ROLE_NAME=openremote-test-cluster-eu-west-1-external-dns
+ EXTERNAL_DNS_DRY_RUN=true
+}
+
+CLUSTER_NAME=test-cluster
+AWS_REGION_NAME=eu-west-1
+AWS_PROFILE_NAME=
+KUBE_CONTEXT=test-cluster@eu-west-1
+
+set_external_dns_test_configuration
+validate_external_dns_configuration
+
+SHARED_CERTIFICATE_ARN="$TEST_SHARED_CERTIFICATE_ARN"
+CLEAR_SHARED_CERTIFICATE=false
+MANAGE_LOAD_BALANCER_CONTROLLER=true
+validate_shared_certificate_configuration
+
+if (SHARED_CERTIFICATE_ARN="arn:aws:acm:us-east-1:123456789012:certificate/test" \
+ validate_shared_certificate_configuration) >/dev/null 2>&1; then
+ fail "A shared certificate from another region was accepted"
+fi
+if (CLEAR_SHARED_CERTIFICATE=true \
+ validate_shared_certificate_configuration) >/dev/null 2>&1; then
+ fail "Setting and clearing the shared certificate together was accepted"
+fi
+if (MANAGE_LOAD_BALANCER_CONTROLLER=false \
+ validate_shared_certificate_configuration) >/dev/null 2>&1; then
+ fail "Shared certificate configuration was accepted while skipping its IngressClass"
+fi
+
+: > "$CALL_LOG"
+: > "$IAM_CONFIG_LOG"
+reconcile_external_dns
+assert_contains $'eksctl\tget\tiamserviceaccount'
+assert_contains $'eksctl\tcreate\tiamserviceaccount'
+assert_contains $'--include\tkube-system/external-dns'
+assert_contains $'helm\tupgrade\t--install\topenremote-external-dns\texternal-dns/external-dns'
+assert_contains $'helm\tshow\tcrds\texternal-dns/external-dns\t--version\t1.21.1'
+assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\twait\t--for=condition=Established\tcustomresourcedefinition/dnsendpoints.externaldns.k8s.io\t--timeout\t2m'
+assert_contains $'--version\t1.21.1'
+assert_contains $'image.tag=v0.22.0'
+assert_contains $'sources[2]=crd'
+assert_contains $'policy=sync'
+assert_contains $'registry=txt'
+assert_contains $'txtOwnerId=test-cluster-eu-west-1'
+assert_contains $'labelFilter=openremote.io/managed-dns=true'
+assert_contains $'extraArgs.aws-zone-type=public'
+assert_contains $'extraArgs.zone-id-filter=Z0123456789'
+assert_contains $'domainFilters[0]=example.com'
+assert_contains $'extraArgs.aws-assume-role=arn:aws:iam::210987654321:role/openremote-route53'
+assert_contains $'extraArgs.aws-assume-role-external-id=test-external-id'
+assert_contains $'extraArgs.dry-run=true'
+grep -F -- 'roleName: "openremote-test-cluster-eu-west-1-external-dns"' \
+ "$IAM_CONFIG_LOG" >/dev/null || fail "IRSA config does not use the deterministic role name"
+grep -F -- 'Action:' "$IAM_CONFIG_LOG" >/dev/null || fail \
+ "IRSA config has no inline policy"
+grep -F -- '- sts:AssumeRole' "$IAM_CONFIG_LOG" >/dev/null || fail \
+ "IRSA config does not permit assuming the DNS-account role"
+grep -F -- 'Resource: "arn:aws:iam::210987654321:role/openremote-route53"' \
+ "$IAM_CONFIG_LOG" >/dev/null || fail "IRSA policy is not scoped to the DNS-account role"
+
+: > "$CALL_LOG"
+OR_EKS_TEST_IAM_SERVICE_ACCOUNTS='[{"metadata":{"name":"external-dns"}}]' \
+ reconcile_external_dns
+assert_contains $'eksctl\tupdate\tiamserviceaccount'
+assert_not_contains $'eksctl\tcreate\tiamserviceaccount'
+
+: > "$CALL_LOG"
+EXTERNAL_DNS_DRY_RUN=false \
+EXTERNAL_DNS_ASSUME_ROLE_EXTERNAL_ID= \
+OR_EKS_TEST_IAM_SERVICE_ACCOUNTS='[{"metadata":{"name":"external-dns"}}]' \
+ reconcile_external_dns
+assert_not_contains 'extraArgs.aws-assume-role-external-id='
+assert_not_contains 'extraArgs.dry-run=true'
+
+: > "$CALL_LOG"
+MANAGE_EXTERNAL_DNS=false reconcile_external_dns
+assert_not_contains $'eksctl\tget\tiamserviceaccount'
+assert_not_contains $'helm\tupgrade'
+
+if (set_external_dns_test_configuration; EXTERNAL_DNS_DOMAIN=; \
+ validate_external_dns_configuration) >/dev/null 2>&1; then
+ fail "Incomplete ExternalDNS configuration was accepted"
+fi
+
+grep -F -- 'route53:ChangeResourceRecordSetsNormalizedRecordNames' \
+ "$EXTERNAL_DNS_ROUTE53_ROLE_TEMPLATE" >/dev/null || fail \
+ "DNS-account role does not restrict writable record names"
+grep -F -- 'route53:ChangeResourceRecordSetsRecordTypes' \
+ "$EXTERNAL_DNS_ROUTE53_ROLE_TEMPLATE" >/dev/null || fail \
+ "DNS-account role does not restrict writable record types"
+grep -F -- 'aws:PrincipalArn:' "$EXTERNAL_DNS_ROUTE53_ROLE_TEMPLATE" >/dev/null || fail \
+ "DNS-account role does not restrict its cluster-account principal"
+grep -F -- 'arn:${AWS::Partition}:route53:::hostedzone/${HostedZoneId}' \
+ "$EXTERNAL_DNS_ROUTE53_ROLE_TEMPLATE" >/dev/null || fail \
+ "DNS-account role does not restrict writes to the configured hosted zone"
+
+: > "$CALL_LOG"
+external_dns_status_output="$(show_external_dns_status)"
+assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\t--namespace\tkube-system\tget\tdeployments,pods'
+grep -Fx -- 'ExternalDNS TXT owner ID: test-cluster-eu-west-1' \
+ <<< "$external_dns_status_output" >/dev/null || fail \
+ "ExternalDNS status did not report the deployed TXT owner ID"
+grep -Fx -- 'ExternalDNS dry run: true' <<< "$external_dns_status_output" >/dev/null || fail \
+ "ExternalDNS status did not report dry-run mode"
+grep -Fx -- 'ExternalDNS DNSEndpoint CRD installed: true' \
+ <<< "$external_dns_status_output" >/dev/null || fail \
+ "ExternalDNS status did not report the DNSEndpoint CRD"
+grep -Fx -- 'ExternalDNS DNSEndpoint source enabled: true' \
+ <<< "$external_dns_status_output" >/dev/null || fail \
+ "ExternalDNS status did not report the CRD source"
+
+external_dns_status_output="$(OR_EKS_TEST_DNS_ENDPOINT_CRD=false \
+ OR_EKS_TEST_EXTERNAL_DNS_ARGS=$'--source=service\n--source=ingress' \
+ show_external_dns_status)"
+grep -Fx -- 'ExternalDNS DNSEndpoint CRD installed: false' \
+ <<< "$external_dns_status_output" >/dev/null || fail \
+ "ExternalDNS status did not report a missing DNSEndpoint CRD"
+grep -Fx -- 'ExternalDNS DNSEndpoint source enabled: false' \
+ <<< "$external_dns_status_output" >/dev/null || fail \
+ "ExternalDNS status did not report a disabled CRD source"
+
+: > "$CALL_LOG"
+destroy_preflight
+assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\tget\tdnsendpoints\t--all-namespaces\t-o\tjson'
+
+: > "$CALL_LOG"
+set +e
+destroy_failure="$(OR_EKS_TEST_DNS_ENDPOINTS='{"items":[{"metadata":{"namespace":"stack-a","name":"acm-validation"}}]}' \
+ destroy_preflight 2>&1)"
+destroy_status=$?
+set -e
+[ "$destroy_status" -ne 0 ] || fail \
+ "Cluster destruction preflight accepted a retained DNSEndpoint"
+grep -Fx -- ' DNSEndpoints: 1' <<< "$destroy_failure" >/dev/null || fail \
+ "Cluster destruction preflight did not report the retained DNSEndpoint"
+
+assert_destroy_blocked() {
+ local fixture_variable="$1"
+ local fixture="$2"
+ local expected="$3"
+ local output
+ : > "$CALL_LOG"
+ if output="$(
+ export "$fixture_variable=$fixture"
+ require_existing_cluster() { :; }
+ CONFIRM_CLUSTER_NAME="$CLUSTER_NAME"
+ destroy_cluster 2>&1
+ )"; then
+ fail "Cluster destruction accepted $fixture_variable=$fixture"
+ fi
+ grep -F -- "$expected" <<< "$output" >/dev/null || fail \
+ "Cluster destruction did not report '$expected': $output"
+ assert_not_contains $'helm\tuninstall'
+ assert_not_contains $'eksctl\tdelete\tcluster'
+}
+
+# These fixtures have no reservations, Pods, PVCs, or public endpoints, so each
+# exercises state that the earlier resource counts alone could not detect.
+for stack_label in openremote.io/stack app.kubernetes.io/part-of; do
+ namespace_fixture="$(jq -cn --arg label "$stack_label" '
+ {items:[{metadata:{name:"retained-stack",labels:{($label):"openremote"}}}]}')"
+ assert_destroy_blocked OR_EKS_TEST_NAMESPACES "$namespace_fixture" 'Namespace/retained-stack'
+done
+
+for controller_kind in Deployment StatefulSet DaemonSet ReplicaSet ReplicationController Job CronJob; do
+ controller_fixture="$(jq -cn --arg kind "$controller_kind" '
+ {items:[{kind:$kind,metadata:{namespace:"user-workload",name:"idle"},
+ spec:{replicas:0,suspend:true}}]}')"
+ assert_destroy_blocked OR_EKS_TEST_WORKLOAD_CONTROLLERS "$controller_fixture" \
+ "$controller_kind/user-workload/idle"
+done
+
+assert_destroy_blocked OR_EKS_TEST_PVS \
+ '{"items":[{"metadata":{"name":"retained-data"},"spec":{"persistentVolumeReclaimPolicy":"Retain"},"status":{"phase":"Released"}}]}' \
+ 'PersistentVolume/retained-data'
+
+for resource in namespaces deployments,statefulsets,daemonsets,replicasets,replicationcontrollers,jobs,cronjobs persistentvolumes; do
+ assert_destroy_blocked OR_EKS_TEST_FAILED_RESOURCE "$resource" 'Unable to inspect'
+done
+assert_destroy_blocked OR_EKS_TEST_PVS '{invalid-json' 'Unable to inspect PersistentVolumes'
+
+# Shared system controllers and empty unrelated namespaces must still permit
+# deletion once all stack state has been removed.
+: > "$CALL_LOG"
+(
+ require_existing_cluster() { :; }
+ CONFIRM_CLUSTER_NAME="$CLUSTER_NAME"
+ OR_EKS_TEST_NAMESPACES='{"items":[{"metadata":{"name":"default"}},{"metadata":{"name":"kube-system"}}]}'
+ OR_EKS_TEST_WORKLOAD_CONTROLLERS="$(jq -cn '{items:[
+ ["kube-system","kube-public","kube-node-lease"][] |
+ {kind:"Deployment",metadata:{namespace:.,name:"system-controller"}}]}')"
+ destroy_cluster
+)
+assert_contains $'helm\tuninstall\topenremote-external-dns'
+assert_contains $'eksctl\tdelete\tcluster'
+
+: > "$CALL_LOG"
+MANAGE_EXTERNAL_DNS=true uninstall_external_dns
+assert_contains $'helm\tuninstall\topenremote-external-dns'
+
+: > "$CALL_LOG"
+MANAGE_EXTERNAL_DNS=false uninstall_external_dns
+assert_not_contains $'helm\tuninstall\topenremote-external-dns'
+
+: > "$CALL_LOG"
+reconcile_load_balancer_controller
+assert_contains $'helm\tupgrade\t--install\taws-load-balancer-controller'
+assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\twait\t--for=condition=Established\tcustomresourcedefinition/ingressclassparams.elbv2.k8s.aws'
+assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\tapply\t-f\t'"$DEFAULT_INGRESS_CLASS_FILE"
+assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\tannotate\tingressclass\topenremote-alb\topenremote.io/shared-certificate-arn='"$TEST_SHARED_CERTIFICATE_ARN"$'\t--overwrite'
+
+grep -F -- 'name: openremote-stacks' "$DEFAULT_INGRESS_CLASS_FILE" >/dev/null || fail \
+ "Shared ALB group is missing from the IngressClass manifest"
+grep -F -- 'app.kubernetes.io/part-of: openremote' "$DEFAULT_INGRESS_CLASS_FILE" >/dev/null || fail \
+ "Shared ALB IngressClass does not restrict namespace membership"
+grep -F -- 'targetType: ip' "$DEFAULT_INGRESS_CLASS_FILE" >/dev/null || fail \
+ "Shared ALB IngressClass does not use VPC CNI Pod IP targets"
+grep -F -- 'sslRedirectPort: "443"' "$DEFAULT_INGRESS_CLASS_FILE" >/dev/null || fail \
+ "Shared ALB IngressClass does not enforce HTTPS redirection"
+
+: > "$CALL_LOG"
+SHARED_CERTIFICATE_ARN= CLEAR_SHARED_CERTIFICATE=true reconcile_load_balancer_controller
+assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\tannotate\tingressclass\topenremote-alb\topenremote.io/shared-certificate-arn-'
+
+: > "$CALL_LOG"
+OR_EKS_TEST_SHARED_CERTIFICATE_ARN= SHARED_CERTIFICATE_ARN= \
+ CLEAR_SHARED_CERTIFICATE=true reconcile_load_balancer_controller
+assert_not_contains $'kubectl\t--context\ttest-cluster@eu-west-1\tannotate\tingressclass\topenremote-alb\topenremote.io/shared-certificate-arn-'
+
+: > "$CALL_LOG"
+SHARED_CERTIFICATE_ARN= CLEAR_SHARED_CERTIFICATE=false reconcile_load_balancer_controller
+assert_not_contains $'kubectl\t--context\ttest-cluster@eu-west-1\tannotate\tingressclass'
+
+: > "$CALL_LOG"
+SHARED_CERTIFICATE_ARN= CLEAR_SHARED_CERTIFICATE=false \
+ MANAGE_LOAD_BALANCER_CONTROLLER=false reconcile_load_balancer_controller
+assert_not_contains $'helm\tupgrade'
+
+shared_certificate_status_output="$(show_shared_certificate_status)"
+grep -Fx -- "Shared ACM certificate ARN: $TEST_SHARED_CERTIFICATE_ARN" \
+ <<< "$shared_certificate_status_output" >/dev/null || fail \
+ "Cluster status did not report its shared certificate ARN"
+shared_certificate_status_output="$(OR_EKS_TEST_SHARED_CERTIFICATE_ARN= \
+ show_shared_certificate_status)"
+grep -Fx -- 'Shared ACM certificate ARN: not configured' \
+ <<< "$shared_certificate_status_output" >/dev/null || fail \
+ "Cluster status did not report missing shared certificate configuration"
+
+: > "$CALL_LOG"
+OR_EKS_TEST_VPC_CNI_CONFIGURATION='{"env":{"WARM_IP_TARGET":"2"}}' \
+ reconcile_network_policy_support
+assert_contains $'aws-eks\tupdate-addon\t--cluster-name\ttest-cluster\t--addon-name\tvpc-cni'
+assert_contains $'--configuration-values\t{"env":{"WARM_IP_TARGET":"2"},"enableNetworkPolicy":"true"}'
+assert_contains $'--service-account-role-arn\tarn:aws:iam::123456789012:role/test-vpc-cni'
+assert_contains $'aws-eks\twait\taddon-active'
+assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\t--namespace\tkube-system\trollout\tstatus\tdaemonset/aws-node'
+
+: > "$CALL_LOG"
+OR_EKS_TEST_VPC_CNI_CONFIGURATION='{"enableNetworkPolicy":true}' \
+ reconcile_network_policy_support
+assert_not_contains $'aws-eks\tupdate-addon'
+
+: > "$CALL_LOG"
+OR_EKS_TEST_VPC_CNI_CONFIGURATION='{"enableNetworkPolicy":true}' \
+OR_EKS_TEST_SERVICE_ACCOUNT_ROLE_ARN='' \
+ reconcile_network_policy_support
+assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\t--namespace\tkube-system\tannotate\tserviceaccount\taws-node'
+assert_contains $'aws-eks\tupdate-addon'
+
+: > "$CALL_LOG"
+if (OR_EKS_TEST_VPC_CNI_CONFIGURATION='{"enableNetworkPolicy":true}' \
+ OR_EKS_TEST_CNI_CONTAINERS='aws-node aws-network-policy-agent' \
+ reconcile_network_policy_support) >/dev/null 2>&1; then
+ fail "NetworkPolicy reconciliation accepted the image name instead of the aws-eks-nodeagent container"
+fi
+
+: > "$CALL_LOG"
+if (OR_EKS_TEST_POLICY_CRD=false reconcile_network_policy_support) >/dev/null 2>&1; then
+ fail "NetworkPolicy reconciliation succeeded without the PolicyEndpoint CRD"
+fi
+
+: > "$CALL_LOG"
+if (OR_EKS_TEST_CNI_CONTAINERS=aws-node reconcile_network_policy_support) >/dev/null 2>&1; then
+ fail "NetworkPolicy reconciliation succeeded without the policy agent"
+fi
+
+: > "$CALL_LOG"
+show_vpc_cni_diagnostics >/dev/null 2>&1
+assert_contains $'aws-eks\tdescribe-addon'
+assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\t--namespace\tkube-system\tget\tdaemonset\taws-node'
+assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\t--namespace\tkube-system\tget\tpods\t--selector\tk8s-app=aws-node'
+
+: > "$CALL_LOG"
+set +e
+failure_output="$(OR_EKS_TEST_ADDON_WAIT_SUCCESS=false \
+ reconcile_network_policy_support 2>&1)"
+failure_status=$?
+set -e
+if [ "$failure_status" -eq 0 ]; then
+ fail "NetworkPolicy reconciliation succeeded while the managed add-on was unhealthy"
+fi
+grep -F -- 'Amazon VPC CNI add-on diagnostics:' <<< "$failure_output" >/dev/null || fail \
+ "Unhealthy add-on failure did not include diagnostics"
+
+registry="$(jq -cn '{metadata:{labels:{"app.kubernetes.io/managed-by":"or-stack"}},data:{"stack-a.example.com":({namespace:"stack-a",namespaceUID:"uid-stack-a",target:"eks",dns:"external-dns",exposure:"haproxy"}|tojson)}}')"
+: > "$CALL_LOG"
+if (OR_EKS_TEST_REGISTRY_JSON="$registry" destroy_preflight) >/dev/null 2>&1; then
+ fail "Cluster destruction accepted an outstanding hostname reservation"
+fi
+assert_not_contains $'helm\tuninstall'
+
+printf 'or-eks-cluster tests passed\n'
diff --git a/kubernetes/test/or-eks-stack-test b/kubernetes/test/or-eks-stack-test
new file mode 100755
index 000000000..029ad4875
--- /dev/null
+++ b/kubernetes/test/or-eks-stack-test
@@ -0,0 +1,2396 @@
+#!/usr/bin/env bash
+
+set -Eeuo pipefail
+
+TEST_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+KUBERNETES_DIR="$(cd "$TEST_DIR/.." && pwd)"
+TEST_TEMP_DIR="$(mktemp -d "${TMPDIR:-/tmp}/or-eks-stack-test.XXXXXX")"
+CALL_LOG="$TEST_TEMP_DIR/calls.log"
+CURL_COUNT_FILE="$TEST_TEMP_DIR/curl-count"
+PROXY_LE_DIR="$TEST_TEMP_DIR/letsencrypt"
+DNS_ENDPOINT_LOG="$TEST_TEMP_DIR/dns-endpoint.json"
+CERTIFICATE_DELETED_FILE="$TEST_TEMP_DIR/certificate-deleted"
+MQTTS_CERTIFICATE_DELETED_FILE="$TEST_TEMP_DIR/mqtts-certificate-deleted"
+INGRESS_DELETED_FILE="$TEST_TEMP_DIR/ingress-deleted"
+NAMESPACE_DELETED_FILE="$TEST_TEMP_DIR/namespace-deleted"
+NAMESPACE_PREPARED_FILE="$TEST_TEMP_DIR/namespace-prepared"
+CERTIFICATE_NAMESPACE_FILE="$TEST_TEMP_DIR/certificate-namespace.json"
+TEST_CLUSTER_ENDPOINT="https://test-cluster.eks.example.com"
+TEST_CERTIFICATE_ARN="arn:aws:acm:eu-west-1:123456789012:certificate/test-certificate"
+TEST_SECOND_CERTIFICATE_ARN="arn:aws:acm:eu-west-1:123456789012:certificate/second-certificate"
+TEST_MQTTS_CERTIFICATE_ARN="arn:aws:acm:eu-west-1:123456789012:certificate/mqtts-certificate"
+TEST_MQTTS_NLB_ARN="arn:aws:elasticloadbalancing:eu-west-1:123456789012:loadbalancer/net/test-mqtt-nlb/1234567890abcdef"
+DEFAULT_LOAD_BALANCERS_JSON="$(jq -cn --arg arn "$TEST_MQTTS_NLB_ARN" \
+ '{LoadBalancers:[{Type:"network",LoadBalancerArn:$arn,DNSName:"test-mqtt-nlb.elb.eu-west-1.amazonaws.com"}]}')"
+DEFAULT_CERTIFICATE_JSON='{"Certificate":{"Status":"ISSUED","DomainName":"example.com","SubjectAlternativeNames":["example.com","*.example.com"]}}'
+DEFAULT_CERTIFICATE_TAGS='{"Tags":[]}'
+DEFAULT_CERTIFICATE_LIST='{"CertificateSummaryList":[]}'
+MANAGED_CERTIFICATE_JSON='{"Certificate":{"Status":"PENDING_VALIDATION","DomainName":"stack-a.example.com","SubjectAlternativeNames":["stack-a.example.com"],"DomainValidationOptions":[{"DomainName":"stack-a.example.com","ValidationMethod":"DNS","ResourceRecord":{"Name":"_validation.stack-a.example.com.","Type":"CNAME","Value":"_target.acm-validations.aws."}}]}}'
+MANAGED_CERTIFICATE_TAGS='{"Tags":[{"Key":"openremote.io/managed-by","Value":"or-eks-stack"},{"Key":"openremote.io/cluster","Value":"test-cluster"},{"Key":"openremote.io/stack","Value":"stack-a"},{"Key":"openremote.io/hostname","Value":"stack-a.example.com"}]}'
+MANAGED_MQTTS_CERTIFICATE_JSON='{"Certificate":{"Status":"PENDING_VALIDATION","DomainName":"mqtt-stack-a.example.com","SubjectAlternativeNames":["mqtt-stack-a.example.com"],"InUseBy":[],"DomainValidationOptions":[{"DomainName":"mqtt-stack-a.example.com","ValidationMethod":"DNS","ResourceRecord":{"Name":"_validation.mqtt-stack-a.example.com.","Type":"CNAME","Value":"_mqtt-target.acm-validations.aws."}}]}}'
+MANAGED_MQTTS_CERTIFICATE_TAGS='{"Tags":[{"Key":"openremote.io/managed-by","Value":"or-eks-stack"},{"Key":"openremote.io/cluster","Value":"test-cluster"},{"Key":"openremote.io/stack","Value":"stack-a"},{"Key":"openremote.io/hostname","Value":"mqtt-stack-a.example.com"},{"Key":"openremote.io/endpoint","Value":"mqtts"}]}'
+MANAGED_MQTTS_CERTIFICATE_TAGS_WITHOUT_ENDPOINT='{"Tags":[{"Key":"openremote.io/managed-by","Value":"or-eks-stack"},{"Key":"openremote.io/cluster","Value":"test-cluster"},{"Key":"openremote.io/stack","Value":"stack-a"},{"Key":"openremote.io/hostname","Value":"mqtt-stack-a.example.com"}]}'
+DEFAULT_DNS_ENDPOINT_JSON='{"metadata":{"labels":{"app.kubernetes.io/managed-by":"or-eks-stack","openremote.io/managed-dns":"true"},"annotations":{"openremote.io/certificate-arn":"arn:aws:acm:eu-west-1:123456789012:certificate/test-certificate"}},"spec":{"endpoints":[{"dnsName":"_validation.stack-a.example.com.","recordType":"CNAME","recordTTL":300,"targets":["_target.acm-validations.aws."]}]}}'
+DEFAULT_MQTTS_DNS_ENDPOINT_JSON='{"metadata":{"labels":{"app.kubernetes.io/managed-by":"or-eks-stack","openremote.io/managed-dns":"true"},"annotations":{"openremote.io/certificate-arn":"arn:aws:acm:eu-west-1:123456789012:certificate/mqtts-certificate","openremote.io/endpoint":"mqtts"}},"spec":{"endpoints":[{"dnsName":"_validation.mqtt-stack-a.example.com.","recordType":"CNAME","recordTTL":300,"targets":["_mqtt-target.acm-validations.aws."]}]}}'
+DEFAULT_INGRESS_JSON='{"metadata":{"labels":{"openremote.io/managed-dns":"true"},"annotations":{"external-dns.kubernetes.io/hostname":"stack-a.example.com","alb.ingress.kubernetes.io/certificate-arn":"arn:aws:acm:eu-west-1:123456789012:certificate/test-certificate"}},"spec":{"rules":[{"host":"stack-a.example.com"}]}}'
+DEFAULT_MQTTS_SERVICE_JSON='{"metadata":{"labels":{"openremote.io/managed-dns":"true"},"annotations":{"external-dns.kubernetes.io/hostname":"mqtt-stack-a.example.com","service.beta.kubernetes.io/aws-load-balancer-type":"external","service.beta.kubernetes.io/aws-load-balancer-ssl-ports":"mqtts","service.beta.kubernetes.io/aws-load-balancer-ssl-cert":"arn:aws:acm:eu-west-1:123456789012:certificate/mqtts-certificate"}},"spec":{"type":"LoadBalancer","ports":[{"name":"mqtts","port":8883}]},"status":{"loadBalancer":{"ingress":[{"hostname":"test-mqtt-nlb.elb.eu-west-1.amazonaws.com"}]}}}'
+DEFAULT_EXTERNAL_DNS_ARGS=$'--source=service\n--source=ingress\n--source=crd\n--policy=sync\n--registry=txt\n--txt-owner-id=test-cluster-eu-west-1\n--label-filter=openremote.io/managed-dns=true\n--domain-filter=example.com'
+DEFAULT_HOSTNAME_REGISTRY_JSON="$(jq -cn '{metadata:{resourceVersion:"1",labels:{"app.kubernetes.io/managed-by":"or-stack"}},
+ data:{"stack-a.example.com":({namespace:"stack-a",namespaceUID:"uid-stack-a",target:"eks",dns:"external-dns",exposure:"ingress"}|tojson)}}')"
+
+cleanup_test() {
+ rm -rf "$TEST_TEMP_DIR"
+}
+
+trap cleanup_test EXIT
+
+# Source the command so external tools and the portable stack invocation can be
+# replaced with deterministic stubs.
+source "$KUBERNETES_DIR/or-eks-stack"
+
+fail() {
+ printf 'FAIL: %s\n' "$*" >&2
+ exit 1
+}
+
+log_call() {
+ local command="$1"
+ shift
+ printf '%s' "$command" >> "$CALL_LOG"
+ for argument in "$@"; do
+ printf '\t%s' "$argument" >> "$CALL_LOG"
+ done
+ printf '\n' >> "$CALL_LOG"
+}
+
+assert_contains() {
+ local expected="$1"
+ grep -F -- "$expected" "$CALL_LOG" >/dev/null || fail \
+ "Expected command log to contain: $expected"
+}
+
+assert_not_contains() {
+ local unexpected="$1"
+ if grep -F -- "$unexpected" "$CALL_LOG" >/dev/null; then
+ fail "Expected command log not to contain: $unexpected"
+ fi
+}
+
+assert_before() {
+ local first="$1"
+ local first_line
+ local second="$2"
+ local second_line
+
+ first_line="$(grep -nF -- "$first" "$CALL_LOG" | head -n 1 | cut -d: -f1)"
+ second_line="$(grep -nF -- "$second" "$CALL_LOG" | head -n 1 | cut -d: -f1)"
+ [ -n "$first_line" ] || fail "Expected command log to contain: $first"
+ [ -n "$second_line" ] || fail "Expected command log to contain: $second"
+ [ "$first_line" -lt "$second_line" ] || fail \
+ "Expected '$first' before '$second'"
+}
+
+reset_test() {
+ : > "$CALL_LOG"
+ : > "$DNS_ENDPOINT_LOG"
+ rm -f "$CERTIFICATE_DELETED_FILE" "$INGRESS_DELETED_FILE" \
+ "$MQTTS_CERTIFICATE_DELETED_FILE" "$NAMESPACE_DELETED_FILE" \
+ "$NAMESPACE_PREPARED_FILE" "$CERTIFICATE_NAMESPACE_FILE"
+ printf '0\n' > "$CURL_COUNT_FILE"
+ rm -rf "$PROXY_LE_DIR"
+ case "${OR_EKS_STACK_TEST_PROXY_CERTIFICATE_STATE:-absent}" in
+ present) create_proxy_certificate ;;
+ incomplete) mkdir -p "$PROXY_LE_DIR/live/stack-a.example.com" ;;
+ esac
+ COMMAND=""
+ STACK_NAME=""
+ CLUSTER_NAME=""
+ AWS_REGION_NAME="eu-west-1"
+ AWS_PROFILE_NAME=""
+ KUBE_CONTEXT=""
+ EXPOSURE="haproxy"
+ HOSTNAME_VALUE=""
+ CERTIFICATE_MODE=""
+ CERTIFICATE_ARN=""
+ VALUES_DIR=""
+ TIMEOUT="20m"
+ HTTP_PORT="80"
+ HTTPS_PORT="443"
+ MQTTS_ENABLED=false
+ MQTTS_PORT="8883"
+ MQTTS_PORT_CONFIGURED=false
+ MQTTS_HOSTNAME_VALUE=""
+ MQTTS_CERTIFICATE_MODE=""
+ MQTTS_CERTIFICATE_ARN=""
+ CONFIRM_STACK_NAME=""
+ OR_STACK_COMMAND="$KUBERNETES_DIR/or-stack"
+ LOAD_BALANCER_ADDRESS=""
+ MQTTS_LOAD_BALANCER_ADDRESS=""
+ EXTERNAL_DNS_ARGS=""
+ CERTIFICATE_JSON=""
+ CERTIFICATE_EXISTS=false
+ MQTTS_CERTIFICATE_JSON=""
+ MQTTS_CERTIFICATE_EXISTS=false
+ STACK_NAMESPACE_EXISTS=false
+ VALIDATION_DNS_RECORDS=""
+ MQTTS_VALIDATION_DNS_RECORDS=""
+ MANAGED_DNS_DOMAIN=""
+ AUTHORITATIVE_NAME_SERVERS=""
+ STACK_INGRESS_JSON=""
+ STACK_MQTTS_SERVICE_JSON=""
+ VALIDATION_DNS_ENDPOINT_EXISTS=false
+ MQTTS_VALIDATION_DNS_ENDPOINT_EXISTS=false
+ MANAGED_CERTIFICATE_DESTROY=false
+ MANAGED_MQTTS_CERTIFICATE_DESTROY=false
+}
+
+require_command() {
+ :
+}
+
+aws_eks() {
+ log_call aws-eks "$@"
+ case " $* " in
+ *" cluster.status "*)
+ printf '%s\n' "${OR_EKS_STACK_TEST_CLUSTER_STATUS:-ACTIVE}"
+ ;;
+ *" cluster.endpoint "*)
+ printf '%s\n' "$TEST_CLUSTER_ENDPOINT"
+ ;;
+ esac
+}
+
+aws_sts() {
+ log_call aws-sts "$@"
+ printf '%s\n' "${OR_EKS_STACK_TEST_ACCOUNT_ID:-123456789012}"
+}
+
+aws_acm() {
+ log_call aws-acm "$@"
+ case " $* " in
+ *" delete-certificate "*)
+ if [[ " $* " == *" $TEST_MQTTS_CERTIFICATE_ARN "* ]]; then
+ : > "$MQTTS_CERTIFICATE_DELETED_FILE"
+ else
+ : > "$CERTIFICATE_DELETED_FILE"
+ fi
+ ;;
+ *" list-certificates "*)
+ printf '%s\n' "${OR_EKS_STACK_TEST_CERTIFICATE_LIST-$DEFAULT_CERTIFICATE_LIST}"
+ ;;
+ *" request-certificate "*)
+ [ "${OR_EKS_STACK_TEST_ACM_REQUEST_FAIL:-false}" = false ] || return 1
+ if [ -n "${OR_EKS_STACK_TEST_REQUESTED_CERTIFICATE_ARN:-}" ]; then
+ printf '%s\n' "$OR_EKS_STACK_TEST_REQUESTED_CERTIFICATE_ARN"
+ elif [[ " $* " == *" --domain-name mqtt-stack-a.example.com "* ]]; then
+ printf '%s\n' "$TEST_MQTTS_CERTIFICATE_ARN"
+ else
+ printf '%s\n' "$TEST_CERTIFICATE_ARN"
+ fi
+ ;;
+ *" list-tags-for-certificate "*)
+ if [[ " $* " == *" $TEST_MQTTS_CERTIFICATE_ARN "* ]]; then
+ printf '%s\n' "${OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_TAGS-${OR_EKS_STACK_TEST_CERTIFICATE_TAGS-$DEFAULT_CERTIFICATE_TAGS}}"
+ else
+ printf '%s\n' "${OR_EKS_STACK_TEST_CERTIFICATE_TAGS-$DEFAULT_CERTIFICATE_TAGS}"
+ fi
+ ;;
+ *" Certificate.Status "*)
+ if { [[ " $* " == *" $TEST_MQTTS_CERTIFICATE_ARN "* ]] && \
+ [ -f "$MQTTS_CERTIFICATE_DELETED_FILE" ]; } || \
+ { [[ " $* " != *" $TEST_MQTTS_CERTIFICATE_ARN "* ]] && \
+ [ -f "$CERTIFICATE_DELETED_FILE" ]; }; then
+ printf '%s\n' 'ResourceNotFoundException' >&2
+ return 1
+ fi
+ if [[ " $* " == *" $TEST_MQTTS_CERTIFICATE_ARN "* ]]; then
+ printf '%s\n' "${OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_STATUS-${OR_EKS_STACK_TEST_CERTIFICATE_STATUS:-ISSUED}}"
+ else
+ printf '%s\n' "${OR_EKS_STACK_TEST_CERTIFICATE_STATUS:-ISSUED}"
+ fi
+ ;;
+ *" describe-certificate "*)
+ if { [[ " $* " == *" $TEST_MQTTS_CERTIFICATE_ARN "* ]] && \
+ [ -f "$MQTTS_CERTIFICATE_DELETED_FILE" ]; } || \
+ { [[ " $* " != *" $TEST_MQTTS_CERTIFICATE_ARN "* ]] && \
+ [ -f "$CERTIFICATE_DELETED_FILE" ]; }; then
+ printf '%s\n' 'ResourceNotFoundException' >&2
+ return 1
+ fi
+ if [[ " $* " == *" $TEST_MQTTS_CERTIFICATE_ARN "* ]]; then
+ printf '%s\n' "${OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_JSON-${OR_EKS_STACK_TEST_CERTIFICATE_JSON-$DEFAULT_CERTIFICATE_JSON}}"
+ else
+ printf '%s\n' "${OR_EKS_STACK_TEST_CERTIFICATE_JSON-$DEFAULT_CERTIFICATE_JSON}"
+ fi
+ ;;
+ esac
+}
+
+aws_elbv2() {
+ log_call aws-elbv2 "$@"
+ case "$1" in
+ describe-load-balancers)
+ if [ -n "${OR_EKS_STACK_TEST_ELB_LIST_ERROR:-}" ]; then
+ printf '%s\n' "$OR_EKS_STACK_TEST_ELB_LIST_ERROR" >&2
+ return 1
+ fi
+ if [ -n "${OR_EKS_STACK_TEST_LOAD_BALANCERS_JSON:-}" ]; then
+ printf '%s\n' "$OR_EKS_STACK_TEST_LOAD_BALANCERS_JSON"
+ elif [ -f "$NAMESPACE_DELETED_FILE" ] || \
+ [ "${OR_EKS_STACK_TEST_NAMESPACE_EXISTS:-true}" = false ]; then
+ printf '%s\n' '{"LoadBalancers":[]}'
+ else
+ printf '%s\n' "$DEFAULT_LOAD_BALANCERS_JSON"
+ fi
+ ;;
+ describe-tags)
+ if [ -n "${OR_EKS_STACK_TEST_ELB_TAGS_ERROR:-}" ]; then
+ printf '%s\n' "$OR_EKS_STACK_TEST_ELB_TAGS_ERROR" >&2
+ return 1
+ fi
+ if [ -n "${OR_EKS_STACK_TEST_ELB_TAGS_JSON:-}" ]; then
+ printf '%s\n' "$OR_EKS_STACK_TEST_ELB_TAGS_JSON"
+ else
+ jq -cn --arg arn "$3" \
+ --arg cluster "${OR_EKS_STACK_TEST_NLB_CLUSTER:-test-cluster}" \
+ --arg service "${OR_EKS_STACK_TEST_NLB_SERVICE:-stack-a/proxy-mqtts}" \
+ '{TagDescriptions:[{ResourceArn:$arn,Tags:[
+ {Key:"elbv2.k8s.aws/cluster",Value:$cluster},
+ {Key:"service.k8s.aws/stack",Value:$service}]}]}'
+ fi
+ ;;
+ *) fail "Unexpected ELB command: $*" ;;
+ esac
+}
+
+helm() {
+ log_call helm "$@"
+ [ "${OR_EKS_STACK_TEST_EXTERNAL_DNS_HELM_READY:-true}" = "true" ]
+}
+
+kubectl() {
+ log_call kubectl "$@"
+ local arguments=" $* "
+ local endpoint_manifest
+
+ case "$arguments" in
+ *" get namespace stack-a"*" -o json "*)
+ [ "${OR_EKS_STACK_TEST_NAMESPACE_READ_SUCCESS:-true}" = true ] || return 1
+ if [ -f "$CERTIFICATE_NAMESPACE_FILE" ]; then
+ cat "$CERTIFICATE_NAMESPACE_FILE"
+ elif [ "${OR_EKS_STACK_TEST_NAMESPACE_EXISTS:-true}" = true ] || [ -f "$NAMESPACE_PREPARED_FILE" ]; then
+ jq -cn --arg mode "${OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE:-}" \
+ --arg arn "${OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN:-}" \
+ --arg mqttsMode "${OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE:-}" \
+ --arg mqttsArn "${OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_ARN:-}" \
+ --arg mqttsHostname "${OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME:-}" \
+ --arg owner "${OR_EKS_STACK_TEST_NAMESPACE_STACK_LABEL:-stack-a}" \
+ '{metadata:{name:"stack-a",uid:"uid-stack-a",resourceVersion:"1",labels:{"openremote.io/stack":$owner},
+ annotations:{"openremote.io/certificate-mode":$mode,"openremote.io/certificate-arn":$arn,
+ "openremote.io/mqtts-certificate-mode":$mqttsMode,
+ "openremote.io/mqtts-certificate-arn":$mqttsArn,
+ "openremote.io/mqtts-hostname":$mqttsHostname}}}'
+ fi
+ ;;
+ *" annotate namespace stack-a "*)
+ local certificate_namespace argument annotation_key annotation_value
+ certificate_namespace="$(kubectl --context "$KUBE_CONTEXT" get namespace stack-a -o json)" || return 1
+ for argument in "$@"; do
+ case "$argument" in
+ --resource-version=*)
+ if [ "${OR_EKS_STACK_TEST_MODE_CONFLICT:-false}" = true ] || \
+ [ "${argument#*=}" != "$(jq -r '.metadata.resourceVersion' <<< "$certificate_namespace")" ]; then
+ printf 'Error from server (Conflict): namespace has changed (requested %s, current %s)\n' \
+ "${argument#*=}" "$(jq -r '.metadata.resourceVersion' <<< "$certificate_namespace")" >&2
+ return 1
+ fi
+ ;;
+ openremote.io/*=*)
+ annotation_key="${argument%%=*}"
+ annotation_value="${argument#*=}"
+ certificate_namespace="$(jq --arg key "$annotation_key" --arg value "$annotation_value" \
+ '.metadata.annotations[$key]=$value' <<< "$certificate_namespace")"
+ ;;
+ esac
+ done
+ jq '.metadata.resourceVersion |= (tonumber + 1 | tostring)' \
+ <<< "$certificate_namespace" > "$CERTIFICATE_NAMESPACE_FILE"
+ ;;
+ *" get configmap openremote-hostnames "*)
+ printf '%s' "${OR_EKS_STACK_TEST_REGISTRY_JSON-$DEFAULT_HOSTNAME_REGISTRY_JSON}"
+ ;;
+ *" get ingressclass openremote-alb "*"shared-certificate-arn"*)
+ printf '%s' "${OR_EKS_STACK_TEST_SHARED_CERTIFICATE_ARN-}"
+ ;;
+ *" config get-contexts "*)
+ if [ "${OR_EKS_STACK_TEST_CONTEXT_EXISTS:-true}" = "true" ]; then
+ printf '%s\n' "${KUBE_CONTEXT:-test-cluster@eu-west-1}"
+ fi
+ ;;
+ *" config view "*)
+ printf '%s' "${OR_EKS_STACK_TEST_CONTEXT_ENDPOINT:-$TEST_CLUSTER_ENDPOINT}"
+ ;;
+ *" get namespace kube-system "*)
+ [ "${OR_EKS_STACK_TEST_CONTEXT_REACHABLE:-true}" = "true" ]
+ ;;
+ *" get namespace stack-a "*"mqtts-certificate-mode"*)
+ if [ -f "$CERTIFICATE_NAMESPACE_FILE" ]; then
+ jq -r '.metadata.annotations["openremote.io/mqtts-certificate-mode"] // empty' "$CERTIFICATE_NAMESPACE_FILE"
+ else
+ printf '%s' "${OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE:-}"
+ fi
+ ;;
+ *" get namespace stack-a "*"mqtts-certificate-arn"*)
+ if [ -f "$CERTIFICATE_NAMESPACE_FILE" ]; then
+ jq -r '.metadata.annotations["openremote.io/mqtts-certificate-arn"] // empty' "$CERTIFICATE_NAMESPACE_FILE"
+ else
+ printf '%s' "${OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_ARN:-}"
+ fi
+ ;;
+ *" get namespace stack-a "*"mqtts-hostname"*)
+ if [ -f "$CERTIFICATE_NAMESPACE_FILE" ]; then
+ jq -r '.metadata.annotations["openremote.io/mqtts-hostname"] // empty' "$CERTIFICATE_NAMESPACE_FILE"
+ else
+ printf '%s' "${OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME:-}"
+ fi
+ ;;
+ *" get namespace stack-a "*"certificate-mode"*)
+ if [ -f "$CERTIFICATE_NAMESPACE_FILE" ]; then
+ jq -r '.metadata.annotations["openremote.io/certificate-mode"] // empty' "$CERTIFICATE_NAMESPACE_FILE"
+ else
+ printf '%s' "${OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE:-acme}"
+ fi
+ ;;
+ *" get namespace stack-a "*"certificate-arn"*)
+ if [ -f "$CERTIFICATE_NAMESPACE_FILE" ]; then
+ jq -r '.metadata.annotations["openremote.io/certificate-arn"] // empty' "$CERTIFICATE_NAMESPACE_FILE"
+ else
+ printf '%s' "${OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN:-}"
+ fi
+ ;;
+ *" get namespace stack-a "*"openremote.io/hostname"*)
+ printf '%s' "${OR_EKS_STACK_TEST_RECORDED_HOSTNAME:-}"
+ ;;
+ *" get namespace stack-a "*"openremote.io/stack"*)
+ if [ "${OR_EKS_STACK_TEST_NAMESPACE_EXISTS:-true}" = "true" ] && \
+ [ ! -f "$NAMESPACE_DELETED_FILE" ]; then
+ printf 'stack-a|%s' "${OR_EKS_STACK_TEST_NAMESPACE_STACK_LABEL:-stack-a}"
+ fi
+ ;;
+ *" rollout status deployment/openremote-external-dns "*)
+ [ "${OR_EKS_STACK_TEST_EXTERNAL_DNS_ROLLOUT_READY:-true}" = "true" ]
+ ;;
+ *" get deployment openremote-external-dns "*)
+ printf '%s\n' "${OR_EKS_STACK_TEST_EXTERNAL_DNS_ARGS-$DEFAULT_EXTERNAL_DNS_ARGS}"
+ ;;
+ *" get customresourcedefinition/dnsendpoints.externaldns.k8s.io "*)
+ if [ "${OR_EKS_STACK_TEST_DNS_ENDPOINT_CRD:-true}" = "true" ]; then
+ printf '%s\n' 'customresourcedefinition.apiextensions.k8s.io/dnsendpoints.externaldns.k8s.io'
+ else
+ return 1
+ fi
+ ;;
+ *" get serviceaccount external-dns "*)
+ printf '%s' "${OR_EKS_STACK_TEST_EXTERNAL_DNS_ROLE_ARN-arn:aws:iam::123456789012:role/openremote-external-dns}"
+ ;;
+ *" get dnsendpoint acm-validation "*)
+ if [ -n "${OR_EKS_STACK_TEST_DNS_ENDPOINT_READ_ERROR:-}" ]; then
+ printf '%s\n' "$OR_EKS_STACK_TEST_DNS_ENDPOINT_READ_ERROR" >&2
+ return 1
+ fi
+ if [ -f "$NAMESPACE_DELETED_FILE" ]; then
+ return 0
+ fi
+ if [ "${OR_EKS_STACK_TEST_DNS_ENDPOINT_EXISTS:-false}" != "true" ]; then
+ [[ "$arguments" == *" --ignore-not-found "* ]] && return 0
+ return 1
+ fi
+ printf '%s\n' "${OR_EKS_STACK_TEST_DNS_ENDPOINT_JSON-$DEFAULT_DNS_ENDPOINT_JSON}"
+ ;;
+ *" get dnsendpoint acm-validation-mqtts "*)
+ if [ -n "${OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_READ_ERROR:-}" ]; then
+ printf '%s\n' "$OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_READ_ERROR" >&2
+ return 1
+ fi
+ [ "${OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_READ_SUCCESS:-true}" = true ] || return 1
+ if [ -f "$NAMESPACE_DELETED_FILE" ]; then
+ return 0
+ fi
+ if [ "${OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_EXISTS:-false}" != true ]; then
+ [[ "$arguments" == *" --ignore-not-found "* ]] && return 0
+ return 1
+ fi
+ printf '%s\n' "${OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_JSON-$DEFAULT_MQTTS_DNS_ENDPOINT_JSON}"
+ ;;
+ *" apply -f - "*)
+ IFS= read -r endpoint_manifest
+ printf '%s\n' "$endpoint_manifest" > "$DNS_ENDPOINT_LOG"
+ ;;
+ *" get service proxy-mqtts "*" -o json "*)
+ [ "${OR_EKS_STACK_TEST_MQTTS_SERVICE_READ_SUCCESS:-true}" = true ] || return 1
+ if [ "${OR_EKS_STACK_TEST_MQTTS_SERVICE_EXISTS:-true}" = true ] && \
+ [ ! -f "$NAMESPACE_DELETED_FILE" ]; then
+ printf '%s\n' "${OR_EKS_STACK_TEST_MQTTS_SERVICE_JSON-$DEFAULT_MQTTS_SERVICE_JSON}"
+ fi
+ ;;
+ *" get service proxy-mqtts "*)
+ printf '%s' "${OR_EKS_STACK_TEST_MQTTS_NLB_ADDRESS:-test-mqtt-nlb.elb.eu-west-1.amazonaws.com}"
+ ;;
+ *" get service proxy "*)
+ printf '%s' "${OR_EKS_STACK_TEST_NLB_ADDRESS:-test-nlb.elb.eu-west-1.amazonaws.com}"
+ ;;
+ *" get ingress proxy "*" -o json "*)
+ if [ "${OR_EKS_STACK_TEST_INGRESS_EXISTS:-false}" = "true" ] && \
+ [ ! -f "$INGRESS_DELETED_FILE" ] && \
+ [ ! -f "$NAMESPACE_DELETED_FILE" ]; then
+ printf '%s\n' "${OR_EKS_STACK_TEST_INGRESS_JSON-$DEFAULT_INGRESS_JSON}"
+ fi
+ ;;
+ *" delete ingress proxy "*)
+ : > "$INGRESS_DELETED_FILE"
+ ;;
+ *" get ingress proxy "*)
+ printf '%s' "${OR_EKS_STACK_TEST_ALB_ADDRESS:-test-alb.elb.eu-west-1.amazonaws.com}"
+ ;;
+ *" exec deployment/proxy "*" -- sh -c "*)
+ [ "${OR_EKS_STACK_TEST_CERTIFICATE_INSPECTION_SUCCESS:-true}" = true ] || return 1
+ # Execute the actual read-only probe against temporary persisted state.
+ while [ "$1" != -- ]; do shift; done
+ shift
+ LE_DIR="$PROXY_LE_DIR" command "$@"
+ ;;
+ *" exec deployment/proxy "*" /entrypoint.sh add "*)
+ if [ "${OR_EKS_STACK_TEST_STARTUP_ISSUES_CERTIFICATE:-false}" = true ]; then
+ create_proxy_certificate
+ fi
+ [ "${OR_EKS_STACK_TEST_ACME_SUCCESS:-true}" = "true" ]
+ ;;
+ esac
+}
+
+dig() {
+ local argument
+ local authoritative=false
+ local dns_name=""
+ local record_type=""
+
+ log_call dig "$@"
+ for argument in "$@"; do
+ case "$argument" in
+ @*) authoritative=true ;;
+ A|AAAA|CNAME|NS) record_type="$argument" ;;
+ +*) ;;
+ *) dns_name="$argument" ;;
+ esac
+ done
+
+ if [ "$record_type" = "NS" ]; then
+ [ "$dns_name" = "example.com" ] && printf '%s\n' 'ns-1.example.net.'
+ return
+ fi
+
+ if [ "$authoritative" = true ]; then
+ printf ';; ->>HEADER<<- opcode: QUERY, status: %s, id: 1\n' "${OR_EKS_STACK_TEST_DNS_STATUS:-NOERROR}"
+ printf ';; flags: %s; QUERY: 1, ANSWER: 0, AUTHORITY: 1\n' "${OR_EKS_STACK_TEST_DNS_FLAGS:-qr aa}"
+ [ "${OR_EKS_STACK_TEST_DNS_STATUS:-NOERROR}" = NOERROR ] || return 0
+ case "$dns_name" in
+ stack-a.example.com)
+ if [ "${OR_EKS_STACK_TEST_NAMESPACE_EXISTS:-true}" = true ] &&
+ [ ! -f "$INGRESS_DELETED_FILE" ] && [ ! -f "$NAMESPACE_DELETED_FILE" ]; then
+ printf '%s. 60 IN A 192.0.2.10\n' "$dns_name"
+ fi
+ ;;
+ mqtt-stack-a.example.com)
+ if [ "${OR_EKS_STACK_TEST_NAMESPACE_EXISTS:-true}" = true ] && \
+ [ ! -f "$NAMESPACE_DELETED_FILE" ]; then
+ printf '%s. 60 IN A 192.0.2.20\n' "$dns_name"
+ fi
+ ;;
+ _validation.stack-a.example.com|_validation.stack-a.example.com.)
+ if [ "${OR_EKS_STACK_TEST_NAMESPACE_EXISTS:-true}" = true ] && [ ! -f "$NAMESPACE_DELETED_FILE" ]; then
+ printf '%s 300 IN CNAME _target.acm-validations.aws.\n' "$dns_name"
+ fi
+ ;;
+ _validation.mqtt-stack-a.example.com|_validation.mqtt-stack-a.example.com.)
+ if [ "${OR_EKS_STACK_TEST_NAMESPACE_EXISTS:-true}" = true ] && \
+ [ ! -f "$NAMESPACE_DELETED_FILE" ]; then
+ printf '%s 300 IN CNAME _mqtt-target.acm-validations.aws.\n' "$dns_name"
+ fi
+ ;;
+ esac
+ return 0
+ fi
+
+ case "$dns_name" in
+ stack-a.example.com)
+ if [ "$authoritative" = false ] || \
+ { [ "${OR_EKS_STACK_TEST_NAMESPACE_EXISTS:-true}" = "true" ] && \
+ [ ! -f "$INGRESS_DELETED_FILE" ]; }; then
+ printf '%s\n' "${OR_EKS_STACK_TEST_HOST_ADDRESS:-192.0.2.10}"
+ fi
+ ;;
+ mqtt-stack-a.example.com)
+ if [ "$authoritative" = false ] || \
+ { [ "${OR_EKS_STACK_TEST_NAMESPACE_EXISTS:-true}" = "true" ] && \
+ [ ! -f "$NAMESPACE_DELETED_FILE" ]; }; then
+ printf '%s\n' "${OR_EKS_STACK_TEST_MQTTS_HOST_ADDRESS:-192.0.2.20}"
+ fi
+ ;;
+ _validation.stack-a.example.com|_validation.stack-a.example.com.)
+ if [ "${OR_EKS_STACK_TEST_NAMESPACE_EXISTS:-true}" = "true" ] && \
+ [ ! -f "$NAMESPACE_DELETED_FILE" ]; then
+ printf '%s\n' '_target.acm-validations.aws.'
+ fi
+ ;;
+ _validation.mqtt-stack-a.example.com|_validation.mqtt-stack-a.example.com.)
+ if [ "${OR_EKS_STACK_TEST_NAMESPACE_EXISTS:-true}" = true ] && \
+ [ ! -f "$NAMESPACE_DELETED_FILE" ]; then
+ printf '%s\n' '_mqtt-target.acm-validations.aws.'
+ fi
+ ;;
+ test-nlb.elb.eu-west-1.amazonaws.com)
+ printf '%s\n' "${OR_EKS_STACK_TEST_NLB_RESOLVED_ADDRESS:-192.0.2.10}"
+ ;;
+ test-alb.elb.eu-west-1.amazonaws.com)
+ printf '%s\n' "${OR_EKS_STACK_TEST_ALB_RESOLVED_ADDRESS:-192.0.2.10}"
+ ;;
+ test-mqtt-nlb.elb.eu-west-1.amazonaws.com)
+ printf '%s\n' "${OR_EKS_STACK_TEST_MQTTS_NLB_RESOLVED_ADDRESS:-192.0.2.20}"
+ ;;
+ esac
+}
+
+curl() {
+ local curl_count
+
+ log_call curl "$@"
+ curl_count="$(< "$CURL_COUNT_FILE")"
+ curl_count="$((curl_count + 1))"
+ printf '%s\n' "$curl_count" > "$CURL_COUNT_FILE"
+ [ "${OR_EKS_STACK_TEST_HTTPS_ALWAYS_FAILS:-false}" = false ] || return 1
+ if [ "${OR_EKS_STACK_TEST_HTTPS_READY_IMMEDIATELY:-false}" = "true" ]; then
+ return 0
+ fi
+ [ "$curl_count" -gt 1 ]
+}
+
+python3() {
+ log_call python3 "$@"
+ # Real TLS behavior is covered separately by mqtts-readiness-test.
+ cat >/dev/null
+ [ "${OR_EKS_STACK_TEST_MQTTS_ALWAYS_FAILS:-false}" = false ]
+}
+
+sleep() {
+ :
+}
+
+run_or_stack() {
+ log_call or-stack "$@"
+ if [ "${1:-}" = "prepare" ] && [ "${OR_EKS_STACK_TEST_PREPARE_FAIL:-false}" = true ]; then
+ return 1
+ fi
+ if [ "${1:-}" = "prepare" ]; then
+ : > "$NAMESPACE_PREPARED_FILE"
+ fi
+ if [ "${1:-}" = "destroy" ]; then
+ : > "$NAMESPACE_DELETED_FILE"
+ fi
+}
+
+run_eks_stack() {
+ reset_test
+ main "$@"
+}
+
+test_apply() {
+ local output
+ local values_dir="$TEST_TEMP_DIR/values"
+
+ mkdir -p "$values_dir"
+ output="$(run_eks_stack apply \
+ --name stack-a \
+ --cluster test-cluster \
+ --hostname stack-a.example.com \
+ --profile '' \
+ --https-port 8443 \
+ --mqtts \
+ --mqtts-port 18883 \
+ --values-dir "$values_dir")"
+
+ assert_contains $'aws-eks\tdescribe-cluster\t--name\ttest-cluster\t--query\tcluster.status'
+ assert_contains $'kubectl\tconfig\tget-contexts\ttest-cluster@eu-west-1\t-o\tname'
+ assert_contains $'aws-eks\tdescribe-cluster\t--name\ttest-cluster\t--query\tcluster.endpoint'
+ assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\tconfig\tview\t--minify'
+ assert_contains $'helm\tstatus\topenremote-external-dns\t--kube-context\ttest-cluster@eu-west-1\t--namespace\tkube-system'
+ assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\t--namespace\tkube-system\trollout\tstatus\tdeployment/openremote-external-dns'
+ assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\t--namespace\tkube-system\tget\tserviceaccount\texternal-dns'
+ assert_contains $'or-stack\tapply\t--name\tstack-a\t--kube-context\ttest-cluster@eu-west-1\t--target\teks\t--exposure\thaproxy\t--dns\texternal-dns\t--hostname\tstack-a.example.com\t--timeout\t20m\t--http-port\t80\t--https-port\t8443\t--mqtts\t--mqtts-port\t18883\t--values-dir\t'"$values_dir"
+ assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\tannotate\tnamespace\tstack-a\topenremote.io/certificate-mode=acme\topenremote.io/hostname=stack-a.example.com\t--overwrite'
+ assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\t--namespace\tstack-a\tget\tservice\tproxy'
+ assert_contains $'dig\t+short\tstack-a.example.com\tA'
+ assert_contains $'dig\t+short\ttest-nlb.elb.eu-west-1.amazonaws.com\tA'
+ assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\t--namespace\tstack-a\texec\tdeployment/proxy\t--container\tproxy\t--\t/entrypoint.sh\tadd\tstack-a.example.com'
+ assert_contains $'curl\t--disable\t--fail\t--silent\t--show-error\t--location'
+ assert_contains $'--proto\t=https\t--proto-redir\t=https\t--max-redirs\t5'
+ assert_contains 'https://stack-a.example.com:8443/manager/'
+ assert_contains $'python3\t-\tstack-a.example.com\t18883\t10'
+ assert_before $'helm\tstatus\topenremote-external-dns' $'or-stack\tapply'
+ assert_before $'or-stack\tapply' $'get\tservice\tproxy'
+ assert_before $'dig\t+short\tstack-a.example.com' $'exec\tdeployment/proxy'
+ grep -F -- 'Stack '\''stack-a'\'' is ready at:' <<< "$output" >/dev/null || fail \
+ "Apply did not report stack readiness"
+ grep -F -- 'https://stack-a.example.com:8443/manager/' <<< "$output" >/dev/null || fail \
+ "Apply did not report the canonical Manager URL"
+ grep -F -- 'mqtts://stack-a.example.com:18883' <<< "$output" >/dev/null || fail \
+ "Apply did not report the MQTTS endpoint"
+}
+
+test_ingress_existing_certificate_apply() {
+ local output
+
+ output="$(OR_EKS_STACK_TEST_SHARED_CERTIFICATE_ARN="$TEST_SECOND_CERTIFICATE_ARN" \
+ run_eks_stack apply \
+ --name stack-a \
+ --cluster test-cluster \
+ --hostname stack-a.example.com \
+ --profile '' \
+ --exposure ingress \
+ --certificate-mode existing \
+ --certificate-arn "$TEST_CERTIFICATE_ARN")"
+
+ assert_contains $'aws-sts\tget-caller-identity\t--query\tAccount\t--output\ttext'
+ assert_contains $'aws-acm\tdescribe-certificate\t--certificate-arn\t'"$TEST_CERTIFICATE_ARN"$'\t--output\tjson'
+ assert_contains $'aws-acm\tlist-tags-for-certificate\t--certificate-arn\t'"$TEST_CERTIFICATE_ARN"$'\t--output\tjson'
+ assert_contains $'or-stack\tapply\t--name\tstack-a\t--kube-context\ttest-cluster@eu-west-1\t--target\teks\t--exposure\tingress\t--dns\texternal-dns\t--hostname\tstack-a.example.com\t--timeout\t20m\t--certificate-arn\t'"$TEST_CERTIFICATE_ARN"
+ assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\tannotate\tnamespace\tstack-a\topenremote.io/certificate-mode=existing\topenremote.io/certificate-arn='"$TEST_CERTIFICATE_ARN"$'\topenremote.io/hostname=stack-a.example.com\t--overwrite'
+ assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\t--namespace\tstack-a\tget\tingress\tproxy'
+ assert_contains $'dig\t+short\ttest-alb.elb.eu-west-1.amazonaws.com\tA'
+ assert_contains 'https://stack-a.example.com/manager/'
+ assert_not_contains $'exec\tdeployment/proxy'
+ assert_before $'aws-acm\tdescribe-certificate' $'or-stack\tapply'
+ assert_before $'or-stack\tapply' $'get\tingress\tproxy'
+ grep -F -- "Validated user-owned ACM certificate '$TEST_CERTIFICATE_ARN'" \
+ <<< "$output" >/dev/null || fail "Apply did not report ACM validation"
+ grep -F -- "The shared ALB is 'test-alb.elb.eu-west-1.amazonaws.com'" \
+ <<< "$output" >/dev/null || fail "Apply did not report shared ALB readiness"
+}
+
+test_ingress_mqtts_existing_certificate_lifecycle() {
+ local failure_output
+ local output
+ local registry
+
+ output="$(OR_EKS_STACK_TEST_SHARED_CERTIFICATE_ARN="$TEST_SECOND_CERTIFICATE_ARN" \
+ run_eks_stack apply \
+ --name stack-a \
+ --cluster test-cluster \
+ --hostname stack-a.example.com \
+ --profile '' \
+ --exposure ingress \
+ --certificate-mode existing \
+ --certificate-arn "$TEST_CERTIFICATE_ARN" \
+ --mqtts \
+ --mqtts-hostname mqtt-stack-a.example.com \
+ --mqtts-certificate-arn "$TEST_MQTTS_CERTIFICATE_ARN")"
+
+ assert_contains $'or-stack\tprepare\t--name\tstack-a\t--kube-context\ttest-cluster@eu-west-1\t--target\teks\t--exposure\tingress\t--dns\texternal-dns\t--hostname\tstack-a.example.com\t--mqtts\t--mqtts-hostname\tmqtt-stack-a.example.com'
+ assert_contains $'aws-acm\tdescribe-certificate\t--certificate-arn\t'"$TEST_MQTTS_CERTIFICATE_ARN"$'\t--output\tjson'
+ assert_contains $'or-stack\tapply\t--name\tstack-a\t--kube-context\ttest-cluster@eu-west-1\t--target\teks\t--exposure\tingress\t--dns\texternal-dns\t--hostname\tstack-a.example.com\t--timeout\t20m\t--certificate-arn\t'"$TEST_CERTIFICATE_ARN"$'\t--mqtts\t--mqtts-port\t8883\t--mqtts-hostname\tmqtt-stack-a.example.com\t--mqtts-certificate-arn\t'"$TEST_MQTTS_CERTIFICATE_ARN"
+ assert_contains $'openremote.io/mqtts-certificate-mode=existing\topenremote.io/mqtts-hostname=mqtt-stack-a.example.com\t--overwrite\t--resource-version=2'
+ assert_contains $'openremote.io/mqtts-certificate-mode=existing\topenremote.io/mqtts-certificate-arn='"$TEST_MQTTS_CERTIFICATE_ARN"$'\topenremote.io/mqtts-hostname=mqtt-stack-a.example.com\t--overwrite'
+ assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\t--namespace\tstack-a\tget\tservice\tproxy-mqtts'
+ assert_contains $'dig\t+short\tmqtt-stack-a.example.com\tA'
+ assert_contains $'dig\t+short\ttest-mqtt-nlb.elb.eu-west-1.amazonaws.com\tA'
+ assert_contains $'python3\t-\tmqtt-stack-a.example.com\t8883\t10'
+ assert_before $'or-stack\tprepare' $'openremote.io/mqtts-certificate-mode=existing\topenremote.io/mqtts-hostname='
+ assert_before $'openremote.io/mqtts-certificate-mode=existing\topenremote.io/mqtts-hostname=' $'or-stack\tapply'
+ assert_before $'or-stack\tapply' $'openremote.io/mqtts-certificate-arn='
+ assert_before $'get\tservice\tproxy-mqtts' $'dig\t+short\tmqtt-stack-a.example.com'
+ grep -F -- "Validated user-owned ACM certificate '$TEST_MQTTS_CERTIFICATE_ARN' for MQTTS hostname 'mqtt-stack-a.example.com'." \
+ <<< "$output" >/dev/null || fail "Apply did not report MQTTS ACM validation"
+ grep -F -- 'mqtts://mqtt-stack-a.example.com:8883' <<< "$output" >/dev/null || fail \
+ "Apply did not report the dedicated MQTTS endpoint"
+ jq -e --arg arn "$TEST_MQTTS_CERTIFICATE_ARN" '
+ .metadata.annotations["openremote.io/mqtts-certificate-mode"] == "existing" and
+ .metadata.annotations["openremote.io/mqtts-certificate-arn"] == $arn and
+ .metadata.annotations["openremote.io/mqtts-hostname"] == "mqtt-stack-a.example.com"' \
+ "$CERTIFICATE_NAMESPACE_FILE" >/dev/null || fail \
+ "Apply did not persist the MQTTS lifecycle configuration"
+
+ OR_EKS_STACK_TEST_CERTIFICATE_TAGS="$MANAGED_CERTIFICATE_TAGS" \
+ OR_EKS_STACK_TEST_CERTIFICATE_JSON="$MANAGED_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_TAGS="$DEFAULT_CERTIFICATE_TAGS" \
+ OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_JSON="$DEFAULT_CERTIFICATE_JSON" \
+ run_eks_stack apply \
+ --name stack-a --cluster test-cluster --hostname stack-a.example.com \
+ --profile '' --exposure ingress --certificate-mode managed \
+ --mqtts --mqtts-hostname mqtt-stack-a.example.com \
+ --mqtts-certificate-arn "$TEST_MQTTS_CERTIFICATE_ARN" >/dev/null
+ assert_contains $'--certificate-arn\t'"$TEST_CERTIFICATE_ARN"$'\t--mqtts\t--mqtts-port\t8883'
+ assert_contains $'openremote.io/mqtts-certificate-arn='"$TEST_MQTTS_CERTIFICATE_ARN"
+
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_ARN="$TEST_MQTTS_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME=mqtt-stack-a.example.com \
+ run_eks_stack apply \
+ --name stack-a --cluster test-cluster --hostname stack-a.example.com \
+ --profile '' --exposure ingress --mqtts >/dev/null
+ assert_contains $'--mqtts-hostname\tmqtt-stack-a.example.com\t--mqtts-certificate-arn\t'"$TEST_MQTTS_CERTIFICATE_ARN"
+
+ output="$(OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_ARN="$TEST_MQTTS_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME=mqtt-stack-a.example.com \
+ run_eks_stack status --name stack-a --cluster test-cluster --profile '')"
+ grep -F 'MQTTS certificate mode: existing' <<< "$output" >/dev/null || fail \
+ "Status did not report the MQTTS certificate mode"
+ grep -F 'MQTTS hostname: mqtt-stack-a.example.com' <<< "$output" >/dev/null || fail \
+ "Status did not report the MQTTS hostname"
+ grep -F "MQTTS certificate ARN: $TEST_MQTTS_CERTIFICATE_ARN" <<< "$output" >/dev/null || fail \
+ "Status did not report the MQTTS certificate ARN"
+
+ if failure_output="$(OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_ARN="$TEST_MQTTS_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME=mqtt-stack-a.example.com \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --profile '' --exposure ingress 2>&1)"; then
+ fail "Ingress reapply silently removed its recorded MQTTS endpoint"
+ fi
+ grep -F 'repeat --mqtts to preserve it' <<< "$failure_output" >/dev/null || fail "$failure_output"
+
+ registry="$(jq -cn '{metadata:{resourceVersion:"1",labels:{"app.kubernetes.io/managed-by":"or-stack"}},
+ data:{
+ "stack-a.example.com":({namespace:"stack-a",namespaceUID:"uid-stack-a",target:"eks",dns:"external-dns",exposure:"ingress",endpoint:"web"}|tojson),
+ "mqtt-stack-a.example.com":({namespace:"stack-a",namespaceUID:"uid-stack-a",target:"eks",dns:"external-dns",exposure:"ingress",endpoint:"mqtts"}|tojson)
+ }}')"
+ OR_EKS_STACK_TEST_REGISTRY_JSON="$registry" \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_ARN="$TEST_MQTTS_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME=mqtt-stack-a.example.com \
+ run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a >/dev/null
+ assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\t--namespace\tstack-a\tget\tservice\tproxy-mqtts\t--ignore-not-found\t-o\tjson'
+ assert_contains $'aws-elbv2\tdescribe-load-balancers\t--output\tjson'
+ assert_not_contains $'aws-elbv2\tdescribe-load-balancers\t--names'
+ assert_contains $'or-stack\trelease-hostname\t--name\tstack-a\t--kube-context\ttest-cluster@eu-west-1\t--hostname\tstack-a.example.com'
+ assert_contains $'or-stack\trelease-hostname\t--name\tstack-a\t--kube-context\ttest-cluster@eu-west-1\t--hostname\tmqtt-stack-a.example.com'
+ assert_before $'or-stack\tdestroy' $'aws-elbv2\tdescribe-load-balancers'
+ assert_before $'mqtt-stack-a.example.com\tA' $'--hostname\tmqtt-stack-a.example.com\t--confirm\tstack-a'
+ assert_before $'--hostname\tmqtt-stack-a.example.com\t--confirm\tstack-a' $'--hostname\tstack-a.example.com\t--confirm\tstack-a'
+ assert_not_contains $'aws-acm\tdelete-certificate'
+
+ OR_EKS_STACK_TEST_REGISTRY_JSON="$registry" \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME=mqtt-stack-a.example.com \
+ OR_EKS_STACK_TEST_MQTTS_SERVICE_EXISTS=false \
+ run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a >/dev/null
+ assert_contains $'or-stack\tdestroy'
+ assert_contains $'aws-elbv2\tdescribe-load-balancers'
+ assert_contains $'--hostname\tmqtt-stack-a.example.com\t--confirm\tstack-a'
+
+ OR_EKS_STACK_TEST_REGISTRY_JSON="$registry" \
+ OR_EKS_STACK_TEST_NAMESPACE_EXISTS=false \
+ run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a >/dev/null
+ assert_not_contains $'or-stack\tdestroy'
+ assert_contains $'aws-elbv2\tdescribe-load-balancers'
+ assert_contains $'--hostname\tmqtt-stack-a.example.com\t--confirm\tstack-a'
+}
+
+test_ingress_mqtts_destroy_checks_retained_nlb() {
+ local registry failure_output state error_kind
+ local OR_EKS_STACK_TEST_REGISTRY_JSON
+ local OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=existing
+ local OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE=existing
+ local OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME=mqtt-stack-a.example.com
+ local OR_EKS_STACK_TEST_LOAD_BALANCERS_JSON="$DEFAULT_LOAD_BALANCERS_JSON"
+ local OR_EKS_STACK_TEST_NAMESPACE_EXISTS=true
+ local OR_EKS_STACK_TEST_MQTTS_SERVICE_EXISTS=true
+
+ registry="$(jq '.data["mqtt-stack-a.example.com"] =
+ ({namespace:"stack-a",namespaceUID:"uid-stack-a",target:"eks",dns:"external-dns",
+ exposure:"ingress",endpoint:"mqtts"}|tojson)' <<< "$DEFAULT_HOSTNAME_REGISTRY_JSON")"
+ OR_EKS_STACK_TEST_REGISTRY_JSON="$registry"
+
+ # AWS can retain an NLB after Kubernetes reports deletion. Each invocation
+ # starts with empty process state, including retries after namespace removal.
+ for state in initial namespace-absent service-absent; do
+ OR_EKS_STACK_TEST_NAMESPACE_EXISTS=true
+ OR_EKS_STACK_TEST_MQTTS_SERVICE_EXISTS=true
+ case "$state" in
+ namespace-absent) OR_EKS_STACK_TEST_NAMESPACE_EXISTS=false ;;
+ service-absent) OR_EKS_STACK_TEST_MQTTS_SERVICE_EXISTS=false ;;
+ esac
+ if failure_output="$(run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a --timeout 1s 2>&1)"; then
+ fail "Destroy released reservations with a retained MQTTS NLB ($state)"
+ fi
+ grep -F 'MQTTS NLB deletion' <<< "$failure_output" >/dev/null || fail "$failure_output"
+ assert_contains $'aws-elbv2\tdescribe-tags\t--resource-arns\t'"$TEST_MQTTS_NLB_ARN"
+ assert_not_contains $'or-stack\trelease-hostname'
+ assert_not_contains $'aws-acm\tdelete-certificate'
+ if [ "$state" = namespace-absent ]; then
+ assert_not_contains $'or-stack\tdestroy'
+ fi
+ done
+
+ OR_EKS_STACK_TEST_NAMESPACE_EXISTS=false
+ # A later retry succeeds only once the AWS resource is gone.
+ OR_EKS_STACK_TEST_LOAD_BALANCERS_JSON='{"LoadBalancers":[]}' \
+ run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a >/dev/null
+ assert_not_contains $'or-stack\tdestroy'
+ assert_before $'aws-elbv2\tdescribe-load-balancers' $'or-stack\trelease-hostname'
+ assert_contains $'--hostname\tmqtt-stack-a.example.com\t--confirm\tstack-a'
+ assert_contains $'--hostname\tstack-a.example.com\t--confirm\tstack-a'
+
+ # Neither another cluster's identically named Service nor another stack's
+ # NLB in this cluster should block releasing this stack's reservations.
+ OR_EKS_STACK_TEST_NLB_CLUSTER=another-cluster \
+ run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a >/dev/null
+ assert_contains $'or-stack\trelease-hostname'
+ OR_EKS_STACK_TEST_NLB_SERVICE=stack-b/proxy-mqtts \
+ run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a >/dev/null
+ assert_contains $'or-stack\trelease-hostname'
+
+ for error_kind in list tags malformed-list malformed-tags empty-list empty-tags; do
+ local OR_EKS_STACK_TEST_ELB_LIST_ERROR=""
+ local OR_EKS_STACK_TEST_ELB_TAGS_ERROR=""
+ local OR_EKS_STACK_TEST_ELB_TAGS_JSON=""
+ OR_EKS_STACK_TEST_LOAD_BALANCERS_JSON="$DEFAULT_LOAD_BALANCERS_JSON"
+ case "$error_kind" in
+ list) OR_EKS_STACK_TEST_ELB_LIST_ERROR='AccessDenied' ;;
+ tags) OR_EKS_STACK_TEST_ELB_TAGS_ERROR='AccessDenied' ;;
+ malformed-list) OR_EKS_STACK_TEST_LOAD_BALANCERS_JSON='{}' ;;
+ malformed-tags) OR_EKS_STACK_TEST_ELB_TAGS_JSON='{}' ;;
+ empty-list) OR_EKS_STACK_TEST_LOAD_BALANCERS_JSON=' ' ;;
+ empty-tags) OR_EKS_STACK_TEST_ELB_TAGS_JSON=' ' ;;
+ esac
+ if failure_output="$(run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a 2>&1)"; then
+ fail "Destroy released reservations after an AWS lookup failure ($error_kind)"
+ fi
+ assert_not_contains $'or-stack\trelease-hostname'
+ assert_not_contains $'aws-acm\tdelete-certificate'
+ done
+
+ # Disappearing between list and describe-tags is a confirmed deletion, not
+ # an authorization failure or a reason to keep a completed cleanup blocked.
+ OR_EKS_STACK_TEST_ELB_TAGS_ERROR='An error occurred (LoadBalancerNotFound)' \
+ run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a >/dev/null
+ assert_contains $'or-stack\trelease-hostname'
+}
+
+test_ingress_mqtts_managed_certificate_lifecycle() {
+ local failure_output
+ local output
+ local registry
+
+ output="$(OR_EKS_STACK_TEST_REQUESTED_CERTIFICATE_ARN="$TEST_MQTTS_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_JSON="$MANAGED_MQTTS_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_TAGS="$MANAGED_MQTTS_CERTIFICATE_TAGS" \
+ run_eks_stack apply \
+ --name stack-a \
+ --cluster test-cluster \
+ --hostname stack-a.example.com \
+ --profile '' \
+ --exposure ingress \
+ --certificate-mode existing \
+ --certificate-arn "$TEST_CERTIFICATE_ARN" \
+ --mqtts \
+ --mqtts-hostname mqtt-stack-a.example.com)"
+
+ grep -F -- 'selected stack-managed MQTTS certificate mode' <<< "$output" >/dev/null || fail \
+ "First MQTTS apply without an ARN did not select managed mode"
+ assert_contains $'aws-acm\trequest-certificate\t--domain-name\tmqtt-stack-a.example.com'
+ assert_contains $'Key=openremote.io/endpoint,Value=mqtts'
+ assert_contains $'openremote.io/mqtts-certificate-mode=managed\topenremote.io/mqtts-hostname=mqtt-stack-a.example.com\t--overwrite\t--resource-version=2'
+ assert_contains $'openremote.io/mqtts-certificate-mode=managed\topenremote.io/mqtts-certificate-arn='"$TEST_MQTTS_CERTIFICATE_ARN"
+ assert_contains $'--mqtts-certificate-arn\t'"$TEST_MQTTS_CERTIFICATE_ARN"
+ assert_before $'openremote.io/mqtts-certificate-mode=managed\topenremote.io/mqtts-hostname=' \
+ $'aws-acm\trequest-certificate'
+ assert_before $'aws-acm\trequest-certificate' \
+ $'openremote.io/mqtts-certificate-mode=managed\topenremote.io/mqtts-certificate-arn='
+ assert_before $'kubectl\t--context\ttest-cluster@eu-west-1\t--namespace\tstack-a\tapply\t-f\t-' \
+ $'or-stack\tapply'
+ jq -e --arg arn "$TEST_MQTTS_CERTIFICATE_ARN" '
+ .metadata.name == "acm-validation-mqtts" and
+ .metadata.annotations["openremote.io/certificate-arn"] == $arn and
+ .metadata.annotations["openremote.io/endpoint"] == "mqtts" and
+ .spec.endpoints[0].dnsName == "_validation.mqtt-stack-a.example.com."' \
+ "$DNS_ENDPOINT_LOG" >/dev/null || fail \
+ "Managed MQTTS apply did not create the expected validation DNSEndpoint"
+ jq -e --arg arn "$TEST_MQTTS_CERTIFICATE_ARN" '
+ .metadata.annotations["openremote.io/mqtts-certificate-mode"] == "managed" and
+ .metadata.annotations["openremote.io/mqtts-certificate-arn"] == $arn and
+ .metadata.annotations["openremote.io/mqtts-hostname"] == "mqtt-stack-a.example.com"' \
+ "$CERTIFICATE_NAMESPACE_FILE" >/dev/null || fail \
+ "Managed MQTTS apply did not persist its lifecycle state"
+
+ OR_EKS_STACK_TEST_SHARED_CERTIFICATE_ARN="$TEST_SECOND_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_REQUESTED_CERTIFICATE_ARN="$TEST_MQTTS_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_JSON="$MANAGED_MQTTS_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_TAGS="$MANAGED_MQTTS_CERTIFICATE_TAGS" \
+ run_eks_stack apply \
+ --name stack-a --cluster test-cluster --hostname stack-a.example.com \
+ --profile '' --exposure ingress \
+ --certificate-mode existing --certificate-arn "$TEST_CERTIFICATE_ARN" \
+ --mqtts --mqtts-hostname mqtt-stack-a.example.com \
+ --mqtts-certificate-mode managed >/dev/null
+ assert_contains $'aws-acm\trequest-certificate\t--domain-name\tmqtt-stack-a.example.com'
+ assert_contains $'openremote.io/mqtts-certificate-mode=managed'
+ assert_not_contains $'--mqtts-certificate-arn\t'"$TEST_SECOND_CERTIFICATE_ARN"
+
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE=managed \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_ARN='' \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME=mqtt-stack-a.example.com \
+ OR_EKS_STACK_TEST_CERTIFICATE_LIST="{\"CertificateSummaryList\":[{\"CertificateArn\":\"$TEST_MQTTS_CERTIFICATE_ARN\",\"DomainName\":\"mqtt-stack-a.example.com\"}]}" \
+ OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_JSON="$MANAGED_MQTTS_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_TAGS="$MANAGED_MQTTS_CERTIFICATE_TAGS" \
+ run_eks_stack apply \
+ --name stack-a --cluster test-cluster --hostname stack-a.example.com \
+ --profile '' --exposure ingress --mqtts >/dev/null
+ assert_contains $'aws-acm\tlist-certificates'
+ assert_not_contains $'aws-acm\trequest-certificate'
+ assert_contains $'openremote.io/mqtts-certificate-arn='"$TEST_MQTTS_CERTIFICATE_ARN"
+
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE=managed \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_ARN="$TEST_MQTTS_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME=mqtt-stack-a.example.com \
+ OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_JSON="$MANAGED_MQTTS_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_TAGS="$MANAGED_MQTTS_CERTIFICATE_TAGS" \
+ run_eks_stack apply \
+ --name stack-a --cluster test-cluster --hostname stack-a.example.com \
+ --profile '' --exposure ingress --mqtts >/dev/null
+ assert_not_contains $'aws-acm\trequest-certificate'
+ assert_contains $'--mqtts-certificate-arn\t'"$TEST_MQTTS_CERTIFICATE_ARN"
+
+ output="$(OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE=managed \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_ARN="$TEST_MQTTS_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME=mqtt-stack-a.example.com \
+ run_eks_stack status --name stack-a --cluster test-cluster --profile '')"
+ grep -F 'MQTTS certificate mode: managed' <<< "$output" >/dev/null || fail \
+ "Status did not report managed MQTTS certificate ownership"
+ grep -F 'MQTTS certificate status: ISSUED' <<< "$output" >/dev/null || fail \
+ "Status did not inspect the managed MQTTS certificate"
+
+ if failure_output="$(OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE=managed \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_ARN="$TEST_MQTTS_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME=mqtt-stack-a.example.com \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --profile '' --exposure ingress \
+ --mqtts --mqtts-certificate-mode existing 2>&1)"; then
+ fail "Managed MQTTS mode accepted in-place migration to existing"
+ fi
+ grep -F 'in-place certificate mode migration' <<< "$failure_output" >/dev/null || fail "$failure_output"
+
+ registry="$(jq -cn '{metadata:{resourceVersion:"1",labels:{"app.kubernetes.io/managed-by":"or-stack"}},
+ data:{
+ "stack-a.example.com":({namespace:"stack-a",namespaceUID:"uid-stack-a",target:"eks",dns:"external-dns",exposure:"ingress",endpoint:"web"}|tojson),
+ "mqtt-stack-a.example.com":({namespace:"stack-a",namespaceUID:"uid-stack-a",target:"eks",dns:"external-dns",exposure:"ingress",endpoint:"mqtts"}|tojson)
+ }}')"
+ OR_EKS_STACK_TEST_REGISTRY_JSON="$registry" \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE=managed \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_ARN="$TEST_MQTTS_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME=mqtt-stack-a.example.com \
+ OR_EKS_STACK_TEST_CERTIFICATE_LIST="{\"CertificateSummaryList\":[{\"CertificateArn\":\"$TEST_MQTTS_CERTIFICATE_ARN\",\"DomainName\":\"mqtt-stack-a.example.com\"}]}" \
+ OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_JSON="$MANAGED_MQTTS_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_TAGS="$MANAGED_MQTTS_CERTIFICATE_TAGS" \
+ OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_EXISTS=true \
+ run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a >/dev/null
+ [ -f "$MQTTS_CERTIFICATE_DELETED_FILE" ] || fail \
+ "Destroy did not delete the owned managed MQTTS certificate"
+ [ ! -f "$CERTIFICATE_DELETED_FILE" ] || fail \
+ "Destroy deleted the externally owned web certificate"
+ assert_contains $'get\tdnsendpoint\tacm-validation-mqtts'
+ assert_contains $'aws-acm\tdelete-certificate\t--certificate-arn\t'"$TEST_MQTTS_CERTIFICATE_ARN"
+ assert_before $'or-stack\tdestroy' $'aws-acm\tdelete-certificate'
+ assert_before $'_validation.mqtt-stack-a.example.com.\tCNAME' \
+ $'aws-acm\tdelete-certificate'
+ assert_before $'aws-acm\tdelete-certificate' \
+ $'--hostname\tmqtt-stack-a.example.com\t--confirm\tstack-a'
+
+ OR_EKS_STACK_TEST_REGISTRY_JSON="$registry" \
+ OR_EKS_STACK_TEST_NAMESPACE_EXISTS=false \
+ OR_EKS_STACK_TEST_CERTIFICATE_LIST="{\"CertificateSummaryList\":[{\"CertificateArn\":\"$TEST_MQTTS_CERTIFICATE_ARN\",\"DomainName\":\"mqtt-stack-a.example.com\"}]}" \
+ OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_JSON="$MANAGED_MQTTS_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_TAGS="$MANAGED_MQTTS_CERTIFICATE_TAGS" \
+ run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a >/dev/null
+ assert_not_contains $'or-stack\tdestroy'
+ assert_contains $'aws-acm\tlist-certificates'
+ assert_contains $'aws-acm\tdelete-certificate\t--certificate-arn\t'"$TEST_MQTTS_CERTIFICATE_ARN"
+ assert_contains $'--hostname\tmqtt-stack-a.example.com\t--confirm\tstack-a'
+}
+
+test_dual_managed_certificate_lifecycle() {
+ local request_count
+ local registry
+
+ OR_EKS_STACK_TEST_CERTIFICATE_JSON="$MANAGED_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_CERTIFICATE_TAGS="$MANAGED_CERTIFICATE_TAGS" \
+ OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_JSON="$MANAGED_MQTTS_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_TAGS="$MANAGED_MQTTS_CERTIFICATE_TAGS" \
+ run_eks_stack apply \
+ --name stack-a --cluster test-cluster --hostname stack-a.example.com \
+ --profile '' --exposure ingress --certificate-mode managed \
+ --mqtts --mqtts-hostname mqtt-stack-a.example.com >/dev/null
+
+ request_count="$(grep -c $'^aws-acm\trequest-certificate\t' "$CALL_LOG")"
+ [ "$request_count" -eq 2 ] || fail \
+ "Dual managed apply did not request exactly one certificate per endpoint"
+ assert_contains $'--domain-name\tstack-a.example.com'
+ assert_contains $'--domain-name\tmqtt-stack-a.example.com'
+ assert_contains $'Key=openremote.io/endpoint,Value=web'
+ assert_contains $'Key=openremote.io/endpoint,Value=mqtts'
+ assert_contains $'--certificate-arn\t'"$TEST_CERTIFICATE_ARN"$'\t--mqtts'
+ assert_contains $'--mqtts-certificate-arn\t'"$TEST_MQTTS_CERTIFICATE_ARN"
+
+ registry="$(jq -cn '{metadata:{resourceVersion:"1",labels:{"app.kubernetes.io/managed-by":"or-stack"}},
+ data:{
+ "stack-a.example.com":({namespace:"stack-a",namespaceUID:"uid-stack-a",target:"eks",dns:"external-dns",exposure:"ingress",endpoint:"web"}|tojson),
+ "mqtt-stack-a.example.com":({namespace:"stack-a",namespaceUID:"uid-stack-a",target:"eks",dns:"external-dns",exposure:"ingress",endpoint:"mqtts"}|tojson)
+ }}')"
+ OR_EKS_STACK_TEST_REGISTRY_JSON="$registry" \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=managed \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_HOSTNAME=stack-a.example.com \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE=managed \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_ARN="$TEST_MQTTS_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME=mqtt-stack-a.example.com \
+ OR_EKS_STACK_TEST_CERTIFICATE_LIST="{\"CertificateSummaryList\":[{\"CertificateArn\":\"$TEST_CERTIFICATE_ARN\",\"DomainName\":\"stack-a.example.com\"},{\"CertificateArn\":\"$TEST_MQTTS_CERTIFICATE_ARN\",\"DomainName\":\"mqtt-stack-a.example.com\"}]}" \
+ OR_EKS_STACK_TEST_CERTIFICATE_JSON="$MANAGED_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_CERTIFICATE_TAGS="$MANAGED_CERTIFICATE_TAGS" \
+ OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_JSON="$MANAGED_MQTTS_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_TAGS="$MANAGED_MQTTS_CERTIFICATE_TAGS" \
+ OR_EKS_STACK_TEST_DNS_ENDPOINT_EXISTS=true \
+ OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_EXISTS=true \
+ OR_EKS_STACK_TEST_INGRESS_EXISTS=true \
+ run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a >/dev/null
+
+ [ -f "$CERTIFICATE_DELETED_FILE" ] || fail \
+ "Dual managed destroy did not delete the web certificate"
+ [ -f "$MQTTS_CERTIFICATE_DELETED_FILE" ] || fail \
+ "Dual managed destroy did not delete the MQTTS certificate"
+ assert_contains $'get\tdnsendpoint\tacm-validation\t--ignore-not-found\t-o\tjson'
+ assert_contains $'get\tdnsendpoint\tacm-validation-mqtts\t--ignore-not-found\t-o\tjson'
+ assert_contains $'aws-acm\tdelete-certificate\t--certificate-arn\t'"$TEST_CERTIFICATE_ARN"
+ assert_contains $'aws-acm\tdelete-certificate\t--certificate-arn\t'"$TEST_MQTTS_CERTIFICATE_ARN"
+ assert_before $'delete\tingress\tproxy' $'or-stack\tdestroy'
+ assert_before $'or-stack\tdestroy' \
+ $'aws-acm\tdelete-certificate\t--certificate-arn\t'"$TEST_MQTTS_CERTIFICATE_ARN"
+}
+
+test_ingress_mqtts_shared_certificate_lifecycle() {
+ local failure_output
+ local output
+ local registry
+
+ output="$(OR_EKS_STACK_TEST_SHARED_CERTIFICATE_ARN="$TEST_MQTTS_CERTIFICATE_ARN" \
+ run_eks_stack apply \
+ --name stack-a \
+ --cluster test-cluster \
+ --hostname stack-a.example.com \
+ --profile '' \
+ --exposure ingress \
+ --certificate-mode existing \
+ --certificate-arn "$TEST_CERTIFICATE_ARN" \
+ --mqtts \
+ --mqtts-hostname mqtt-stack-a.example.com)"
+
+ grep -F -- "Selected the cluster's shared ACM certificate '$TEST_MQTTS_CERTIFICATE_ARN' for MQTTS." \
+ <<< "$output" >/dev/null || fail \
+ "First MQTTS apply did not automatically select the shared certificate"
+ grep -F -- "Validated externally managed shared ACM certificate '$TEST_MQTTS_CERTIFICATE_ARN' for MQTTS hostname 'mqtt-stack-a.example.com'." \
+ <<< "$output" >/dev/null || fail \
+ "Shared MQTTS certificate was not validated as externally managed"
+ assert_contains $'openremote.io/mqtts-certificate-mode=shared\topenremote.io/mqtts-hostname=mqtt-stack-a.example.com\t--overwrite\t--resource-version=2'
+ assert_contains $'openremote.io/mqtts-certificate-mode=shared\topenremote.io/mqtts-certificate-arn='"$TEST_MQTTS_CERTIFICATE_ARN"
+ assert_contains $'--mqtts-certificate-arn\t'"$TEST_MQTTS_CERTIFICATE_ARN"
+ assert_not_contains $'aws-acm\trequest-certificate'
+
+ OR_EKS_STACK_TEST_SHARED_CERTIFICATE_ARN="$TEST_MQTTS_CERTIFICATE_ARN" \
+ run_eks_stack apply \
+ --name stack-a --cluster test-cluster --hostname stack-a.example.com \
+ --profile '' --exposure ingress \
+ --certificate-mode existing --certificate-arn "$TEST_CERTIFICATE_ARN" \
+ --mqtts --mqtts-hostname mqtt-stack-a.example.com \
+ --mqtts-certificate-mode shared >/dev/null
+ assert_contains $'openremote.io/mqtts-certificate-mode=shared'
+ assert_contains $'--mqtts-certificate-arn\t'"$TEST_MQTTS_CERTIFICATE_ARN"
+ assert_not_contains $'aws-acm\trequest-certificate'
+
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE=shared \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_ARN="$TEST_MQTTS_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME=mqtt-stack-a.example.com \
+ OR_EKS_STACK_TEST_SHARED_CERTIFICATE_ARN="$TEST_SECOND_CERTIFICATE_ARN" \
+ run_eks_stack apply \
+ --name stack-a --cluster test-cluster --hostname stack-a.example.com \
+ --profile '' --exposure ingress --mqtts >/dev/null
+ assert_contains $'--mqtts-certificate-arn\t'"$TEST_SECOND_CERTIFICATE_ARN"
+ assert_contains $'openremote.io/mqtts-certificate-mode=shared\topenremote.io/mqtts-certificate-arn='"$TEST_SECOND_CERTIFICATE_ARN"
+ assert_not_contains $'aws-acm\trequest-certificate'
+
+ output="$(OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE=shared \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_ARN="$TEST_MQTTS_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME=mqtt-stack-a.example.com \
+ run_eks_stack status --name stack-a --cluster test-cluster --profile '')"
+ grep -F 'MQTTS certificate mode: shared' <<< "$output" >/dev/null || fail \
+ "Status did not report shared MQTTS certificate mode"
+ grep -F 'MQTTS certificate status: ISSUED' <<< "$output" >/dev/null || fail \
+ "Status did not inspect the shared MQTTS certificate"
+
+ if failure_output="$(OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE=shared \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_ARN="$TEST_MQTTS_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME=mqtt-stack-a.example.com \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --profile '' --exposure ingress \
+ --mqtts 2>&1)"; then
+ fail "Recorded shared MQTTS mode accepted a missing cluster certificate"
+ fi
+ grep -F -- 'MQTTS certificate mode shared requires a certificate configured' \
+ <<< "$failure_output" >/dev/null || fail "$failure_output"
+ assert_not_contains $'or-stack\tprepare'
+
+ registry="$(jq -cn '{metadata:{resourceVersion:"1",labels:{"app.kubernetes.io/managed-by":"or-stack"}},
+ data:{
+ "stack-a.example.com":({namespace:"stack-a",namespaceUID:"uid-stack-a",target:"eks",dns:"external-dns",exposure:"ingress",endpoint:"web"}|tojson),
+ "mqtt-stack-a.example.com":({namespace:"stack-a",namespaceUID:"uid-stack-a",target:"eks",dns:"external-dns",exposure:"ingress",endpoint:"mqtts"}|tojson)
+ }}')"
+ OR_EKS_STACK_TEST_REGISTRY_JSON="$registry" \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE=shared \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_ARN="$TEST_MQTTS_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME=mqtt-stack-a.example.com \
+ run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a >/dev/null
+ assert_contains $'or-stack\tdestroy'
+ assert_contains $'--hostname\tmqtt-stack-a.example.com\t--confirm\tstack-a'
+ assert_not_contains $'aws-acm\tdelete-certificate'
+}
+
+test_ingress_mqtts_validation() {
+ local failure_output
+
+ if failure_output="$(run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --profile '' --exposure ingress \
+ --certificate-mode existing --certificate-arn "$TEST_CERTIFICATE_ARN" \
+ --mqtts --mqtts-certificate-arn "$TEST_MQTTS_CERTIFICATE_ARN" 2>&1)"; then
+ fail "First Ingress MQTTS apply accepted a missing hostname"
+ fi
+ grep -F -- 'requires --mqtts-hostname on first apply' <<< "$failure_output" >/dev/null || fail "$failure_output"
+ assert_not_contains $'or-stack\tprepare'
+
+ if failure_output="$(run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --profile '' --exposure ingress \
+ --certificate-mode existing --certificate-arn "$TEST_CERTIFICATE_ARN" \
+ --mqtts --mqtts-hostname mqtt-stack-a.example.com \
+ --mqtts-certificate-mode existing 2>&1)"; then
+ fail "First Ingress MQTTS apply accepted a missing certificate ARN"
+ fi
+ grep -F -- 'certificate mode existing requires --mqtts-certificate-arn on first apply' \
+ <<< "$failure_output" >/dev/null || fail "$failure_output"
+ assert_not_contains $'or-stack\tprepare'
+
+ if (run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --profile '' --exposure ingress \
+ --certificate-mode existing --certificate-arn "$TEST_CERTIFICATE_ARN" \
+ --mqtts --mqtts-hostname stack-a.example.com \
+ --mqtts-certificate-arn "$TEST_MQTTS_CERTIFICATE_ARN") >/dev/null 2>&1; then
+ fail "Ingress MQTTS accepted the web hostname"
+ fi
+ assert_not_contains $'or-stack\tprepare'
+
+ if failure_output="$(OR_EKS_STACK_TEST_EXTERNAL_DNS_ARGS="${DEFAULT_EXTERNAL_DNS_ARGS/--source=service/}" \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --profile '' --exposure ingress \
+ --certificate-mode existing --certificate-arn "$TEST_CERTIFICATE_ARN" \
+ --mqtts --mqtts-hostname mqtt-stack-a.example.com \
+ --mqtts-certificate-arn "$TEST_MQTTS_CERTIFICATE_ARN" 2>&1)"; then
+ fail "Ingress MQTTS accepted ExternalDNS without its Service source"
+ fi
+ grep -F -- "missing required argument '--source=service'" <<< "$failure_output" >/dev/null || fail "$failure_output"
+ assert_not_contains $'or-stack\tprepare'
+
+ if failure_output="$(OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_JSON='{"Certificate":{"Status":"ISSUED","DomainName":"another.example.net","SubjectAlternativeNames":["another.example.net"]}}' \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --profile '' --exposure ingress \
+ --certificate-mode existing --certificate-arn "$TEST_CERTIFICATE_ARN" \
+ --mqtts --mqtts-hostname mqtt-stack-a.example.com \
+ --mqtts-certificate-arn "$TEST_MQTTS_CERTIFICATE_ARN" 2>&1)"; then
+ fail "Ingress MQTTS accepted a certificate for another hostname"
+ fi
+ grep -F -- "does not cover 'mqtt-stack-a.example.com'" <<< "$failure_output" >/dev/null || fail "$failure_output"
+ assert_not_contains $'or-stack\tprepare'
+
+ if failure_output="$(OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_TAGS='{"Tags":[{"Key":"openremote.io/managed-by","Value":"or-eks-stack"}]}' \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --profile '' --exposure ingress \
+ --certificate-mode existing --certificate-arn "$TEST_CERTIFICATE_ARN" \
+ --mqtts --mqtts-hostname mqtt-stack-a.example.com \
+ --mqtts-certificate-arn "$TEST_MQTTS_CERTIFICATE_ARN" 2>&1)"; then
+ fail "Ingress MQTTS accepted an or-eks-stack-managed certificate"
+ fi
+ grep -F -- 'cannot be used as an existing certificate' <<< "$failure_output" >/dev/null || fail "$failure_output"
+ assert_not_contains $'or-stack\tprepare'
+
+ if failure_output="$(OR_EKS_STACK_TEST_EXTERNAL_DNS_ARGS="${DEFAULT_EXTERNAL_DNS_ARGS/--source=crd/}" \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --profile '' --exposure ingress \
+ --certificate-mode existing --certificate-arn "$TEST_CERTIFICATE_ARN" \
+ --mqtts --mqtts-hostname mqtt-stack-a.example.com 2>&1)"; then
+ fail "Managed MQTTS mode accepted ExternalDNS without its CRD source"
+ fi
+ grep -F -- "missing required argument '--source=crd'" \
+ <<< "$failure_output" >/dev/null || fail "$failure_output"
+ assert_not_contains $'or-stack\tprepare'
+
+ if (run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --profile '' --exposure ingress \
+ --certificate-mode existing --certificate-arn "$TEST_CERTIFICATE_ARN" \
+ --mqtts --mqtts-hostname mqtt-stack-a.example.com \
+ --mqtts-certificate-mode managed \
+ --mqtts-certificate-arn "$TEST_MQTTS_CERTIFICATE_ARN") >/dev/null 2>&1; then
+ fail "Managed MQTTS mode accepted a caller-supplied ARN"
+ fi
+ assert_not_contains $'or-stack\tprepare'
+
+ if (run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --profile '' --exposure ingress \
+ --certificate-mode existing --certificate-arn "$TEST_CERTIFICATE_ARN" \
+ --mqtts --mqtts-hostname mqtt-stack-a.example.com \
+ --mqtts-certificate-mode shared \
+ --mqtts-certificate-arn "$TEST_MQTTS_CERTIFICATE_ARN") >/dev/null 2>&1; then
+ fail "Shared MQTTS mode accepted a caller-supplied ARN"
+ fi
+ assert_not_contains $'or-stack\tprepare'
+
+ if failure_output="$(OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_JSON="$MANAGED_MQTTS_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_MQTTS_CERTIFICATE_TAGS="$MANAGED_MQTTS_CERTIFICATE_TAGS_WITHOUT_ENDPOINT" \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --profile '' --exposure ingress \
+ --certificate-mode existing --certificate-arn "$TEST_CERTIFICATE_ARN" \
+ --mqtts --mqtts-hostname mqtt-stack-a.example.com 2>&1)"; then
+ fail "Managed MQTTS mode accepted certificate tags for the web endpoint"
+ fi
+ grep -F -- 'does not carry the exact managed MQTTS ownership tags' \
+ <<< "$failure_output" >/dev/null || fail "$failure_output"
+ assert_not_contains $'or-stack\tapply'
+}
+
+test_ingress_shared_certificate_apply() {
+ local output
+
+ output="$(OR_EKS_STACK_TEST_SHARED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ run_eks_stack apply \
+ --name stack-a \
+ --cluster test-cluster \
+ --hostname stack-a.example.com \
+ --profile '' \
+ --exposure ingress \
+ --certificate-mode shared)"
+
+ assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\tget\tingressclass\topenremote-alb\t-o\tgo-template={{with .metadata.annotations}}{{with index . "openremote.io/shared-certificate-arn"}}{{.}}{{end}}{{end}}'
+ assert_contains $'aws-acm\tdescribe-certificate\t--certificate-arn\t'"$TEST_CERTIFICATE_ARN"$'\t--output\tjson'
+ assert_contains $'or-stack\tapply\t--name\tstack-a\t--kube-context\ttest-cluster@eu-west-1\t--target\teks\t--exposure\tingress\t--dns\texternal-dns\t--hostname\tstack-a.example.com\t--timeout\t20m\t--certificate-arn\t'"$TEST_CERTIFICATE_ARN"
+ assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\tannotate\tnamespace\tstack-a\topenremote.io/certificate-mode=shared\topenremote.io/certificate-arn='"$TEST_CERTIFICATE_ARN"$'\topenremote.io/hostname=stack-a.example.com\t--overwrite'
+ assert_before $'get\tingressclass\topenremote-alb' $'aws-acm\tdescribe-certificate'
+ assert_before $'aws-acm\tdescribe-certificate' $'or-stack\tapply'
+ grep -F -- "Selected the cluster's shared ACM certificate '$TEST_CERTIFICATE_ARN'" \
+ <<< "$output" >/dev/null || fail "Apply did not report shared certificate selection"
+ grep -F -- "Validated externally managed shared ACM certificate '$TEST_CERTIFICATE_ARN'" \
+ <<< "$output" >/dev/null || fail "Apply did not report shared certificate validation"
+
+ OR_EKS_STACK_TEST_SHARED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ run_eks_stack apply \
+ --name stack-a \
+ --cluster test-cluster \
+ --hostname stack-a.example.com \
+ --profile '' \
+ --exposure ingress >/dev/null
+ assert_contains $'openremote.io/certificate-mode=shared'
+}
+
+test_ingress_managed_certificate_apply() {
+ local output
+
+ output="$(OR_EKS_STACK_TEST_CERTIFICATE_TAGS="$MANAGED_CERTIFICATE_TAGS" \
+ OR_EKS_STACK_TEST_CERTIFICATE_JSON="$MANAGED_CERTIFICATE_JSON" \
+ run_eks_stack apply \
+ --name stack-a \
+ --cluster test-cluster \
+ --hostname stack-a.example.com \
+ --profile '' \
+ --exposure ingress \
+ --certificate-mode managed)"
+
+ assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\tget\tcustomresourcedefinition/dnsendpoints.externaldns.k8s.io'
+ assert_contains $'or-stack\tprepare\t--name\tstack-a\t--kube-context\ttest-cluster@eu-west-1\t--target\teks\t--exposure\tingress\t--dns\texternal-dns'
+ assert_contains $'aws-acm\tlist-certificates\t--certificate-statuses\tPENDING_VALIDATION\tISSUED\tINACTIVE\tEXPIRED\tVALIDATION_TIMED_OUT\tREVOKED\tFAILED\t--output\tjson'
+ assert_contains $'aws-acm\trequest-certificate\t--domain-name\tstack-a.example.com\t--validation-method\tDNS\t--idempotency-token'
+ assert_contains 'Key=openremote.io/managed-by,Value=or-eks-stack'
+ assert_contains 'Key=openremote.io/cluster,Value=test-cluster'
+ assert_contains 'Key=openremote.io/stack,Value=stack-a'
+ assert_contains 'Key=openremote.io/hostname,Value=stack-a.example.com'
+ assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\tannotate\tnamespace\tstack-a\topenremote.io/certificate-mode=managed\topenremote.io/certificate-arn='"$TEST_CERTIFICATE_ARN"
+ assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\t--namespace\tstack-a\tapply\t-f\t-'
+ assert_contains $'or-stack\tapply\t--name\tstack-a\t--kube-context\ttest-cluster@eu-west-1\t--target\teks\t--exposure\tingress\t--dns\texternal-dns\t--hostname\tstack-a.example.com\t--timeout\t20m\t--certificate-arn\t'"$TEST_CERTIFICATE_ARN"
+ assert_before $'or-stack\tprepare' $'aws-acm\trequest-certificate'
+ assert_before $'--resource-version=1' $'aws-acm\trequest-certificate'
+ assert_before $'aws-acm\trequest-certificate' $'openremote.io/certificate-mode=managed\topenremote.io/certificate-arn='
+ assert_before $'annotate\tnamespace\tstack-a' $'--namespace\tstack-a\tapply\t-f\t-'
+ assert_before $'--namespace\tstack-a\tapply\t-f\t-' $'or-stack\tapply'
+
+ jq -e \
+ --arg certificate_arn "$TEST_CERTIFICATE_ARN" \
+ '.apiVersion == "externaldns.k8s.io/v1alpha1" and
+ .kind == "DNSEndpoint" and
+ .metadata.name == "acm-validation" and
+ .metadata.namespace == "stack-a" and
+ .metadata.labels["openremote.io/managed-dns"] == "true" and
+ .metadata.labels["app.kubernetes.io/managed-by"] == "or-eks-stack" and
+ .metadata.annotations["openremote.io/certificate-arn"] == $certificate_arn and
+ .spec.endpoints == [{
+ "dnsName": "_validation.stack-a.example.com.",
+ "recordType": "CNAME",
+ "recordTTL": 300,
+ "targets": ["_target.acm-validations.aws."]
+ }]' "$DNS_ENDPOINT_LOG" >/dev/null || fail \
+ "Managed apply did not render the expected ACM validation DNSEndpoint"
+ grep -F -- "Managed ACM certificate '$TEST_CERTIFICATE_ARN' is issued." \
+ <<< "$output" >/dev/null || fail \
+ "Managed apply did not report certificate issuance"
+
+ output="$(OR_EKS_STACK_TEST_CERTIFICATE_TAGS="$MANAGED_CERTIFICATE_TAGS" \
+ OR_EKS_STACK_TEST_CERTIFICATE_JSON="$MANAGED_CERTIFICATE_JSON" \
+ run_eks_stack apply \
+ --name stack-a \
+ --cluster test-cluster \
+ --hostname stack-a.example.com \
+ --profile '' \
+ --exposure ingress)"
+ grep -F -- 'selected stack-managed certificate mode' <<< "$output" >/dev/null || fail \
+ "Ingress without a shared certificate did not select managed mode"
+ assert_contains $'openremote.io/certificate-mode=managed'
+}
+
+test_managed_certificate_reuse_and_safety() {
+ local failure_output
+ local failure_status
+
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=managed \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_CERTIFICATE_TAGS="$MANAGED_CERTIFICATE_TAGS" \
+ OR_EKS_STACK_TEST_CERTIFICATE_JSON="$MANAGED_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_DNS_ENDPOINT_EXISTS=true \
+ run_eks_stack apply \
+ --name stack-a \
+ --cluster test-cluster \
+ --hostname stack-a.example.com \
+ --profile '' \
+ --exposure ingress \
+ --certificate-mode managed >/dev/null
+ assert_not_contains $'aws-acm\tlist-certificates'
+ assert_not_contains $'aws-acm\trequest-certificate'
+ assert_contains $'or-stack\tapply'
+
+ OR_EKS_STACK_TEST_CERTIFICATE_LIST="{\"CertificateSummaryList\":[{\"CertificateArn\":\"$TEST_CERTIFICATE_ARN\",\"DomainName\":\"stack-a.example.com\"}]}" \
+ OR_EKS_STACK_TEST_CERTIFICATE_TAGS="$MANAGED_CERTIFICATE_TAGS" \
+ OR_EKS_STACK_TEST_CERTIFICATE_JSON="$MANAGED_CERTIFICATE_JSON" \
+ run_eks_stack apply \
+ --name stack-a \
+ --cluster test-cluster \
+ --hostname stack-a.example.com \
+ --profile '' \
+ --exposure ingress \
+ --certificate-mode managed >/dev/null
+ assert_contains $'aws-acm\tlist-certificates'
+ assert_not_contains $'aws-acm\trequest-certificate'
+ assert_contains $'or-stack\tapply'
+
+ if (OR_EKS_STACK_TEST_CERTIFICATE_LIST="{\"CertificateSummaryList\":[{\"CertificateArn\":\"$TEST_CERTIFICATE_ARN\",\"DomainName\":\"stack-a.example.com\"},{\"CertificateArn\":\"$TEST_SECOND_CERTIFICATE_ARN\",\"DomainName\":\"stack-a.example.com\"}]}" \
+ OR_EKS_STACK_TEST_CERTIFICATE_TAGS="$MANAGED_CERTIFICATE_TAGS" \
+ run_eks_stack apply \
+ --name stack-a \
+ --cluster test-cluster \
+ --hostname stack-a.example.com \
+ --profile '' \
+ --exposure ingress \
+ --certificate-mode managed) >/dev/null 2>&1; then
+ fail "Managed apply selected one of multiple owned certificates"
+ fi
+ assert_not_contains $'aws-acm\trequest-certificate'
+ assert_not_contains $'or-stack\tapply\t--name'
+
+ if (OR_EKS_STACK_TEST_CERTIFICATE_TAGS="$MANAGED_CERTIFICATE_TAGS" \
+ OR_EKS_STACK_TEST_CERTIFICATE_JSON="$MANAGED_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_DNS_ENDPOINT_EXISTS=true \
+ OR_EKS_STACK_TEST_DNS_ENDPOINT_JSON='{"metadata":{"labels":{"app.kubernetes.io/managed-by":"someone-else"},"annotations":{}}}' \
+ run_eks_stack apply \
+ --name stack-a \
+ --cluster test-cluster \
+ --hostname stack-a.example.com \
+ --profile '' \
+ --exposure ingress \
+ --certificate-mode managed) >/dev/null 2>&1; then
+ fail "Managed apply replaced an unowned ACM validation DNSEndpoint"
+ fi
+ assert_not_contains $'or-stack\tapply\t--name'
+
+ set +e
+ failure_output="$(OR_EKS_STACK_TEST_CERTIFICATE_TAGS="$MANAGED_CERTIFICATE_TAGS" \
+ OR_EKS_STACK_TEST_CERTIFICATE_JSON="${MANAGED_CERTIFICATE_JSON/PENDING_VALIDATION/FAILED}" \
+ run_eks_stack apply \
+ --name stack-a \
+ --cluster test-cluster \
+ --hostname stack-a.example.com \
+ --profile '' \
+ --exposure ingress \
+ --certificate-mode managed 2>&1)"
+ failure_status=$?
+ set -e
+ [ "$failure_status" -ne 0 ] || fail \
+ "Managed apply accepted a certificate in a terminal failure state"
+ grep -F -- "is 'FAILED' and cannot be reconciled automatically" \
+ <<< "$failure_output" >/dev/null || fail \
+ "Managed apply did not explain its terminal ACM failure"
+ assert_not_contains $'--namespace\tstack-a\tapply\t-f\t-'
+ assert_not_contains $'or-stack\tapply\t--name'
+
+ set +e
+ failure_output="$(OR_EKS_STACK_TEST_CERTIFICATE_TAGS="$MANAGED_CERTIFICATE_TAGS" \
+ OR_EKS_STACK_TEST_CERTIFICATE_JSON="${MANAGED_CERTIFICATE_JSON/_validation.stack-a.example.com./_validation.unrelated.test./}" \
+ run_eks_stack apply \
+ --name stack-a \
+ --cluster test-cluster \
+ --hostname stack-a.example.com \
+ --profile '' \
+ --exposure ingress \
+ --certificate-mode managed 2>&1)"
+ failure_status=$?
+ set -e
+ [ "$failure_status" -ne 0 ] || fail \
+ "Managed apply accepted an ACM validation record outside its DNS domain"
+ grep -F -- "is outside the configured ExternalDNS domains" \
+ <<< "$failure_output" >/dev/null || fail \
+ "Managed apply did not explain the out-of-scope validation record"
+ assert_not_contains $'--namespace\tstack-a\tapply\t-f\t-'
+ assert_not_contains $'or-stack\tapply\t--name'
+}
+
+create_proxy_certificate() {
+ mkdir -p "$PROXY_LE_DIR/live/stack-a.example.com"
+ printf 'persisted key\n' > "$PROXY_LE_DIR/live/stack-a.example.com/privkey.pem"
+ printf 'persisted chain\n' > "$PROXY_LE_DIR/live/stack-a.example.com/fullchain.pem"
+}
+
+test_validation_dns_endpoint_read_safety() {
+ local endpoint scenario apply_records endpoint_name failure_output read_error
+ local OR_EKS_STACK_TEST_DNS_ENDPOINT_EXISTS
+ local OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_EXISTS
+ local OR_EKS_STACK_TEST_DNS_ENDPOINT_JSON
+ local OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_JSON
+ local OR_EKS_STACK_TEST_DNS_ENDPOINT_READ_ERROR
+ local OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_READ_ERROR
+
+ for endpoint in web mqtts; do
+ apply_records=apply_managed_certificate_validation_records
+ endpoint_name=acm-validation
+ if [ "$endpoint" = mqtts ]; then
+ apply_records=apply_managed_mqtts_certificate_validation_records
+ endpoint_name=acm-validation-mqtts
+ fi
+ for scenario in absent owned unowned forbidden unavailable; do
+ reset_test
+ STACK_NAME=stack-a
+ KUBE_CONTEXT=test-cluster@eu-west-1
+ CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN"
+ MQTTS_CERTIFICATE_ARN="$TEST_MQTTS_CERTIFICATE_ARN"
+ CERTIFICATE_JSON="$MANAGED_CERTIFICATE_JSON"
+ MQTTS_CERTIFICATE_JSON="$MANAGED_MQTTS_CERTIFICATE_JSON"
+ EXTERNAL_DNS_ARGS="$DEFAULT_EXTERNAL_DNS_ARGS"
+ OR_EKS_STACK_TEST_DNS_ENDPOINT_EXISTS=true
+ OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_EXISTS=true
+ OR_EKS_STACK_TEST_DNS_ENDPOINT_JSON="$DEFAULT_DNS_ENDPOINT_JSON"
+ OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_JSON="$DEFAULT_MQTTS_DNS_ENDPOINT_JSON"
+ read_error=""
+ case "$scenario" in
+ absent)
+ OR_EKS_STACK_TEST_DNS_ENDPOINT_EXISTS=false
+ OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_EXISTS=false ;;
+ unowned)
+ OR_EKS_STACK_TEST_DNS_ENDPOINT_JSON='{"metadata":{"labels":{"app.kubernetes.io/managed-by":"someone-else"}}}'
+ OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_JSON="$OR_EKS_STACK_TEST_DNS_ENDPOINT_JSON" ;;
+ forbidden) read_error='Error from server (Forbidden): cannot get DNSEndpoint' ;;
+ unavailable) read_error='Unable to connect to the server: connection refused' ;;
+ esac
+ OR_EKS_STACK_TEST_DNS_ENDPOINT_READ_ERROR="$read_error"
+ OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_READ_ERROR="$read_error"
+ case "$scenario" in
+ absent|owned)
+ "$apply_records" >/dev/null
+ assert_contains $'apply\t-f\t-'
+ [ "$(jq -r '.metadata.name' "$DNS_ENDPOINT_LOG")" = "$endpoint_name" ] || fail \
+ "Did not write the expected $endpoint validation resource"
+ ;;
+ *)
+ if failure_output="$("$apply_records" 2>&1)"; then
+ fail "Validation records were applied after $scenario lookup for $endpoint"
+ fi
+ assert_not_contains $'apply\t-f\t-'
+ if [ -n "$read_error" ]; then
+ grep -F "$read_error" <<< "$failure_output" >/dev/null || fail "$failure_output"
+ grep -F 'without ownership checks' <<< "$failure_output" >/dev/null || fail "$failure_output"
+ else
+ grep -F 'not owned' <<< "$failure_output" >/dev/null || fail "$failure_output"
+ fi
+ ;;
+ esac
+ assert_contains $'get\tdnsendpoint\t'"$endpoint_name"$'\t--ignore-not-found\t-o\tjson'
+ done
+ done
+}
+
+test_apply_reuses_valid_certificate() {
+ OR_EKS_STACK_TEST_PROXY_CERTIFICATE_STATE=present \
+ OR_EKS_STACK_TEST_HTTPS_READY_IMMEDIATELY=true \
+ run_eks_stack apply \
+ --name stack-a \
+ --cluster test-cluster \
+ --hostname stack-a.example.com \
+ --profile '' >/dev/null
+
+ assert_not_contains $'/entrypoint.sh\tadd'
+ assert_not_contains $'/entrypoint.sh\trenew'
+ [ "$(< "$CURL_COUNT_FILE")" -eq 1 ] || fail \
+ "Apply did not stop after verifying the retained valid certificate"
+}
+
+test_acme_state_and_readiness() {
+ local failure_output
+
+ # A transient Manager/network failure with persisted state must only retry
+ # readiness; it must never issue or force-renew a certificate.
+ OR_EKS_STACK_TEST_PROXY_CERTIFICATE_STATE=present \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com >/dev/null
+ assert_not_contains $'/entrypoint.sh\tadd'
+ assert_not_contains $'/entrypoint.sh\trenew'
+ [ "$(< "$CURL_COUNT_FILE")" -eq 2 ] || fail "Apply did not retry HTTPS readiness"
+
+ if failure_output="$(OR_EKS_STACK_TEST_PROXY_CERTIFICATE_STATE=present \
+ OR_EKS_STACK_TEST_HTTPS_ALWAYS_FAILS=true \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --timeout 1s 2>&1)"; then
+ fail "Apply accepted an unavailable HTTPS endpoint"
+ fi
+ grep -F 'Trusted HTTPS did not become ready' <<< "$failure_output" >/dev/null || fail \
+ "Apply did not identify a readiness timeout"
+ assert_not_contains $'/entrypoint.sh\tadd'
+ assert_not_contains $'/entrypoint.sh\trenew'
+
+ if failure_output="$(OR_EKS_STACK_TEST_PROXY_CERTIFICATE_STATE=incomplete \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com 2>&1)"; then
+ fail "Apply accepted incomplete persisted certificate state"
+ fi
+ grep -F 'is incomplete' <<< "$failure_output" >/dev/null || fail "Missing incomplete-state diagnostic"
+ assert_not_contains $'/entrypoint.sh\tadd'
+
+ if failure_output="$(OR_EKS_STACK_TEST_CERTIFICATE_INSPECTION_SUCCESS=false \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com 2>&1)"; then
+ fail "Apply treated a failed inspection as absent certificate state"
+ fi
+ grep -F 'Unable to inspect HAProxy certificate state' <<< "$failure_output" >/dev/null || fail \
+ "Missing certificate inspection failure diagnostic"
+ assert_not_contains $'/entrypoint.sh\tadd'
+
+ # Startup can create the lineage between inspection and add. A failed add is
+ # recoverable only with persisted files, and still requires trusted HTTPS.
+ OR_EKS_STACK_TEST_STARTUP_ISSUES_CERTIFICATE=true OR_EKS_STACK_TEST_ACME_SUCCESS=false \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com >/dev/null
+ assert_contains $'/entrypoint.sh\tadd'
+ assert_contains $'curl\t'
+
+ if failure_output="$(OR_EKS_STACK_TEST_ACME_SUCCESS=false \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com 2>&1)"; then
+ fail "Apply ignored failed issuance without persisted certificate state"
+ fi
+ grep -F 'HAProxy ACME issuance failed (certificate state: absent)' <<< "$failure_output" >/dev/null || fail \
+ "Missing issuance failure diagnostic"
+ assert_not_contains $'curl\t'
+
+ if failure_output="$(OR_EKS_STACK_TEST_HTTPS_READY_IMMEDIATELY=true \
+ OR_EKS_STACK_TEST_MQTTS_ALWAYS_FAILS=true \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --mqtts --timeout 1s 2>&1)"; then
+ fail "Apply accepted an unavailable MQTTS endpoint"
+ fi
+ grep -F 'Trusted MQTTS did not become ready' <<< "$failure_output" >/dev/null || fail \
+ "Apply did not identify an MQTTS readiness timeout"
+ assert_contains $'python3\t-\tstack-a.example.com\t8883\t1'
+}
+
+test_lifecycle_delegation() {
+ local status_output
+
+ status_output="$(OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ run_eks_stack status --name stack-a --cluster test-cluster --profile '')"
+ assert_contains $'or-stack\tstatus\t--name\tstack-a\t--kube-context\ttest-cluster@eu-west-1'
+ assert_contains $'aws-acm\tdescribe-certificate\t--certificate-arn\t'"$TEST_CERTIFICATE_ARN"$'\t--query\tCertificate.Status\t--output\ttext'
+ grep -Fx -- 'Certificate mode: existing' <<< "$status_output" >/dev/null || fail \
+ "Status did not report the recorded certificate mode"
+ grep -Fx -- "Certificate ARN: $TEST_CERTIFICATE_ARN" <<< "$status_output" >/dev/null || fail \
+ "Status did not report the recorded certificate ARN"
+ grep -Fx -- 'Certificate status: ISSUED' <<< "$status_output" >/dev/null || fail \
+ "Status did not report the current ACM certificate status"
+
+ status_output="$(OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=shared \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ run_eks_stack status --name stack-a --cluster test-cluster --profile '')"
+ grep -Fx -- 'Certificate mode: shared' <<< "$status_output" >/dev/null || fail \
+ "Status did not report shared certificate ownership"
+ grep -Fx -- 'Certificate status: ISSUED' <<< "$status_output" >/dev/null || fail \
+ "Status did not report the shared ACM certificate status"
+
+ status_output="$(OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=managed \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ run_eks_stack status --name stack-a --cluster test-cluster --profile '')"
+ grep -Fx -- 'Certificate mode: managed' <<< "$status_output" >/dev/null || fail \
+ "Status did not report managed certificate ownership"
+ grep -Fx -- 'Certificate status: ISSUED' <<< "$status_output" >/dev/null || fail \
+ "Status did not report the managed ACM certificate status"
+
+ run_eks_stack credentials --name stack-a --cluster test-cluster --profile '' >/dev/null
+ assert_contains $'or-stack\tcredentials\t--name\tstack-a\t--kube-context\ttest-cluster@eu-west-1'
+
+ run_eks_stack uninstall --name stack-a --cluster test-cluster \
+ --profile '' --timeout 15m >/dev/null
+ assert_contains $'or-stack\tuninstall\t--name\tstack-a\t--kube-context\ttest-cluster@eu-west-1\t--timeout\t15m'
+ assert_not_contains 'delete-certificate'
+
+ run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a --timeout 15m >/dev/null
+ assert_contains $'or-stack\tdestroy\t--name\tstack-a\t--kube-context\ttest-cluster@eu-west-1\t--confirm\tstack-a\t--timeout\t15m'
+ assert_not_contains 'delete-certificate'
+
+}
+
+test_managed_destroy() {
+ local managed_certificate_list
+ local output
+ local status
+
+ managed_certificate_list="{\"CertificateSummaryList\":[{\"CertificateArn\":\"$TEST_CERTIFICATE_ARN\",\"DomainName\":\"stack-a.example.com\"}]}"
+ set +e
+ output="$(OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=managed \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_CERTIFICATE_LIST="$managed_certificate_list" \
+ OR_EKS_STACK_TEST_CERTIFICATE_TAGS="$MANAGED_CERTIFICATE_TAGS" \
+ OR_EKS_STACK_TEST_CERTIFICATE_JSON="$MANAGED_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_DNS_ENDPOINT_EXISTS=true \
+ OR_EKS_STACK_TEST_INGRESS_EXISTS=true \
+ run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a --timeout 15m 2>&1)"
+ status=$?
+ set -e
+ [ "$status" -eq 0 ] || fail \
+ "Managed destroy failed unexpectedly: $output"
+
+ assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\tget\tnamespace\tstack-a\t--ignore-not-found'
+ assert_contains $'aws-acm\tlist-certificates'
+ assert_contains $'kubectl\t--context\ttest-cluster@eu-west-1\t--namespace\tstack-a\tdelete\tingress\tproxy\t--wait=true\t--timeout\t15m'
+ assert_contains $'dig\t+time=5\t+tries=2\t+short\texample.com\tNS'
+ assert_contains $'dig\t+time=3\t+tries=1\t+norecurse\t+noall\t+comments\t+answer\t+authority\[email protected].\tstack-a.example.com\tA'
+ assert_contains $'or-stack\tdestroy\t--name\tstack-a\t--kube-context\ttest-cluster@eu-west-1\t--confirm\tstack-a\t--timeout\t15m'
+ assert_contains $'dig\t+time=3\t+tries=1\t+norecurse\t+noall\t+comments\t+answer\t+authority\[email protected].\t_validation.stack-a.example.com.\tCNAME'
+ assert_contains $'aws-acm\tdelete-certificate\t--certificate-arn\t'"$TEST_CERTIFICATE_ARN"
+ assert_before $'delete\tingress\tproxy' $'stack-a.example.com\tA'
+ assert_before $'stack-a.example.com\tA' $'or-stack\tdestroy'
+ assert_before $'or-stack\tdestroy' $'_validation.stack-a.example.com.\tCNAME'
+ assert_before $'_validation.stack-a.example.com.\tCNAME' $'aws-acm\tdelete-certificate'
+ grep -F -- "Stack 'stack-a' and its managed external certificate resources are destroyed." \
+ <<< "$output" >/dev/null || fail \
+ "Managed destroy did not report complete external cleanup"
+
+ set +e
+ output="$(OR_EKS_STACK_TEST_NAMESPACE_EXISTS=false \
+ OR_EKS_STACK_TEST_CERTIFICATE_LIST="$managed_certificate_list" \
+ OR_EKS_STACK_TEST_CERTIFICATE_TAGS="$MANAGED_CERTIFICATE_TAGS" \
+ OR_EKS_STACK_TEST_CERTIFICATE_JSON="$MANAGED_CERTIFICATE_JSON" \
+ run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a --timeout 15m 2>&1)"
+ status=$?
+ set -e
+ [ "$status" -eq 0 ] || fail \
+ "Managed destroy recovery failed unexpectedly: $output"
+ assert_not_contains $'or-stack\tdestroy'
+ assert_contains $'aws-acm\tdelete-certificate'
+}
+
+test_managed_destroy_safety() {
+ local managed_certificate_list
+
+ managed_certificate_list="{\"CertificateSummaryList\":[{\"CertificateArn\":\"$TEST_CERTIFICATE_ARN\",\"DomainName\":\"stack-a.example.com\"}]}"
+
+ if (OR_EKS_STACK_TEST_NAMESPACE_STACK_LABEL=someone-else \
+ run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a --timeout 15m) >/dev/null 2>&1; then
+ fail "Managed destroy accepted a namespace without matching stack ownership"
+ fi
+ assert_not_contains $'aws-acm\tlist-certificates'
+ assert_not_contains $'delete\tingress\tproxy'
+ assert_not_contains $'or-stack\tdestroy'
+ assert_not_contains $'delete-certificate'
+
+ if (OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=managed \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a --timeout 15m) >/dev/null 2>&1; then
+ fail "Managed destroy accepted a certificate without exact ownership tags"
+ fi
+ assert_not_contains $'delete\tingress\tproxy'
+ assert_not_contains $'or-stack\tdestroy'
+ assert_not_contains $'delete-certificate'
+
+ if (OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=managed \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_CERTIFICATE_LIST="$managed_certificate_list" \
+ OR_EKS_STACK_TEST_CERTIFICATE_TAGS="$MANAGED_CERTIFICATE_TAGS" \
+ OR_EKS_STACK_TEST_CERTIFICATE_JSON="$MANAGED_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_DNS_ENDPOINT_EXISTS=true \
+ OR_EKS_STACK_TEST_DNS_ENDPOINT_JSON='{"metadata":{"labels":{"app.kubernetes.io/managed-by":"someone-else","openremote.io/managed-dns":"true"},"annotations":{"openremote.io/certificate-arn":"arn:aws:acm:eu-west-1:123456789012:certificate/test-certificate"}},"spec":{"endpoints":[]}}' \
+ run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a --timeout 15m) >/dev/null 2>&1; then
+ fail "Managed destroy accepted an unowned validation DNSEndpoint"
+ fi
+ assert_not_contains $'delete\tingress\tproxy'
+ assert_not_contains $'or-stack\tdestroy'
+ assert_not_contains $'delete-certificate'
+
+ if (OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=managed \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_CERTIFICATE_LIST="$managed_certificate_list" \
+ OR_EKS_STACK_TEST_CERTIFICATE_TAGS="$MANAGED_CERTIFICATE_TAGS" \
+ OR_EKS_STACK_TEST_CERTIFICATE_JSON="$MANAGED_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_DNS_ENDPOINT_EXISTS=true \
+ OR_EKS_STACK_TEST_INGRESS_EXISTS=true \
+ OR_EKS_STACK_TEST_INGRESS_JSON='{"metadata":{"labels":{"openremote.io/managed-dns":"true"},"annotations":{"external-dns.kubernetes.io/hostname":"stack-a.example.com","alb.ingress.kubernetes.io/certificate-arn":"arn:aws:acm:eu-west-1:123456789012:certificate/someone-elses-certificate"}},"spec":{"rules":[{"host":"stack-a.example.com"}]}}' \
+ run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a --timeout 15m) >/dev/null 2>&1; then
+ fail "Managed destroy accepted an Ingress using another certificate"
+ fi
+ assert_not_contains $'delete\tingress\tproxy'
+ assert_not_contains $'or-stack\tdestroy'
+ assert_not_contains $'delete-certificate'
+
+ if (OR_EKS_STACK_TEST_CERTIFICATE_LIST="{\"CertificateSummaryList\":[{\"CertificateArn\":\"$TEST_CERTIFICATE_ARN\"},{\"CertificateArn\":\"$TEST_SECOND_CERTIFICATE_ARN\"}]}" \
+ OR_EKS_STACK_TEST_CERTIFICATE_TAGS="$MANAGED_CERTIFICATE_TAGS" \
+ run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a --timeout 15m) >/dev/null 2>&1; then
+ fail "Managed destroy selected one of multiple tagged certificates"
+ fi
+ assert_not_contains $'delete\tingress\tproxy'
+ assert_not_contains $'or-stack\tdestroy'
+ assert_not_contains $'delete-certificate'
+}
+
+test_managed_domain_boundary() {
+ local exposure hostname failure_output
+ local dns_args
+
+ # Reject the domain itself, including normalized spelling, before namespace
+ # reservation, workload installation, or managed certificate issuance.
+ for exposure in haproxy ingress; do
+ for hostname in example.com EXAMPLE.COM.; do
+ if failure_output="$(run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname "$hostname" --exposure "$exposure" 2>&1)"; then
+ fail "Apply accepted managed domain '$hostname' for $exposure"
+ fi
+ grep -F 'must be a strict subdomain' <<< "$failure_output" >/dev/null || fail \
+ "Apply did not explain the managed-domain restriction: $failure_output"
+ assert_not_contains $'or-stack\tprepare'
+ assert_not_contains $'or-stack\tapply'
+ assert_not_contains $'aws-acm\trequest-certificate'
+ done
+ done
+
+ # The boundary is the configured managed suffix, not necessarily a zone apex.
+ dns_args="${DEFAULT_EXTERNAL_DNS_ARGS/--domain-filter=example.com/--domain-filter=stack-a.example.com.}"
+ if failure_output="$(OR_EKS_STACK_TEST_EXTERNAL_DNS_ARGS="$dns_args" \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com 2>&1)"; then
+ fail "Apply accepted a hostname equal to a managed subdomain"
+ fi
+ grep -F 'must be a strict subdomain' <<< "$failure_output" >/dev/null || fail \
+ "The trailing-dot filter bypassed the managed-domain check"
+ assert_not_contains $'or-stack\tprepare'
+
+ # Exercise the DNS preflight without certificate coverage or endpoint stubs
+ # restricting the number of hostname labels accepted by the DNS boundary.
+ for hostname in test.example.com staging.test.example.com; do
+ reset_test
+ HOSTNAME_VALUE="$hostname"
+ require_managed_external_dns
+ require_stack_hostname_subdomain
+ done
+ reset_test
+ HOSTNAME_VALUE=staging.stack-a.example.com
+ OR_EKS_STACK_TEST_EXTERNAL_DNS_ARGS="$dns_args" require_managed_external_dns
+ require_stack_hostname_subdomain
+}
+
+test_validation() {
+ if (run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress \
+ --certificate-mode existing) >/dev/null 2>&1; then
+ fail "Existing certificate mode accepted a missing ARN"
+ fi
+
+ if (run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress \
+ --certificate-mode managed --certificate-arn "$TEST_CERTIFICATE_ARN") \
+ >/dev/null 2>&1; then
+ fail "Managed certificate mode accepted a caller-supplied ARN"
+ fi
+
+ if (OR_EKS_STACK_TEST_SHARED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress \
+ --certificate-mode shared --certificate-arn "$TEST_CERTIFICATE_ARN") \
+ >/dev/null 2>&1; then
+ fail "Shared certificate mode accepted a per-stack certificate ARN"
+ fi
+
+ if (run_eks_stack apply --name stack-a --cluster test-cluster) >/dev/null 2>&1; then
+ fail "Apply accepted a missing hostname"
+ fi
+
+ if (run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --timeout 0) >/dev/null 2>&1; then
+ fail "Apply accepted a zero timeout"
+ fi
+
+ if (run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --http-port 8080) >/dev/null 2>&1; then
+ fail "The ACME workflow accepted a non-standard public HTTP port"
+ fi
+
+ if (run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --mqtts-port 18883) >/dev/null 2>&1; then
+ fail "A custom MQTTS port was accepted without --mqtts"
+ fi
+
+ if (run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress --mqtts) >/dev/null 2>&1; then
+ fail "Ingress exposure accepted MQTTS"
+ fi
+
+ if (run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --mqtts --mqtts-port 443) >/dev/null 2>&1; then
+ fail "HAProxy exposure accepted an MQTTS port that collides with HTTPS"
+ fi
+
+ if (run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --confirm another-stack) >/dev/null 2>&1; then
+ fail "Destroy accepted an incorrect confirmation"
+ fi
+
+ if (OR_EKS_STACK_TEST_CLUSTER_STATUS=CREATING \
+ run_eks_stack status --name stack-a --cluster test-cluster) >/dev/null 2>&1; then
+ fail "A non-ACTIVE EKS cluster was accepted"
+ fi
+
+ if (OR_EKS_STACK_TEST_CONTEXT_ENDPOINT=https://another-cluster.example.com \
+ run_eks_stack status --name stack-a --cluster test-cluster) >/dev/null 2>&1; then
+ fail "A Kubernetes context for another cluster was accepted"
+ fi
+
+ if (OR_EKS_STACK_TEST_EXTERNAL_DNS_ARGS="${DEFAULT_EXTERNAL_DNS_ARGS/--domain-filter=example.com/--domain-filter=another.example.com}" \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com) >/dev/null 2>&1; then
+ fail "ExternalDNS with an unrelated domain filter was accepted"
+ fi
+
+ if (OR_EKS_STACK_TEST_EXTERNAL_DNS_ARGS="$DEFAULT_EXTERNAL_DNS_ARGS"$'\n--dry-run' \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com) >/dev/null 2>&1; then
+ fail "ExternalDNS dry-run mode was accepted"
+ fi
+
+ if (OR_EKS_STACK_TEST_EXTERNAL_DNS_ROLE_ARN= \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com) >/dev/null 2>&1; then
+ fail "ExternalDNS without an IRSA role was accepted"
+ fi
+
+ if (OR_EKS_STACK_TEST_EXTERNAL_DNS_ARGS="${DEFAULT_EXTERNAL_DNS_ARGS/--source=crd/}" \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress \
+ --certificate-mode managed) >/dev/null 2>&1; then
+ fail "Managed certificate mode accepted ExternalDNS without its CRD source"
+ fi
+ assert_not_contains $'or-stack\tprepare'
+
+ if (OR_EKS_STACK_TEST_DNS_ENDPOINT_CRD=false \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress \
+ --certificate-mode managed) >/dev/null 2>&1; then
+ fail "Managed certificate mode accepted a missing DNSEndpoint CRD"
+ fi
+ assert_not_contains $'or-stack\tprepare'
+
+ if (run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress \
+ --certificate-mode existing \
+ --certificate-arn arn:aws:acm:us-east-1:123456789012:certificate/test) \
+ >/dev/null 2>&1; then
+ fail "An ACM certificate from another region was accepted"
+ fi
+ assert_not_contains $'or-stack\tapply'
+
+ if (run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress \
+ --certificate-mode existing \
+ --certificate-arn arn:aws:acm:eu-west-1:210987654321:certificate/test) \
+ >/dev/null 2>&1; then
+ fail "An ACM certificate from another account was accepted"
+ fi
+ assert_not_contains $'or-stack\tapply'
+
+ if (OR_EKS_STACK_TEST_CERTIFICATE_JSON='{"Certificate":{"Status":"PENDING_VALIDATION","DomainName":"*.example.com","SubjectAlternativeNames":["*.example.com"]}}' \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress \
+ --certificate-mode existing --certificate-arn "$TEST_CERTIFICATE_ARN") \
+ >/dev/null 2>&1; then
+ fail "A non-issued ACM certificate was accepted"
+ fi
+ assert_not_contains $'or-stack\tapply'
+
+ if (OR_EKS_STACK_TEST_CERTIFICATE_JSON='{"Certificate":{"Status":"ISSUED","DomainName":"*.example.com","SubjectAlternativeNames":["*.example.com"]}}' \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname nested.stack-a.example.com --exposure ingress \
+ --certificate-mode existing --certificate-arn "$TEST_CERTIFICATE_ARN") \
+ >/dev/null 2>&1; then
+ fail "A one-level wildcard covered a multi-level hostname"
+ fi
+ assert_not_contains $'or-stack\tapply'
+
+ if (OR_EKS_STACK_TEST_CERTIFICATE_TAGS='{"Tags":[{"Key":"openremote.io/managed-by","Value":"or-eks-stack"}]}' \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress \
+ --certificate-mode existing --certificate-arn "$TEST_CERTIFICATE_ARN") \
+ >/dev/null 2>&1; then
+ fail "A stack-managed ACM certificate was accepted in existing mode"
+ fi
+ assert_not_contains $'or-stack\tapply'
+
+ if (OR_EKS_STACK_TEST_SHARED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_CERTIFICATE_TAGS='{"Tags":[{"Key":"openremote.io/managed-by","Value":"or-eks-stack"}]}' \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress \
+ --certificate-mode shared) >/dev/null 2>&1; then
+ fail "A stack-managed ACM certificate was accepted in shared mode"
+ fi
+ assert_not_contains $'or-stack\tapply'
+
+ if addresses_intersect '192.0.2.10' '198.51.100.20'; then
+ fail "Unrelated stale and current load-balancer addresses matched"
+ fi
+ addresses_intersect $'192.0.2.10\n192.0.2.11' $'198.51.100.20\n192.0.2.11' || fail \
+ "Matching hostname and load-balancer address sets were rejected"
+}
+
+
+test_hostname_reservation_lifecycle() {
+ local exposure registry
+ registry="$(jq -cn '{metadata:{resourceVersion:"1",labels:{"app.kubernetes.io/managed-by":"or-stack"}},
+ data:{"stack-a.example.com":({namespace:"stack-a",namespaceUID:"uid-stack-a",target:"eks",dns:"external-dns",exposure:"haproxy"}|tojson)}}')"
+
+ for exposure in haproxy ingress; do
+ if (OR_EKS_STACK_TEST_PREPARE_FAIL=true run_eks_stack apply --name stack-a \
+ --cluster test-cluster --hostname stack-a.example.com --exposure "$exposure" --profile '') >/dev/null 2>&1; then
+ fail "Failed hostname reservation did not stop $exposure apply"
+ fi
+ assert_contains $'or-stack\tprepare'
+ assert_contains $'--hostname\tstack-a.example.com'
+ assert_not_contains $'aws-acm\trequest-certificate'
+ assert_not_contains $'or-stack\tapply'
+ done
+
+ OR_EKS_STACK_TEST_REGISTRY_JSON="$registry" run_eks_stack destroy --name stack-a \
+ --cluster test-cluster --confirm stack-a --profile '' >/dev/null
+ assert_before $'or-stack\tdestroy' $'stack-a.example.com\tA'
+ assert_before $'stack-a.example.com\tA' $'or-stack\trelease-hostname'
+
+ OR_EKS_STACK_TEST_REGISTRY_JSON="$registry" OR_EKS_STACK_TEST_NAMESPACE_EXISTS=false \
+ run_eks_stack destroy --name stack-a --cluster test-cluster --confirm stack-a --profile '' >/dev/null
+ assert_not_contains $'or-stack\tdestroy'
+ assert_contains $'or-stack\trelease-hostname'
+
+ if (OR_EKS_STACK_TEST_REGISTRY_JSON="$registry" OR_EKS_STACK_TEST_DNS_STATUS=SERVFAIL \
+ run_eks_stack destroy --name stack-a --cluster test-cluster --confirm stack-a \
+ --profile '' --timeout 1s) >/dev/null 2>&1; then
+ fail "Failed DNS confirmation released the hostname reservation"
+ fi
+ assert_not_contains $'or-stack\trelease-hostname'
+
+ if (OR_EKS_STACK_TEST_REGISTRY_JSON="$registry" OR_EKS_STACK_TEST_DNS_FLAGS=qr \
+ run_eks_stack destroy --name stack-a --cluster test-cluster --confirm stack-a \
+ --profile '' --timeout 1s) >/dev/null 2>&1; then
+ fail "Non-authoritative DNS confirmation released the hostname reservation"
+ fi
+ assert_not_contains $'or-stack\trelease-hostname'
+
+ registry="$(jq '.data["stack-a.example.com"] |= (fromjson | .exposure="ingress" | tojson)' <<< "$registry")"
+ OR_EKS_STACK_TEST_REGISTRY_JSON="$registry" test_managed_destroy
+ assert_before $'aws-acm\tdelete-certificate' $'or-stack\trelease-hostname'
+}
+
+test_destroy_handles_incomplete_mqtts_lifecycle() {
+ local failure_output
+ local output hostname scenario expected_error
+ local registry
+
+ registry="$(jq -cn '{metadata:{resourceVersion:"1",labels:{"app.kubernetes.io/managed-by":"or-stack"}},
+ data:{
+ "stack-a.example.com":({namespace:"stack-a",namespaceUID:"uid-stack-a",target:"eks",dns:"external-dns",exposure:"ingress",endpoint:"web"}|tojson),
+ "mqtt-stack-a.example.com":({namespace:"stack-a",namespaceUID:"uid-stack-a",target:"eks",dns:"external-dns",exposure:"ingress",endpoint:"mqtts"}|tojson)
+ }}')"
+
+ if failure_output="$(OR_EKS_STACK_TEST_REGISTRY_JSON="$registry" \
+ run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a 2>&1)"; then
+ fail "or-eks-stack destroy accepted a low-level Ingress MQTTS endpoint"
+ fi
+ grep -F -- 'low-level or unowned Ingress MQTTS endpoint' <<< "$failure_output" >/dev/null || fail "$failure_output"
+ assert_not_contains $'or-stack\tdestroy'
+ assert_not_contains $'or-stack\trelease-hostname'
+
+ local OR_EKS_STACK_TEST_REGISTRY_JSON="$registry"
+ local OR_EKS_STACK_TEST_MQTTS_SERVICE_EXISTS=false
+ local OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE=""
+ local OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_ARN=""
+ local OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME=""
+ local OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_EXISTS=false
+ local OR_EKS_STACK_TEST_MQTTS_SERVICE_READ_SUCCESS=true
+ local OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_READ_SUCCESS=true
+ local OR_EKS_STACK_TEST_LOAD_BALANCERS_JSON=""
+ local OR_EKS_STACK_TEST_DNS_STATUS=NOERROR
+
+ # Interruption can occur just after the reservation write, or after prepare
+ # has also annotated the hostname but before the facade records mode.
+ for hostname in '' mqtt-stack-a.example.com; do
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME="$hostname"
+ output="$(run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a)"
+ grep -F 'incomplete MQTTS prepare' <<< "$output" >/dev/null || fail "$output"
+ assert_contains $'or-stack\tdestroy'
+ assert_before $'get\tdnsendpoint\tacm-validation-mqtts' $'or-stack\tdestroy'
+ assert_before $'aws-elbv2\tdescribe-load-balancers' $'or-stack\trelease-hostname'
+ assert_before $'mqtt-stack-a.example.com\tA' $'or-stack\trelease-hostname'
+ assert_contains $'--hostname\tmqtt-stack-a.example.com\t--confirm\tstack-a'
+ assert_contains $'--hostname\tstack-a.example.com\t--confirm\tstack-a'
+ assert_not_contains $'aws-acm\tdelete-certificate'
+ done
+
+ for scenario in certificate validation hostname mode service-read validation-read nlb dns; do
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE=""
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_ARN=""
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME=mqtt-stack-a.example.com
+ OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_EXISTS=false
+ OR_EKS_STACK_TEST_MQTTS_SERVICE_READ_SUCCESS=true
+ OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_READ_SUCCESS=true
+ OR_EKS_STACK_TEST_LOAD_BALANCERS_JSON=""
+ OR_EKS_STACK_TEST_DNS_STATUS=NOERROR
+ case "$scenario" in
+ certificate)
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_ARN="$TEST_MQTTS_CERTIFICATE_ARN"
+ expected_error='low-level or unowned Ingress MQTTS endpoint' ;;
+ validation)
+ OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_EXISTS=true
+ expected_error='MQTTS validation resources without a recorded certificate mode' ;;
+ hostname)
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_HOSTNAME=other.example.com
+ expected_error="but reserves 'mqtt-stack-a.example.com'" ;;
+ mode)
+ OR_EKS_STACK_TEST_RECORDED_MQTTS_CERTIFICATE_MODE=unsupported
+ expected_error='low-level or unowned Ingress MQTTS endpoint' ;;
+ service-read)
+ OR_EKS_STACK_TEST_MQTTS_SERVICE_READ_SUCCESS=false
+ expected_error="Unable to inspect Service 'stack-a/proxy-mqtts'" ;;
+ validation-read)
+ OR_EKS_STACK_TEST_MQTTS_DNS_ENDPOINT_READ_SUCCESS=false
+ expected_error='Unable to inspect validation resources for the incomplete MQTTS configuration' ;;
+ nlb)
+ OR_EKS_STACK_TEST_LOAD_BALANCERS_JSON="$DEFAULT_LOAD_BALANCERS_JSON"
+ expected_error='MQTTS NLB deletion' ;;
+ dns)
+ OR_EKS_STACK_TEST_DNS_STATUS=SERVFAIL
+ expected_error='DNS' ;;
+ esac
+ if failure_output="$(run_eks_stack destroy --name stack-a --cluster test-cluster \
+ --profile '' --confirm stack-a --timeout 1s 2>&1)"; then
+ fail "Destroy accepted unsafe incomplete MQTTS cleanup ($scenario)"
+ fi
+ grep -F "$expected_error" <<< "$failure_output" >/dev/null || fail "$failure_output"
+ case "$scenario" in
+ nlb|dns) assert_contains $'or-stack\tdestroy' ;;
+ *) assert_not_contains $'or-stack\tdestroy' ;;
+ esac
+ assert_not_contains $'or-stack\trelease-hostname'
+ assert_not_contains $'aws-acm\tdelete-certificate'
+ done
+}
+
+test_destroy_requires_current_metadata() {
+ local mode failure_output ingress_json
+ for mode in acme existing shared managed; do
+ if failure_output="$(OR_EKS_STACK_TEST_REGISTRY_JSON='' \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE="$mode" \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ run_eks_stack destroy --name stack-a --cluster test-cluster --confirm stack-a 2>&1)"; then
+ fail "Destroy accepted $mode mode without a hostname reservation"
+ fi
+ grep -F 'has no hostname reservation' <<< "$failure_output" >/dev/null || fail "$failure_output"
+ assert_not_contains $'aws-acm\t'
+ assert_not_contains $'or-stack\tdestroy'
+ assert_not_contains $'delete\tingress'
+ assert_not_contains $'or-stack\trelease-hostname'
+ done
+
+ if (OR_EKS_STACK_TEST_REGISTRY_JSON='' OR_EKS_STACK_TEST_NAMESPACE_EXISTS=false \
+ run_eks_stack destroy --name stack-a --cluster test-cluster --confirm stack-a) >/dev/null 2>&1; then
+ fail "Destroy recovered external resources without retained hostname ownership"
+ fi
+ assert_not_contains $'aws-acm\t'
+
+ if failure_output="$(OR_EKS_STACK_TEST_RECORDED_HOSTNAME=another.example.com \
+ run_eks_stack destroy --name stack-a --cluster test-cluster --confirm stack-a 2>&1)"; then
+ fail "Destroy accepted namespace hostname metadata that conflicts with the reservation"
+ fi
+ grep -F "but reserves 'stack-a.example.com'" <<< "$failure_output" >/dev/null || fail "$failure_output"
+ assert_not_contains $'or-stack\tdestroy'
+
+ ingress_json="$(jq '.metadata.annotations["external-dns.alpha.kubernetes.io/hostname"] =
+ .metadata.annotations["external-dns.kubernetes.io/hostname"] |
+ del(.metadata.annotations["external-dns.kubernetes.io/hostname"])' <<< "$DEFAULT_INGRESS_JSON")"
+ if (OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=managed \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_CERTIFICATE_TAGS="$MANAGED_CERTIFICATE_TAGS" \
+ OR_EKS_STACK_TEST_CERTIFICATE_JSON="$MANAGED_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_INGRESS_EXISTS=true OR_EKS_STACK_TEST_INGRESS_JSON="$ingress_json" \
+ run_eks_stack destroy --name stack-a --cluster test-cluster --confirm stack-a) >/dev/null 2>&1; then
+ fail "Destroy accepted a stack Ingress without its required GA hostname annotation"
+ fi
+ assert_not_contains $'delete\tingress'
+ assert_not_contains $'or-stack\tdestroy'
+}
+
+test_persistent_certificate_modes() {
+ local recorded_mode recorded_arn requested_mode failure_output registry
+ local mode_args=()
+
+ # First apply defaults to managed; adding a shared certificate must not
+ # change the persisted mode or ARN when the original command is repeated.
+ OR_EKS_STACK_TEST_NAMESPACE_EXISTS=false \
+ OR_EKS_STACK_TEST_CERTIFICATE_JSON="$MANAGED_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_CERTIFICATE_TAGS="$MANAGED_CERTIFICATE_TAGS" \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress >/dev/null
+ recorded_mode="$(jq -r '.metadata.annotations["openremote.io/certificate-mode"]' "$CERTIFICATE_NAMESPACE_FILE")"
+ recorded_arn="$(jq -r '.metadata.annotations["openremote.io/certificate-arn"]' "$CERTIFICATE_NAMESPACE_FILE")"
+ [ "$recorded_mode" = managed ] && [ "$recorded_arn" = "$TEST_CERTIFICATE_ARN" ] || fail "Initial managed mode was not recorded"
+
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE="$recorded_mode" \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$recorded_arn" \
+ OR_EKS_STACK_TEST_SHARED_CERTIFICATE_ARN="$TEST_SECOND_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_CERTIFICATE_JSON="$MANAGED_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_CERTIFICATE_TAGS="$MANAGED_CERTIFICATE_TAGS" \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress >/dev/null
+ assert_not_contains $'get\tingressclass\topenremote-alb'
+ assert_not_contains $'aws-acm\trequest-certificate'
+ assert_contains $'--certificate-arn\t'"$TEST_CERTIFICATE_ARN"
+ jq -e '.metadata.annotations["openremote.io/certificate-mode"] == "managed"' \
+ "$CERTIFICATE_NAMESPACE_FILE" >/dev/null || fail "Reapply changed managed mode"
+ # A remains normally destroyable even while B is the cluster default.
+ OR_EKS_STACK_TEST_SHARED_CERTIFICATE_ARN="$TEST_SECOND_CERTIFICATE_ARN" test_managed_destroy
+
+ for recorded_mode in managed existing shared; do
+ for requested_mode in managed existing shared; do
+ [ "$recorded_mode" != "$requested_mode" ] || continue
+ mode_args=(--certificate-mode "$requested_mode")
+ if [ "$requested_mode" = existing ]; then
+ mode_args+=(--certificate-arn "$TEST_SECOND_CERTIFICATE_ARN")
+ fi
+ if failure_output="$(OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE="$recorded_mode" \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_SHARED_CERTIFICATE_ARN="$TEST_SECOND_CERTIFICATE_ARN" \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress "${mode_args[@]}" 2>&1)"; then
+ fail "Accepted mode migration from $recorded_mode to $requested_mode"
+ fi
+ grep -F 'in-place certificate mode migration' <<< "$failure_output" >/dev/null || fail "$failure_output"
+ assert_not_contains $'or-stack\tprepare'
+ assert_not_contains $'annotate\tnamespace'
+ assert_not_contains $'aws-acm\trequest-certificate'
+ done
+ done
+
+ # Existing mode can reuse its recorded ARN, or explicitly rotate within mode.
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_SHARED_CERTIFICATE_ARN="$TEST_SECOND_CERTIFICATE_ARN" \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress >/dev/null
+ assert_contains $'--certificate-arn\t'"$TEST_CERTIFICATE_ARN"
+ assert_not_contains $'get\tingressclass\topenremote-alb'
+
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=existing \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress \
+ --certificate-mode existing --certificate-arn "$TEST_SECOND_CERTIFICATE_ARN" >/dev/null
+ assert_contains $'--certificate-arn\t'"$TEST_SECOND_CERTIFICATE_ARN"
+ assert_not_contains $'aws-acm\trequest-certificate'
+
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=shared \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN="$TEST_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_SHARED_CERTIFICATE_ARN="$TEST_SECOND_CERTIFICATE_ARN" \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress >/dev/null
+ assert_contains $'--certificate-arn\t'"$TEST_SECOND_CERTIFICATE_ARN"
+ assert_not_contains $'aws-acm\trequest-certificate'
+
+ if failure_output="$(OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=shared \
+ OR_EKS_STACK_TEST_SHARED_CERTIFICATE_ARN='' \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress 2>&1)"; then
+ fail "Shared mode fell back to managed after cluster configuration was removed"
+ fi
+ grep -F 'shared requires a certificate configured' <<< "$failure_output" >/dev/null || fail "$failure_output"
+ assert_not_contains $'or-stack\tprepare'
+ assert_not_contains $'aws-acm\trequest-certificate'
+
+ # Interruption between pinning the mode and recording the ACM ARN resumes
+ # using discovery, even if the cluster's default has changed meanwhile.
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=managed \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN='' \
+ OR_EKS_STACK_TEST_SHARED_CERTIFICATE_ARN="$TEST_SECOND_CERTIFICATE_ARN" \
+ OR_EKS_STACK_TEST_CERTIFICATE_LIST="{\"CertificateSummaryList\":[{\"CertificateArn\":\"$TEST_CERTIFICATE_ARN\",\"DomainName\":\"stack-a.example.com\"}]}" \
+ OR_EKS_STACK_TEST_CERTIFICATE_JSON="$MANAGED_CERTIFICATE_JSON" \
+ OR_EKS_STACK_TEST_CERTIFICATE_TAGS="$MANAGED_CERTIFICATE_TAGS" \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress >/dev/null
+ assert_contains $'aws-acm\tlist-certificates'
+ assert_not_contains $'aws-acm\trequest-certificate'
+
+ if (OR_EKS_STACK_TEST_NAMESPACE_READ_SUCCESS=false \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress) >/dev/null 2>&1; then
+ fail "Namespace read error was treated as absent certificate metadata"
+ fi
+ assert_not_contains $'or-stack\tprepare'
+
+ if (OR_EKS_STACK_TEST_MODE_CONFLICT=true \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress --certificate-mode managed) >/dev/null 2>&1; then
+ fail "Mode reservation conflict did not stop apply"
+ fi
+ assert_contains '--resource-version=1'
+ assert_not_contains $'aws-acm\trequest-certificate'
+ assert_not_contains $'or-stack\tapply'
+
+ if (OR_EKS_STACK_TEST_ACM_REQUEST_FAIL=true \
+ run_eks_stack apply --name stack-a --cluster test-cluster \
+ --hostname stack-a.example.com --exposure ingress --certificate-mode managed) >/dev/null 2>&1; then
+ fail "Failed ACM request did not stop apply"
+ fi
+ jq -e '.metadata.annotations["openremote.io/certificate-mode"] == "managed" and
+ ((.metadata.annotations["openremote.io/certificate-arn"] // "") == "")' \
+ "$CERTIFICATE_NAMESPACE_FILE" >/dev/null || fail "Failed ACM request lost the initial mode"
+ assert_not_contains $'or-stack\tapply'
+
+ registry="$(jq -cn '{metadata:{resourceVersion:"1",labels:{"app.kubernetes.io/managed-by":"or-stack"}},
+ data:{"stack-a.example.com":({namespace:"stack-a",namespaceUID:"uid-stack-a",target:"eks",dns:"external-dns",exposure:"ingress"}|tojson)}}')"
+ OR_EKS_STACK_TEST_REGISTRY_JSON="$registry" \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=managed \
+ OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_ARN='' \
+ run_eks_stack destroy --name stack-a --cluster test-cluster --confirm stack-a >/dev/null
+ assert_contains $'or-stack\tdestroy'
+ assert_contains $'or-stack\trelease-hostname'
+ assert_not_contains $'aws-acm\tdelete-certificate'
+
+ if (OR_EKS_STACK_TEST_RECORDED_CERTIFICATE_MODE=managed \
+ OR_EKS_STACK_TEST_DNS_ENDPOINT_EXISTS=true \
+ run_eks_stack destroy --name stack-a --cluster test-cluster --confirm stack-a) >/dev/null 2>&1; then
+ fail "Destroy accepted validation records without an identifiable certificate"
+ fi
+ assert_not_contains $'or-stack\tdestroy'
+}
+
+
+test_apply
+test_ingress_existing_certificate_apply
+test_ingress_mqtts_existing_certificate_lifecycle
+test_ingress_mqtts_destroy_checks_retained_nlb
+test_ingress_mqtts_managed_certificate_lifecycle
+test_dual_managed_certificate_lifecycle
+test_ingress_mqtts_shared_certificate_lifecycle
+test_ingress_mqtts_validation
+test_ingress_shared_certificate_apply
+test_ingress_managed_certificate_apply
+test_managed_certificate_reuse_and_safety
+test_validation_dns_endpoint_read_safety
+test_apply_reuses_valid_certificate
+test_acme_state_and_readiness
+test_lifecycle_delegation
+test_managed_destroy
+test_managed_destroy_safety
+test_validation
+test_managed_domain_boundary
+
+test_hostname_reservation_lifecycle
+test_destroy_handles_incomplete_mqtts_lifecycle
+test_destroy_requires_current_metadata
+test_persistent_certificate_modes
+
+printf 'or-eks-stack tests passed\n'
diff --git a/kubernetes/test/or-setup-test b/kubernetes/test/or-setup-test
new file mode 100755
index 000000000..2ba799db0
--- /dev/null
+++ b/kubernetes/test/or-setup-test
@@ -0,0 +1,440 @@
+#!/usr/bin/env bash
+
+set -Eeuo pipefail
+
+TEST_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+OR_SETUP_CHART="$(cd "$TEST_DIR/../or-setup" && pwd)"
+MANAGER_CHART="$(cd "$TEST_DIR/../manager" && pwd)"
+KEYCLOAK_CHART="$(cd "$TEST_DIR/../keycloak" && pwd)"
+PROXY_CHART="$(cd "$TEST_DIR/../proxy" && pwd)"
+LOCAL_TARGET_VALUES="$TEST_DIR/../stack/targets/local/or-setup.yaml"
+EKS_TARGET_VALUES="$TEST_DIR/../stack/targets/eks/or-setup.yaml"
+EKS_MANAGER_VALUES="$TEST_DIR/../stack/targets/eks/manager.yaml"
+NONE_EXPOSURE_VALUES="$TEST_DIR/../stack/exposures/none/or-setup.yaml"
+INGRESS_EXPOSURE_VALUES="$TEST_DIR/../stack/exposures/ingress/or-setup.yaml"
+INGRESS_PROXY_VALUES="$TEST_DIR/../stack/exposures/ingress/proxy.yaml"
+EKS_INGRESS_PROXY_VALUES="$TEST_DIR/../stack/exposures/ingress/eks/proxy.yaml"
+HAPROXY_EXPOSURE_VALUES="$TEST_DIR/../stack/exposures/haproxy/or-setup.yaml"
+HAPROXY_PROXY_VALUES="$TEST_DIR/../stack/exposures/haproxy/proxy.yaml"
+LOCAL_HAPROXY_PROXY_VALUES="$TEST_DIR/../stack/exposures/haproxy/local/proxy.yaml"
+EKS_HAPROXY_PROXY_VALUES="$TEST_DIR/../stack/exposures/haproxy/eks/proxy.yaml"
+LEGACY_EKS_PROXY_VALUES="$PROXY_CHART/values-eks.yaml"
+TEST_CERTIFICATE_ARN="arn:aws:acm:eu-west-1:123456789012:certificate/test-certificate"
+TEST_MQTTS_CERTIFICATE_ARN="arn:aws:acm:eu-west-1:123456789012:certificate/test-mqtts-certificate"
+
+fail() {
+ printf 'FAIL: %s\n' "$*" >&2
+ exit 1
+}
+
+secret_value() {
+ local key="$1"
+ awk -v key="$key" '$1 == key ":" { gsub(/"/, "", $2); print $2; exit }'
+}
+
+assert_env_value() {
+ local name="$1"
+ local expected="$2"
+ local actual
+ actual="$(awk -v name="$name" '$1 == "-" && $2 == "name:" && $3 == name {getline; print $2}')"
+ [ "$actual" = "\"$expected\"" ] || fail \
+ "Expected $name to have quoted value '$expected', got '$actual'"
+}
+
+helm lint "$OR_SETUP_CHART" >/dev/null
+helm lint "$MANAGER_CHART" >/dev/null
+helm lint "$KEYCLOAK_CHART" >/dev/null
+helm lint "$PROXY_CHART" >/dev/null
+
+rendered_a="$(helm template or-setup "$OR_SETUP_CHART" \
+ --namespace stack-a \
+ --values "$LOCAL_TARGET_VALUES" \
+ --values "$NONE_EXPOSURE_VALUES")"
+rendered_b="$(helm template or-setup "$OR_SETUP_CHART" \
+ --namespace stack-b \
+ --values "$EKS_TARGET_VALUES" \
+ --values "$INGRESS_EXPOSURE_VALUES")"
+rendered_proxy_policy="$(helm template or-setup "$OR_SETUP_CHART" \
+ --namespace stack-proxy \
+ --values "$LOCAL_TARGET_VALUES" \
+ --values "$HAPROXY_EXPOSURE_VALUES")"
+rendered_mqtts_policy="$(helm template or-setup "$OR_SETUP_CHART" \
+ --namespace stack-mqtts \
+ --values "$LOCAL_TARGET_VALUES" \
+ --values "$HAPROXY_EXPOSURE_VALUES" \
+ --set networkPolicy.proxy.mqtts.enabled=true)"
+rendered_ingress_mqtts_policy="$(helm template or-setup "$OR_SETUP_CHART" \
+ --namespace stack-ingress-mqtts \
+ --values "$EKS_TARGET_VALUES" \
+ --values "$INGRESS_EXPOSURE_VALUES" \
+ --set networkPolicy.proxy.mqtts.enabled=true \
+ --set-string networkPolicy.proxy.mqtts.port=1883)"
+
+grep -F -- 'kind: NetworkPolicy' <<< "$rendered_a" >/dev/null || fail \
+ "Stack ingress NetworkPolicy was not rendered"
+grep -F -- 'name: openremote-stack-ingress' <<< "$rendered_a" >/dev/null || fail \
+ "Stack ingress NetworkPolicy has the wrong name"
+grep -F -- 'podSelector: {}' <<< "$rendered_a" >/dev/null || fail \
+ "Stack ingress NetworkPolicy does not allow same-namespace Pods"
+if grep -F -- '- Egress' <<< "$rendered_a" >/dev/null; then
+ fail "Stack NetworkPolicy unexpectedly restricts outbound traffic"
+fi
+if grep -F -- 'name: openremote-public-http-ingress' <<< "$rendered_a" >/dev/null; then
+ fail "None exposure unexpectedly allows public HTTP ingress"
+fi
+
+[ "$(grep -c '^kind: NetworkPolicy$' <<< "$rendered_b")" -eq 2 ] || fail \
+ "Ingress exposure did not render both stack and gateway policies"
+grep -F -- 'name: openremote-proxy-ingress' <<< "$rendered_b" >/dev/null || fail \
+ "Ingress gateway NetworkPolicy was not rendered"
+grep -F -- 'app.kubernetes.io/name: proxy' <<< "$rendered_b" >/dev/null || fail \
+ "Ingress gateway policy does not select only the proxy"
+grep -F -- 'port: 8080' <<< "$rendered_b" >/dev/null || fail \
+ "Ingress gateway policy does not restrict access to the HTTP port"
+if grep -F -- 'port: 8443' <<< "$rendered_b" >/dev/null; then
+ fail "Ingress gateway policy unexpectedly opens the proxy TLS port"
+fi
+if grep -F -- 'openremote.io/public-http' <<< "$rendered_b" >/dev/null; then
+ fail "Ingress exposure still contains the direct Manager/Keycloak public selector"
+fi
+
+[ "$(grep -c '^kind: NetworkPolicy$' <<< "$rendered_proxy_policy")" -eq 2 ] || fail \
+ "HAProxy exposure did not render both stack and proxy ingress policies"
+grep -F -- 'name: openremote-proxy-ingress' <<< "$rendered_proxy_policy" >/dev/null || fail \
+ "HAProxy exposure did not render its public proxy policy"
+grep -F -- 'app.kubernetes.io/name: proxy' <<< "$rendered_proxy_policy" >/dev/null || fail \
+ "HAProxy ingress policy does not select only the proxy"
+grep -F -- 'port: 8443' <<< "$rendered_proxy_policy" >/dev/null || fail \
+ "HAProxy ingress policy does not allow the proxy HTTPS container port"
+if grep -F -- 'openremote.io/public-http' <<< "$rendered_proxy_policy" >/dev/null; then
+ fail "HAProxy exposure also enabled direct public Manager/Keycloak access"
+fi
+grep -F -- 'port: 8883' <<< "$rendered_mqtts_policy" >/dev/null || fail \
+ "MQTTS-enabled HAProxy policy does not allow the proxy TLS listener"
+grep -F -- 'port: 1883' <<< "$rendered_ingress_mqtts_policy" >/dev/null || fail \
+ "MQTTS-enabled Ingress policy does not allow the edge-terminated proxy listener"
+
+grep -F -- 'helm.sh/resource-policy: keep' <<< "$rendered_a" >/dev/null || fail \
+ "Generated Secret is not retained during Helm uninstall"
+
+admin_password_a="$(secret_value admin-password <<< "$rendered_a")"
+admin_password_b="$(secret_value admin-password <<< "$rendered_b")"
+postgresql_password_a="$(secret_value postgresql-password <<< "$rendered_a")"
+
+[ "${#admin_password_a}" -eq 44 ] || fail \
+ "Generated Manager password is not the encoded form of a 32-character value"
+[ "${#postgresql_password_a}" -eq 44 ] || fail \
+ "Generated PostgreSQL password is not the encoded form of a 32-character value"
+[ "$admin_password_a" != "$admin_password_b" ] || fail \
+ "Independent chart renders generated the same Manager password"
+[ "$admin_password_a" != "c2VjcmV0" ] || fail \
+ "Chart still uses the legacy default Manager password"
+[ "$postgresql_password_a" != "cG9zdGdyZXM=" ] || fail \
+ "Chart still uses the legacy default PostgreSQL password"
+grep -F -- 'postgresql-username: "cG9zdGdyZXM="' <<< "$rendered_a" >/dev/null || fail \
+ "Default PostgreSQL username is not postgres"
+
+rendered_explicit="$(helm template or-setup "$OR_SETUP_CHART" \
+ --namespace stack-explicit \
+ --set-string credentials.adminPassword=chosen-admin \
+ --set-string credentials.postgresqlUsername=database-user \
+ --set-string credentials.postgresqlPassword=chosen-database)"
+
+grep -F -- 'admin-password: "Y2hvc2VuLWFkbWlu"' <<< "$rendered_explicit" >/dev/null || fail \
+ "Explicit Manager password was not rendered"
+grep -F -- 'postgresql-username: "ZGF0YWJhc2UtdXNlcg=="' <<< "$rendered_explicit" >/dev/null || fail \
+ "Explicit PostgreSQL username was not rendered"
+grep -F -- 'postgresql-password: "Y2hvc2VuLWRhdGFiYXNl"' <<< "$rendered_explicit" >/dev/null || fail \
+ "Explicit PostgreSQL password was not rendered"
+
+rendered_manager="$(helm template manager "$MANAGER_CHART" --namespace stack-a)"
+
+assert_env_value OR_WEBSERVER_LISTEN_HOST 0.0.0.0 <<< "$rendered_manager"
+assert_env_value OR_WEBSERVER_LISTEN_PORT 8080 <<< "$rendered_manager"
+assert_env_value OR_WEBSERVER_DUMP_REQUESTS false <<< "$rendered_manager"
+[ "$(grep -c -- '- name: OR_WEBSERVER_' <<< "$rendered_manager")" -eq 3 ] || fail \
+ "Default Manager chart should omit empty and null web server settings"
+grep -F -- 'containerPort: 8080' <<< "$rendered_manager" >/dev/null || fail \
+ "Default Manager HTTP container port is not 8080"
+
+for origin in https://localhost:8443 https://localhost:9443 https://stack.example.com; do
+ rendered_public_manager="$(helm template manager "$MANAGER_CHART" \
+ --namespace stack-a --set-string "or.webserver.allowedOrigins=$origin")"
+ assert_env_value OR_WEBSERVER_ALLOWED_ORIGINS "$origin" <<< "$rendered_public_manager"
+done
+
+rendered_webserver_manager="$(helm template manager "$MANAGER_CHART" \
+ --namespace stack-a --values - <<'YAML'
+service:
+ http:
+ port: 9090
+or:
+ webserver:
+ listenHost: "::"
+ listenPort: 9080
+ allowedOrigins: "https://stack.example.com,https://localhost:9443"
+ allowedMethods: "GET,POST,OPTIONS"
+ allowedHeaders: "Content-Type,Authorization"
+ exposedHeaders: "X-Request-ID"
+ dumpRequests: true
+ workerThreadsMax: 24
+ ioThreadsMax: 4
+ maxEntitySize: 0
+ multipartMaxEntitySize: 60000000
+YAML
+)"
+
+while read -r name expected; do
+ assert_env_value "$name" "$expected" <<< "$rendered_webserver_manager"
+done <<'ENV'
+OR_WEBSERVER_LISTEN_HOST ::
+OR_WEBSERVER_LISTEN_PORT 9080
+OR_WEBSERVER_ALLOWED_ORIGINS https://stack.example.com,https://localhost:9443
+OR_WEBSERVER_ALLOWED_METHODS GET,POST,OPTIONS
+OR_WEBSERVER_ALLOWED_HEADERS Content-Type,Authorization
+OR_WEBSERVER_EXPOSED_HEADERS X-Request-ID
+OR_WEBSERVER_DUMP_REQUESTS true
+OR_WEBSERVER_WORKER_THREADS_MAX 24
+OR_WEBSERVER_IO_THREADS_MAX 4
+OR_WEBSERVER_MAX_ENTITY_SIZE 0
+OR_WEBSERVER_MULTIPART_MAX_ENTITY_SIZE 60000000
+ENV
+
+grep -A1 -F -- '- name: http' <<< "$rendered_webserver_manager" | \
+ grep -F -- 'containerPort: 9080' >/dev/null || fail \
+ "Manager's named HTTP port does not follow or.webserver.listenPort"
+grep -A1 -F -- '- port: 9090' <<< "$rendered_webserver_manager" | \
+ grep -F -- 'targetPort: http' >/dev/null || fail \
+ "Manager Service does not route its independent port to the named HTTP port"
+[ "$(grep -c 'port: http' <<< "$rendered_webserver_manager")" -eq 3 ] || fail \
+ "Manager's three default health probes do not use the named HTTP port"
+
+rendered_default_webserver_manager="$(helm template manager "$MANAGER_CHART" \
+ --namespace stack-a --set or.webserver=null)"
+if grep -F -- '- name: OR_WEBSERVER_' <<< "$rendered_default_webserver_manager" >/dev/null; then
+ fail "Null web server settings should retain the application's defaults"
+fi
+grep -F -- 'containerPort: 8080' <<< "$rendered_default_webserver_manager" >/dev/null || fail \
+ "Unset Manager listen port should use the application's default port 8080"
+
+grep -F -- '- name: OR_DB_USERNAME' <<< "$rendered_manager" >/dev/null || fail \
+ "Manager does not receive its PostgreSQL username through OR_DB_USERNAME"
+grep -F -- '- name: OR_DB_PASSWORD' <<< "$rendered_manager" >/dev/null || fail \
+ "Manager does not receive its PostgreSQL password through OR_DB_PASSWORD"
+if grep -F -- '- name: KC_DB_PASSWORD' <<< "$rendered_manager" >/dev/null; then
+ fail "Manager incorrectly receives its own database password through KC_DB_PASSWORD"
+fi
+
+rendered_eks_internal_manager="$(helm template manager "$MANAGER_CHART" \
+ --namespace stack-internal \
+ --values "$EKS_MANAGER_VALUES")"
+rendered_internal_keycloak="$(helm template keycloak "$KEYCLOAK_CHART" \
+ --namespace stack-internal)"
+for rendered_internal_component in "$rendered_eks_internal_manager" "$rendered_internal_keycloak"; do
+ if grep -F -- 'kind: Ingress' <<< "$rendered_internal_component" >/dev/null; then
+ fail "Target values leaked an Ingress into none or HAProxy exposure"
+ fi
+ if grep -F -- 'openremote.io/public-http' <<< "$rendered_internal_component" >/dev/null; then
+ fail "Target values leaked a public HTTP Pod label into none or HAProxy exposure"
+ fi
+done
+
+rendered_eks_gateway="$(helm template proxy "$PROXY_CHART" \
+ --namespace stack-a \
+ --values "$INGRESS_PROXY_VALUES" \
+ --values "$EKS_INGRESS_PROXY_VALUES" \
+ --set-string or.hostname=stack-a.example.com \
+ --set-string or.managerHost=manager.stack-a.svc.cluster.local \
+ --set-string or.keycloakHost=keycloak.stack-a.svc.cluster.local \
+ --set-string or.nameserver=10.100.0.10:53 \
+ --set-string ingress.hostname=stack-a.example.com \
+ --set dns.enabled=true \
+ --set-string dns.hostname=stack-a.example.com \
+ --set-string "ingress.annotations.alb\\.ingress\\.kubernetes\\.io/certificate-arn=$TEST_CERTIFICATE_ARN")"
+
+grep -F -- 'ingressClassName: openremote-alb' <<< "$rendered_eks_gateway" >/dev/null || fail \
+ "EKS gateway does not use the shared OpenRemote ALB IngressClass"
+grep -F -- 'host: "stack-a.example.com"' <<< "$rendered_eks_gateway" >/dev/null || fail \
+ "EKS gateway Ingress does not use the stack hostname"
+grep -F -- "alb.ingress.kubernetes.io/certificate-arn: $TEST_CERTIFICATE_ARN" \
+ <<< "$rendered_eks_gateway" >/dev/null || fail \
+ "EKS gateway Ingress does not reference the stack certificate"
+grep -F -- 'alb.ingress.kubernetes.io/listen-ports: '\''[{"HTTP":80},{"HTTPS":443}]'\''' \
+ <<< "$rendered_eks_gateway" >/dev/null || fail \
+ "EKS gateway Ingress does not configure HTTP and HTTPS listeners"
+grep -F -- 'alb.ingress.kubernetes.io/healthcheck-path: /docker-health' \
+ <<< "$rendered_eks_gateway" >/dev/null || fail \
+ "EKS gateway Ingress does not use the proxy health endpoint"
+grep -F -- 'name: proxy' <<< "$rendered_eks_gateway" >/dev/null || fail \
+ "EKS gateway Ingress does not route to the proxy Service"
+[ "$(grep -c '^kind: Ingress$' <<< "$rendered_eks_gateway")" -eq 1 ] || fail \
+ "EKS gateway did not render exactly one Ingress"
+[ "$(grep -c 'external-dns.kubernetes.io/hostname: "stack-a.example.com"' <<< "$rendered_eks_gateway")" -eq 1 ] || fail \
+ "Ingress DNS intent was not declared exactly once"
+[ "$(grep -c 'openremote.io/managed-dns: "true"' <<< "$rendered_eks_gateway")" -eq 1 ] || fail \
+ "Ingress DNS selection label was not declared exactly once"
+grep -A12 '^kind: Ingress$' <<< "$rendered_eks_gateway" | \
+ grep -F -- 'openremote.io/managed-dns: "true"' >/dev/null || fail \
+ "Ingress DNS intent is missing its controller selection label"
+grep -F -- 'image: "openremote/proxy:3.2.19.0"' <<< "$rendered_eks_gateway" >/dev/null || fail \
+ "Ingress gateway does not use the expected proxy image"
+grep -A1 -F -- '- name: HAPROXY_CONFIG' <<< "$rendered_eks_gateway" | \
+ grep -F -- 'value: "/etc/haproxy/haproxy-edge-terminated-tls.cfg"' >/dev/null || fail \
+ "Ingress gateway does not use the edge-terminated TLS configuration"
+grep -A1 -F -- '- name: DISABLE_ACME' <<< "$rendered_eks_gateway" | \
+ grep -F -- 'value: "true"' >/dev/null || fail \
+ "Ingress gateway did not disable proxy-managed certificates"
+if grep -F -- 'kind: PersistentVolumeClaim' <<< "$rendered_eks_gateway" >/dev/null; then
+ fail "Ingress gateway unexpectedly rendered certificate persistence"
+fi
+if grep -F -- 'port: 443' <<< "$rendered_eks_gateway" >/dev/null; then
+ fail "Ingress gateway unexpectedly exposed the proxy TLS Service port"
+fi
+
+rendered_eks_mqtts_gateway="$(helm template proxy "$PROXY_CHART" \
+ --namespace stack-mqtts \
+ --values "$INGRESS_PROXY_VALUES" \
+ --values "$EKS_INGRESS_PROXY_VALUES" \
+ --set service.mqtts.enabled=true \
+ --set-string service.mqtts.port=18883 \
+ --set-string or.mqttPort=1883 \
+ --set service.mqtts.dns.enabled=true \
+ --set-string service.mqtts.dns.hostname=mqtt-stack.example.com \
+ --set-string "service.mqtts.annotations.service\\.beta\\.kubernetes\\.io/aws-load-balancer-ssl-cert=$TEST_MQTTS_CERTIFICATE_ARN")"
+grep -F -- 'name: proxy-mqtts' <<< "$rendered_eks_mqtts_gateway" >/dev/null || fail \
+ "Ingress MQTTS did not render a dedicated proxy-mqtts Service"
+grep -F -- 'service.beta.kubernetes.io/aws-load-balancer-type: external' \
+ <<< "$rendered_eks_mqtts_gateway" >/dev/null || fail \
+ "Ingress MQTTS Service is not managed by the AWS Load Balancer Controller"
+grep -F -- 'service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip' \
+ <<< "$rendered_eks_mqtts_gateway" >/dev/null || fail \
+ "Ingress MQTTS Service does not use NLB IP targets"
+grep -F -- 'service.beta.kubernetes.io/aws-load-balancer-ssl-ports: mqtts' \
+ <<< "$rendered_eks_mqtts_gateway" >/dev/null || fail \
+ "Ingress MQTTS NLB does not terminate TLS on its MQTTS listener"
+grep -F -- 'service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp' \
+ <<< "$rendered_eks_mqtts_gateway" >/dev/null || fail \
+ "Ingress MQTTS NLB does not forward plaintext TCP to the proxy gateway"
+grep -F -- "service.beta.kubernetes.io/aws-load-balancer-ssl-cert: $TEST_MQTTS_CERTIFICATE_ARN" \
+ <<< "$rendered_eks_mqtts_gateway" >/dev/null || fail \
+ "Ingress MQTTS NLB does not reference its ACM certificate"
+grep -F -- 'external-dns.kubernetes.io/hostname: "mqtt-stack.example.com"' \
+ <<< "$rendered_eks_mqtts_gateway" >/dev/null || fail \
+ "Ingress MQTTS Service does not declare its dedicated DNS hostname"
+grep -F -- '- port: 18883' <<< "$rendered_eks_mqtts_gateway" >/dev/null || fail \
+ "Ingress MQTTS public Service port was not configurable"
+grep -F -- 'targetPort: 1883' <<< "$rendered_eks_mqtts_gateway" >/dev/null || fail \
+ "Ingress MQTTS Service does not target the proxy's plaintext MQTT listener"
+grep -F -- 'containerPort: 1883' <<< "$rendered_eks_mqtts_gateway" >/dev/null || fail \
+ "Ingress proxy does not declare its edge-terminated MQTT listener"
+[ "$(grep -c 'external-dns.kubernetes.io/hostname: "mqtt-stack.example.com"' <<< "$rendered_eks_mqtts_gateway")" -eq 1 ] || fail \
+ "Ingress MQTTS DNS intent was not declared exactly once"
+
+rendered_local_proxy="$(helm template proxy "$PROXY_CHART" \
+ --namespace stack-a \
+ --values "$HAPROXY_PROXY_VALUES" \
+ --values "$LOCAL_HAPROXY_PROXY_VALUES" \
+ --set-string or.managerHost=manager.stack-a.svc.cluster.local \
+ --set-string or.keycloakHost=keycloak.stack-a.svc.cluster.local \
+ --set-string or.nameserver=10.96.0.10:53 \
+ --set-string service.http.port=8080 \
+ --set-string service.https.port=8443)"
+
+grep -F -- 'kind: PersistentVolumeClaim' <<< "$rendered_local_proxy" >/dev/null || fail \
+ "HAProxy chart did not render durable certificate storage"
+grep -F -- 'helm.sh/resource-policy: keep' <<< "$rendered_local_proxy" >/dev/null || fail \
+ "HAProxy certificate PVC is not retained during Helm uninstall"
+grep -F -- 'mountPath: /deployment' <<< "$rendered_local_proxy" >/dev/null || fail \
+ "HAProxy certificate state is not mounted at the proxy image's persistent path"
+grep -F -- 'type: Recreate' <<< "$rendered_local_proxy" >/dev/null || fail \
+ "HAProxy Deployment can roll two Pods against a ReadWriteOnce certificate volume"
+grep -F -- 'value: "manager.stack-a.svc.cluster.local"' <<< "$rendered_local_proxy" >/dev/null || fail \
+ "HAProxy does not use the namespace-qualified Manager service"
+grep -F -- 'value: "keycloak.stack-a.svc.cluster.local"' <<< "$rendered_local_proxy" >/dev/null || fail \
+ "HAProxy does not use the namespace-qualified Keycloak service"
+grep -F -- 'value: "10.96.0.10:53"' <<< "$rendered_local_proxy" >/dev/null || fail \
+ "HAProxy does not use the discovered cluster DNS endpoint"
+grep -F -- 'port: 8080' <<< "$rendered_local_proxy" >/dev/null || fail \
+ "HAProxy local HTTP Service port was not configurable"
+grep -F -- 'port: 8443' <<< "$rendered_local_proxy" >/dev/null || fail \
+ "HAProxy local HTTPS Service port was not configurable"
+if grep -F -- 'name: mqtts' <<< "$rendered_local_proxy" >/dev/null; then
+ fail "HAProxy exposed MQTTS without explicit opt-in"
+fi
+
+rendered_local_mqtts_proxy="$(helm template proxy "$PROXY_CHART" \
+ --namespace stack-mqtts \
+ --values "$HAPROXY_PROXY_VALUES" \
+ --values "$LOCAL_HAPROXY_PROXY_VALUES" \
+ --set service.mqtts.enabled=true \
+ --set-string service.mqtts.port=18883 \
+ --set-string or.mqttPort=8883)"
+grep -F -- '- port: 18883' <<< "$rendered_local_mqtts_proxy" >/dev/null || fail \
+ "MQTTS public Service port was not configurable"
+grep -F -- 'targetPort: 8883' <<< "$rendered_local_mqtts_proxy" >/dev/null || fail \
+ "MQTTS Service does not target HAProxy's TLS listener"
+grep -F -- 'containerPort: 8883' <<< "$rendered_local_mqtts_proxy" >/dev/null || fail \
+ "MQTTS-enabled proxy does not declare its TLS listener"
+if grep -F -- 'port: 1883' <<< "$rendered_local_mqtts_proxy" >/dev/null; then
+ fail "MQTTS opt-in also exposed plaintext MQTT"
+fi
+
+rendered_eks_proxy="$(helm template proxy "$PROXY_CHART" \
+ --namespace stack-eks \
+ --values "$HAPROXY_PROXY_VALUES" \
+ --values "$EKS_HAPROXY_PROXY_VALUES" \
+ --set dns.enabled=true \
+ --set-string dns.hostname=stack-eks.example.com)"
+grep -F -- 'storageClassName: "openremote-ebs"' <<< "$rendered_eks_proxy" >/dev/null || fail \
+ "EKS HAProxy certificate state does not use openremote-ebs"
+grep -F -- 'service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip' \
+ <<< "$rendered_eks_proxy" >/dev/null || fail \
+ "EKS HAProxy Service does not request an IP-target NLB"
+if grep -F -- 'aws-load-balancer-internal' <<< "$rendered_eks_proxy" >/dev/null; then
+ fail "EKS HAProxy unexpectedly creates a second metrics LoadBalancer"
+fi
+[ "$(grep -c 'external-dns.kubernetes.io/hostname: "stack-eks.example.com"' <<< "$rendered_eks_proxy")" -eq 1 ] || fail \
+ "HAProxy Service DNS intent was not declared exactly once"
+[ "$(grep -c 'openremote.io/managed-dns: "true"' <<< "$rendered_eks_proxy")" -eq 1 ] || fail \
+ "HAProxy Service DNS selection label was not declared exactly once"
+grep -A15 '^kind: Service$' <<< "$rendered_eks_proxy" | \
+ grep -F -- 'openremote.io/managed-dns: "true"' >/dev/null || fail \
+ "HAProxy Service DNS intent is missing its controller selection label"
+
+rendered_legacy_eks_proxy_pvc="$(helm template proxy "$PROXY_CHART" \
+ --values "$LEGACY_EKS_PROXY_VALUES" --show-only templates/pvc.yaml)"
+grep -F -- 'storageClassName: "openremote-ebs"' <<< "$rendered_legacy_eks_proxy_pvc" >/dev/null || fail \
+ "Legacy EKS HAProxy certificate state does not use openremote-ebs"
+grep -F -- 'helm.sh/resource-policy: keep' <<< "$rendered_legacy_eks_proxy_pvc" >/dev/null || fail \
+ "Legacy EKS HAProxy certificate state is not retained on Helm uninstall"
+
+rendered_external_cert="$(helm template proxy "$PROXY_CHART" \
+ --namespace stack-cert \
+ --set persistence.enabled=false \
+ --set-string certificate.existingSecret=stack-certificate)"
+grep -F -- 'secretName: stack-certificate' <<< "$rendered_external_cert" >/dev/null || fail \
+ "HAProxy chart did not mount an externally managed certificate Secret"
+grep -F -- 'mountPath: /data/proxy/certs' <<< "$rendered_external_cert" >/dev/null || fail \
+ "HAProxy chart mounted external certificates at the wrong image path"
+grep -A1 -F -- '- name: DISABLE_ACME' <<< "$rendered_external_cert" | \
+ grep -F -- 'value: "true"' >/dev/null || fail \
+ "HAProxy still enables ACME when an external certificate Secret is used"
+if grep -F -- 'kind: PersistentVolumeClaim' <<< "$rendered_external_cert" >/dev/null; then
+ fail "Disabled HAProxy persistence still rendered a PVC"
+fi
+
+rendered_without_policy="$(helm template or-setup "$OR_SETUP_CHART" \
+ --namespace stack-disabled)"
+if grep -F -- 'kind: NetworkPolicy' <<< "$rendered_without_policy" >/dev/null; then
+ fail "Disabled stack NetworkPolicy was still rendered"
+fi
+
+rendered_with_ingress_peer="$(helm template or-setup "$OR_SETUP_CHART" \
+ --namespace stack-ingress \
+ --set networkPolicy.enabled=true \
+ --set 'networkPolicy.additionalIngressFrom[0].namespaceSelector.matchLabels.kubernetes\.io/metadata\.name=ingress-nginx')"
+grep -F -- 'kubernetes.io/metadata.name: ingress-nginx' \
+ <<< "$rendered_with_ingress_peer" >/dev/null || fail \
+ "Additional trusted ingress namespace was not rendered"
+
+printf 'or-setup tests passed\n'
diff --git a/kubernetes/test/or-stack-test b/kubernetes/test/or-stack-test
new file mode 100755
index 000000000..91b633327
--- /dev/null
+++ b/kubernetes/test/or-stack-test
@@ -0,0 +1,788 @@
+#!/usr/bin/env bash
+
+set -Eeuo pipefail
+
+TEST_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+KUBERNETES_DIR="$(cd "$TEST_DIR/.." && pwd)"
+OR_STACK="$KUBERNETES_DIR/or-stack"
+TEST_BIN="$TEST_DIR/command-stubs"
+TEST_TEMP_DIR="$(mktemp -d "${TMPDIR:-/tmp}/or-stack-test.XXXXXX")"
+CALL_LOG="$TEST_TEMP_DIR/calls.log"
+TEST_CERTIFICATE_ARN="arn:aws:acm:eu-west-1:123456789012:certificate/test-certificate"
+TEST_MQTTS_CERTIFICATE_ARN="arn:aws:acm:eu-west-1:123456789012:certificate/test-mqtts-certificate"
+
+cleanup() {
+ rm -rf "$TEST_TEMP_DIR"
+}
+
+trap cleanup EXIT
+
+fail() {
+ printf 'FAIL: %s\n' "$*" >&2
+ exit 1
+}
+
+assert_contains() {
+ local expected="$1"
+ grep -F -- "$expected" "$CALL_LOG" >/dev/null || fail \
+ "Expected command log to contain: $expected"
+}
+
+assert_not_contains() {
+ local unexpected="$1"
+ if grep -F -- "$unexpected" "$CALL_LOG" >/dev/null; then
+ fail "Expected command log not to contain: $unexpected"
+ fi
+}
+
+assert_before() {
+ local first="$1"
+ local second="$2"
+ local first_line
+ local second_line
+
+ first_line="$(grep -nF -- "$first" "$CALL_LOG" | head -n 1 | cut -d: -f1)"
+ second_line="$(grep -nF -- "$second" "$CALL_LOG" | head -n 1 | cut -d: -f1)"
+ [ -n "$first_line" ] || fail "Expected command log to contain: $first"
+ [ -n "$second_line" ] || fail "Expected command log to contain: $second"
+ [ "$first_line" -lt "$second_line" ] || fail \
+ "Expected '$first' before '$second'"
+}
+
+assert_on_line_before() {
+ local first="$1"
+ local second="$2"
+ local line
+
+ line="$(grep -F -- "$first" "$CALL_LOG" | grep -F -- "$second" | head -n 1)"
+ [ -n "$line" ] || fail "Expected one command containing '$first' and '$second'"
+ case "$line" in
+ *"$first"*"$second"*)
+ ;;
+ *)
+ fail "Expected '$first' before '$second' in the same command"
+ ;;
+ esac
+}
+
+reset_log() {
+ : > "$CALL_LOG"
+}
+
+run_stack() {
+ PATH="$TEST_BIN:$PATH" \
+ OR_STACK_TEST_LOG="$CALL_LOG" \
+ OR_STACK_TEST_CONTEXT="test-context" \
+ "$OR_STACK" "$@"
+}
+
+test_prepare() {
+ local output
+
+ reset_log
+ output="$(OR_STACK_TEST_NAMESPACE_EXISTS=false run_stack prepare \
+ --name stack-prepared \
+ --hostname stack-prepared.example.com \
+ --kube-context test-context \
+ --target eks \
+ --exposure ingress \
+ --dns external-dns)"
+
+ assert_contains $'kubectl\t--context\ttest-context\tcreate\tnamespace\tstack-prepared'
+ assert_contains $'kubectl\t--context\ttest-context\tlabel\tnamespace\tstack-prepared'
+ assert_contains $'openremote.io/stack=stack-prepared'
+ assert_contains $'openremote.io/target=eks'
+ assert_contains $'openremote.io/exposure=ingress'
+ assert_contains $'openremote.io/dns=external-dns'
+ assert_not_contains $'helm\t'
+ assert_not_contains $'get\tstorageclass'
+ assert_not_contains $'get\tservice\tkube-dns'
+ grep -Fx -- '[or-stack] No Helm releases or workloads were installed.' \
+ <<< "$output" >/dev/null || fail "Prepare did not report its namespace-only behavior"
+
+ reset_log
+ OR_STACK_TEST_NAMESPACE_EXISTS=true \
+ OR_STACK_TEST_TARGET_LABEL=eks \
+ OR_STACK_TEST_EXPOSURE_LABEL=ingress \
+ run_stack prepare \
+ --name stack-prepared \
+ --hostname stack-prepared.example.com \
+ --kube-context test-context \
+ --target eks \
+ --exposure ingress \
+ --dns external-dns >/dev/null
+
+ assert_not_contains $'create\tnamespace\tstack-prepared'
+ assert_contains $'label\tnamespace\tstack-prepared'
+ assert_not_contains $'helm\t'
+
+ reset_log
+ OR_STACK_TEST_NAMESPACE_EXISTS=false run_stack prepare \
+ --name stack-prepared-mqtts \
+ --hostname stack-prepared-mqtts.example.com \
+ --kube-context test-context \
+ --target eks \
+ --exposure ingress \
+ --dns external-dns \
+ --mqtts \
+ --mqtts-hostname mqtt-stack-prepared.example.com >/dev/null
+
+ assert_contains $'openremote.io/reserved-hostname=stack-prepared-mqtts.example.com'
+ assert_contains $'openremote.io/reserved-mqtts-hostname=mqtt-stack-prepared.example.com'
+ assert_contains $'openremote.io/mqtts-hostname=mqtt-stack-prepared.example.com'
+ assert_not_contains $'helm\t'
+}
+
+test_local_apply() {
+ local output
+ local values_dir="$TEST_TEMP_DIR/values"
+ mkdir -p "$values_dir"
+ : > "$values_dir/or-setup.yaml"
+ : > "$values_dir/manager.yaml"
+ : > "$values_dir/proxy.yaml"
+
+ reset_log
+ output="$(OR_STACK_TEST_NAMESPACE_EXISTS=false run_stack apply \
+ --name stack-a \
+ --kube-context test-context \
+ --target local \
+ --hostname stack-a.localhost \
+ --http-port 8080 \
+ --https-port 8443 \
+ --mqtts \
+ --mqtts-port 18883 \
+ --values-dir "$values_dir")"
+
+ assert_contains $'kubectl\t--context\ttest-context\tcreate\tnamespace\tstack-a'
+ assert_contains $'kubectl\t--context\ttest-context\tlabel\tnamespace\tstack-a'
+ assert_contains $'openremote.io/target=local'
+ assert_contains $'openremote.io/exposure=haproxy'
+ assert_contains $'openremote.io/dns=none'
+ assert_contains $'kubectl\t--context\ttest-context\t--namespace\tkube-system\tget\tservice\tkube-dns'
+ assert_contains $'helm\tupgrade\t--install\tor-setup'
+ assert_contains "$KUBERNETES_DIR/stack/targets/local/or-setup.yaml"
+ assert_contains "$KUBERNETES_DIR/stack/exposures/haproxy/or-setup.yaml"
+ assert_contains "$values_dir/or-setup.yaml"
+ assert_contains $'--set\tnetworkPolicy.proxy.mqtts.enabled=true'
+ assert_contains $'--set-string\tnetworkPolicy.proxy.mqtts.port=8883'
+ assert_on_line_before "$KUBERNETES_DIR/stack/targets/local/or-setup.yaml" "$KUBERNETES_DIR/stack/exposures/haproxy/or-setup.yaml"
+ assert_on_line_before "$KUBERNETES_DIR/stack/exposures/haproxy/or-setup.yaml" "$values_dir/or-setup.yaml"
+ assert_contains $'helm\tupgrade\t--install\tpostgresql'
+ assert_contains "$KUBERNETES_DIR/stack/targets/local/postgresql.yaml"
+ assert_contains $'helm\tupgrade\t--install\tkeycloak'
+ assert_on_line_before $'helm\tupgrade\t--install\tkeycloak' $'--set-string\tor.hostname=https://stack-a.localhost:8443/auth'
+ assert_contains $'--set-string\tor.hostname=stack-a.localhost'
+ assert_not_contains $'--set-string\tingress.hostname=stack-a.localhost'
+ assert_not_contains $'--set\tdns.enabled=true'
+ assert_not_contains 'certificate-arn='
+ assert_contains $'helm\tupgrade\t--install\tmanager'
+ assert_contains "$values_dir/manager.yaml"
+ assert_contains $'helm\tupgrade\t--install\tproxy'
+ assert_contains "$KUBERNETES_DIR/stack/exposures/haproxy/proxy.yaml"
+ assert_contains "$KUBERNETES_DIR/stack/exposures/haproxy/local/proxy.yaml"
+ assert_contains "$values_dir/proxy.yaml"
+ assert_on_line_before "$KUBERNETES_DIR/stack/exposures/haproxy/local/proxy.yaml" "$values_dir/proxy.yaml"
+ assert_contains $'--set-string\tor.managerHost=manager.stack-a.svc.cluster.local'
+ assert_contains $'--set-string\tor.keycloakHost=keycloak.stack-a.svc.cluster.local'
+ assert_contains $'--set-string\tor.nameserver=10.96.0.10:53'
+ assert_contains $'--set-string\tor.httpsForwardedPort=8443'
+ assert_contains $'--set-string\tservice.http.port=8080'
+ assert_contains $'--set-string\tservice.https.port=8443'
+ assert_contains $'--set\tservice.mqtts.enabled=true'
+ assert_contains $'--set-string\tservice.mqtts.port=18883'
+ assert_contains $'--set-string\tor.mqttPort=8883'
+ assert_not_contains "openremote-ebs"
+ assert_not_contains "policyendpoints.networking.k8s.aws"
+ grep -F -- "credentials --name stack-a --kube-context test-context" <<< "$output" >/dev/null || fail \
+ "Apply output did not explain how to retrieve Manager credentials"
+ grep -F -- "https://stack-a.localhost:8443/manager" <<< "$output" >/dev/null || fail \
+ "Apply output did not use the configured HAProxy HTTPS port"
+ grep -F -- "mqtts://stack-a.localhost:18883" <<< "$output" >/dev/null || fail \
+ "Apply output did not report the configured MQTTS endpoint"
+}
+
+test_local_public_origins() {
+ local https_port
+ local origin
+
+ for https_port in 8443 9443 443; do
+ origin="https://localhost"
+ if [ "$https_port" != 443 ]; then
+ origin="$origin:$https_port"
+ fi
+
+ reset_log
+ OR_STACK_TEST_NAMESPACE_EXISTS=false run_stack apply \
+ --name stack-local-origin \
+ --kube-context test-context \
+ --target local \
+ --https-port "$https_port" >/dev/null
+
+ assert_on_line_before $'helm\tupgrade\t--install\tmanager' $'--set-string\tor.webserver.allowedOrigins='"$origin"$'\t'
+ assert_on_line_before $'helm\tupgrade\t--install\tmanager' $'--set-string\tor.hostname=localhost\t'
+ assert_on_line_before $'helm\tupgrade\t--install\tkeycloak' $'--set-string\tor.hostname='"$origin/auth"$'\t'
+ done
+}
+
+test_local_ingress_apply() {
+ local output
+
+ reset_log
+ output="$(OR_STACK_TEST_NAMESPACE_EXISTS=false run_stack apply \
+ --name stack-local-ingress \
+ --kube-context test-context \
+ --target local \
+ --exposure ingress \
+ --hostname stack-local-ingress.example.com)"
+
+ assert_contains "$KUBERNETES_DIR/stack/targets/local/or-setup.yaml"
+ assert_contains "$KUBERNETES_DIR/stack/exposures/ingress/or-setup.yaml"
+ assert_contains "$KUBERNETES_DIR/stack/exposures/ingress/proxy.yaml"
+ assert_contains $'--set-string\tingress.hostname=stack-local-ingress.example.com'
+ assert_on_line_before $'helm\tupgrade\t--install\tmanager' $'--set-string\tor.webserver.allowedOrigins=https://stack-local-ingress.example.com\t'
+ assert_on_line_before $'helm\tupgrade\t--install\tkeycloak' $'--set-string\tor.hostname=https://stack-local-ingress.example.com/auth'
+ assert_contains $'--set-string\tingress.tls[0].hosts[0]=stack-local-ingress.example.com'
+ assert_not_contains 'certificate-arn='
+ assert_contains $'get\tservice\tkube-dns'
+ assert_contains $'helm\tupgrade\t--install\tproxy'
+ assert_contains $'--set-string\tor.managerHost=manager.stack-local-ingress.svc.cluster.local'
+ assert_contains $'--set-string\tor.keycloakHost=keycloak.stack-local-ingress.svc.cluster.local'
+ if grep -F -- "shared OpenRemote ALB" <<< "$output" >/dev/null; then
+ fail "Local ingress output claimed to use the EKS shared ALB"
+ fi
+}
+
+test_eks_apply() {
+ reset_log
+ OR_STACK_TEST_NAMESPACE_EXISTS=true \
+ OR_STACK_TEST_TARGET_LABEL=eks \
+ OR_STACK_TEST_EXPOSURE_LABEL=ingress \
+ run_stack apply \
+ --name stack-eks \
+ --kube-context test-context \
+ --target eks \
+ --exposure ingress \
+ --dns external-dns \
+ --hostname stack-eks.example.com \
+ --certificate-arn "$TEST_CERTIFICATE_ARN"
+
+ assert_contains $'kubectl\t--context\ttest-context\tget\tstorageclass\topenremote-ebs'
+ assert_contains $'kubectl\t--context\ttest-context\tget\tcsidriver\tebs.csi.aws.com'
+ assert_contains $'kubectl\t--context\ttest-context\tget\tcustomresourcedefinition/policyendpoints.networking.k8s.aws'
+ assert_contains $'kubectl\t--context\ttest-context\t--namespace\tkube-system\tget\tdeployment\taws-load-balancer-controller'
+ assert_contains $'kubectl\t--context\ttest-context\tget\tingressclass\topenremote-alb'
+ assert_contains $'kubectl\t--context\ttest-context\tget\tingressclassparams.elbv2.k8s.aws\topenremote-alb'
+ assert_contains $'kubectl\t--context\ttest-context\t--namespace\tkube-system\tget\tdaemonset\taws-node'
+ assert_not_contains $'create\tnamespace\tstack-eks'
+ assert_contains "$KUBERNETES_DIR/stack/targets/eks/postgresql.yaml"
+ assert_contains "$KUBERNETES_DIR/stack/targets/eks/or-setup.yaml"
+ assert_contains "$KUBERNETES_DIR/stack/targets/eks/manager.yaml"
+ assert_contains "$KUBERNETES_DIR/stack/exposures/ingress/or-setup.yaml"
+ assert_contains "$KUBERNETES_DIR/stack/exposures/ingress/proxy.yaml"
+ assert_contains "$KUBERNETES_DIR/stack/exposures/ingress/eks/proxy.yaml"
+ assert_contains $'--set-string\tingress.hostname=stack-eks.example.com'
+ assert_on_line_before $'helm\tupgrade\t--install\tmanager' $'--set-string\tor.webserver.allowedOrigins=https://stack-eks.example.com\t'
+ assert_on_line_before $'helm\tupgrade\t--install\tkeycloak' $'--set-string\tor.hostname=https://stack-eks.example.com/auth'
+ assert_contains $'openremote.io/dns=external-dns'
+ assert_contains $'--set\tdns.enabled=true'
+ assert_contains $'--set-string\tdns.hostname=stack-eks.example.com'
+ assert_contains "certificate-arn=$TEST_CERTIFICATE_ARN"
+ assert_contains $'helm\tupgrade\t--install\tproxy'
+ assert_contains $'--set-string\tor.managerHost=manager.stack-eks.svc.cluster.local'
+ assert_contains $'--set-string\tor.keycloakHost=keycloak.stack-eks.svc.cluster.local'
+}
+
+test_eks_ingress_mqtts_apply() {
+ local output
+
+ reset_log
+ output="$(OR_STACK_TEST_NAMESPACE_EXISTS=false run_stack apply \
+ --name stack-eks-mqtts \
+ --kube-context test-context \
+ --target eks \
+ --exposure ingress \
+ --dns external-dns \
+ --hostname stack-eks-mqtts.example.com \
+ --certificate-arn "$TEST_CERTIFICATE_ARN" \
+ --mqtts \
+ --mqtts-hostname mqtt-stack-eks-mqtts.example.com \
+ --mqtts-certificate-arn "$TEST_MQTTS_CERTIFICATE_ARN" \
+ --mqtts-port 18883)"
+
+ assert_contains $'openremote.io/reserved-hostname=stack-eks-mqtts.example.com'
+ assert_contains $'openremote.io/reserved-mqtts-hostname=mqtt-stack-eks-mqtts.example.com'
+ assert_contains $'openremote.io/mqtts-hostname=mqtt-stack-eks-mqtts.example.com'
+ assert_contains $'--set\tnetworkPolicy.proxy.mqtts.enabled=true'
+ assert_contains $'--set-string\tnetworkPolicy.proxy.mqtts.port=1883'
+ assert_contains $'--set\tservice.mqtts.enabled=true'
+ assert_contains $'--set-string\tor.mqttPort=1883'
+ assert_contains $'--set-string\tservice.mqtts.port=18883'
+ assert_contains "aws-load-balancer-ssl-cert=$TEST_MQTTS_CERTIFICATE_ARN"
+ assert_contains $'--set\tservice.mqtts.dns.enabled=true'
+ assert_contains $'--set-string\tservice.mqtts.dns.hostname=mqtt-stack-eks-mqtts.example.com'
+ grep -F -- 'mqtts://mqtt-stack-eks-mqtts.example.com:18883' <<< "$output" >/dev/null || fail \
+ "Ingress apply output did not report its dedicated MQTTS endpoint"
+ grep -F -- 'get service proxy-mqtts' <<< "$output" >/dev/null || fail \
+ "Ingress apply output did not identify the dedicated MQTTS Service"
+}
+
+test_eks_haproxy_apply() {
+ reset_log
+ OR_STACK_TEST_NAMESPACE_EXISTS=false run_stack apply \
+ --name stack-eks-proxy \
+ --kube-context test-context \
+ --target eks \
+ --dns external-dns \
+ --hostname stack-eks-proxy.example.com
+
+ assert_contains $'kubectl\t--context\ttest-context\t--namespace\tkube-system\tget\tdeployment\taws-load-balancer-controller'
+ assert_not_contains $'get\tingressclass\topenremote-alb'
+ assert_not_contains 'certificate-arn='
+ assert_contains $'helm\tupgrade\t--install\tproxy'
+ assert_contains "$KUBERNETES_DIR/stack/exposures/haproxy/eks/proxy.yaml"
+ assert_contains $'--set-string\tor.hostname=stack-eks-proxy.example.com'
+ assert_on_line_before $'helm\tupgrade\t--install\tkeycloak' $'--set-string\tor.hostname=https://stack-eks-proxy.example.com/auth'
+ assert_contains $'--set\tdns.enabled=true'
+ assert_contains $'--set-string\tdns.hostname=stack-eks-proxy.example.com'
+ assert_contains $'--set-string\tor.managerHost=manager.stack-eks-proxy.svc.cluster.local'
+ assert_not_contains $'--set\tservice.mqtts.enabled=true'
+}
+
+test_eks_none_apply() {
+ reset_log
+ OR_STACK_TEST_NAMESPACE_EXISTS=false \
+ OR_STACK_TEST_EKS_LOAD_BALANCER_CONTROLLER=false \
+ run_stack apply \
+ --name stack-eks-none \
+ --kube-context test-context \
+ --target eks \
+ --exposure none
+
+ assert_contains "$KUBERNETES_DIR/stack/exposures/none/or-setup.yaml"
+ assert_not_contains 'or.webserver.allowedOrigins='
+ assert_not_contains $'--set-string\tor.hostname=https://'
+ assert_not_contains $'get\tdeployment\taws-load-balancer-controller'
+ assert_not_contains $'get\tingressclass\topenremote-alb'
+ assert_not_contains $'helm\tupgrade\t--install\tproxy'
+ assert_not_contains $'--set-string\tingress.hostname='
+}
+
+test_status() {
+ local output
+
+ reset_log
+ output="$(OR_STACK_TEST_NAMESPACE_EXISTS=true \
+ OR_STACK_TEST_PROXY_MQTTS_PORT=8883 \
+ OR_STACK_TEST_HOSTNAME=stack-a.example.com \
+ run_stack status \
+ --name stack-a \
+ --kube-context test-context)"
+
+ grep -Fx -- "Target: local" <<< "$output" >/dev/null || fail \
+ "Status did not show the stored target"
+ grep -Fx -- "Exposure: haproxy" <<< "$output" >/dev/null || fail \
+ "Status did not show the stored exposure"
+ grep -Fx -- "DNS: none" <<< "$output" >/dev/null || fail \
+ "Status did not show the stored DNS mode"
+ grep -Fx -- "MQTTS endpoint: mqtts://stack-a.example.com:8883" <<< "$output" >/dev/null || fail \
+ "Status did not show the active MQTTS endpoint"
+ assert_contains $'helm\tlist\t--kube-context\ttest-context\t--namespace\tstack-a'
+ assert_contains $'kubectl\t--context\ttest-context\t--namespace\tstack-a\tget\tdeployments,statefulsets,pods'
+ assert_contains $'kubectl\t--context\ttest-context\t--namespace\tstack-a\tget\tservices,ingresses,networkpolicies'
+ assert_contains $'kubectl\t--context\ttest-context\t--namespace\tstack-a\tget\tpersistentvolumeclaims'
+ assert_contains $'kubectl\t--context\ttest-context\t--namespace\tstack-a\tget\tservice\tproxy'
+ assert_contains $'kubectl\t--context\ttest-context\t--namespace\tstack-a\texec\tdeployment/proxy'
+
+ reset_log
+ output="$(OR_STACK_TEST_NAMESPACE_EXISTS=true \
+ OR_STACK_TEST_EXPOSURE_LABEL=ingress \
+ OR_STACK_TEST_PROXY_MQTTS_PORT=8883 \
+ OR_STACK_TEST_MQTTS_HOSTNAME=mqtt-stack-a.example.com \
+ run_stack status \
+ --name stack-a \
+ --kube-context test-context)"
+
+ grep -Fx -- "MQTTS endpoint: mqtts://mqtt-stack-a.example.com:8883" <<< "$output" >/dev/null || fail \
+ "Ingress status did not show the dedicated MQTTS endpoint"
+ assert_contains $'kubectl\t--context\ttest-context\t--namespace\tstack-a\tget\tservice\tproxy-mqtts'
+ assert_not_contains $'exec\tdeployment/proxy'
+}
+
+test_uninstall() {
+ reset_log
+ OR_STACK_TEST_NAMESPACE_EXISTS=true run_stack uninstall \
+ --name stack-a \
+ --kube-context test-context
+
+ assert_contains "metadata.name"
+ assert_contains "helm\\.sh/resource-policy"
+ assert_contains $'helm\tuninstall\tproxy\t--kube-context\ttest-context\t--namespace\tstack-a'
+ assert_contains $'helm\tuninstall\tmanager\t--kube-context\ttest-context\t--namespace\tstack-a'
+ assert_contains $'helm\tuninstall\tkeycloak\t--kube-context\ttest-context\t--namespace\tstack-a'
+ assert_contains $'helm\tuninstall\tpostgresql\t--kube-context\ttest-context\t--namespace\tstack-a'
+ assert_contains $'helm\tuninstall\tor-setup\t--kube-context\ttest-context\t--namespace\tstack-a'
+ assert_before $'helm\tuninstall\tproxy' $'helm\tuninstall\tmanager'
+ assert_before $'helm\tuninstall\tmanager' $'helm\tuninstall\tkeycloak'
+ assert_before $'helm\tuninstall\tkeycloak' $'helm\tuninstall\tpostgresql'
+ assert_before $'helm\tuninstall\tpostgresql' $'helm\tuninstall\tor-setup'
+ assert_not_contains $'kubectl\t--context\ttest-context\tdelete'
+}
+
+test_credentials() {
+ local output
+
+ reset_log
+ output="$(OR_STACK_TEST_NAMESPACE_EXISTS=true \
+ OR_STACK_TEST_ADMIN_PASSWORD=stack-a-password \
+ run_stack credentials --name stack-a --kube-context test-context)"
+
+ grep -Fx -- "Manager username: admin" <<< "$output" >/dev/null || fail \
+ "Credentials did not show the Manager username"
+ grep -Fx -- "Manager password: stack-a-password" <<< "$output" >/dev/null || fail \
+ "Credentials did not show the decoded Manager password"
+ assert_contains $'kubectl\t--context\ttest-context\t--namespace\tstack-a\tget\tsecret\topenremote-secret'
+ assert_not_contains "postgresql-password"
+}
+
+test_destroy() {
+ reset_log
+ OR_STACK_TEST_NAMESPACE_EXISTS=true run_stack destroy \
+ --name stack-a \
+ --kube-context test-context \
+ --confirm stack-a
+
+ assert_contains "spec.volumeName"
+ assert_contains $'kubectl\t--context\ttest-context\tdelete\tnamespace\tstack-a'
+ assert_contains $'kubectl\t--context\ttest-context\twait\t--for=delete\tpersistentvolume/pvc-manager'
+ assert_contains $'kubectl\t--context\ttest-context\twait\t--for=delete\tpersistentvolume/pvc-postgresql'
+ assert_contains $'kubectl\t--context\ttest-context\twait\t--for=delete\tpersistentvolume/pvc-proxy'
+ assert_before $'helm\tuninstall\tor-setup' $'kubectl\t--context\ttest-context\tdelete\tnamespace\tstack-a'
+}
+
+test_lifecycle_safety() {
+ reset_log
+ if OR_STACK_TEST_NAMESPACE_EXISTS=true OR_STACK_TEST_NAMESPACE_LABEL= run_stack prepare \
+ --name stack-a --kube-context test-context --target local >/dev/null 2>&1; then
+ fail "Prepare adopted an unlabeled namespace"
+ fi
+ assert_not_contains $'label\tnamespace\tstack-a'
+ assert_not_contains $'helm\t'
+
+ reset_log
+ if OR_STACK_TEST_NAMESPACE_EXISTS=true \
+ OR_STACK_TEST_TARGET_LABEL=eks \
+ OR_STACK_TEST_EXPOSURE_LABEL=haproxy \
+ run_stack prepare --name stack-a --kube-context test-context --target local \
+ >/dev/null 2>&1; then
+ fail "Prepare changed an existing stack's target"
+ fi
+ assert_not_contains $'label\tnamespace\tstack-a'
+
+ reset_log
+ if OR_STACK_TEST_NAMESPACE_EXISTS=true OR_STACK_TEST_NAMESPACE_LABEL= run_stack apply \
+ --name stack-a --kube-context test-context --target local >/dev/null 2>&1; then
+ fail "Apply adopted an unlabeled namespace"
+ fi
+ assert_not_contains $'helm\tupgrade'
+
+ reset_log
+ if OR_STACK_TEST_NAMESPACE_EXISTS=true run_stack destroy \
+ --name stack-a --kube-context test-context --confirm another-stack >/dev/null 2>&1; then
+ fail "Destroy accepted an incorrect confirmation"
+ fi
+ assert_not_contains $'delete\tnamespace'
+
+ reset_log
+ if OR_STACK_TEST_NAMESPACE_EXISTS=true OR_STACK_TEST_NAMESPACE_LABEL= run_stack destroy \
+ --name stack-a --kube-context test-context --confirm stack-a >/dev/null 2>&1; then
+ fail "Destroy accepted an unlabeled namespace"
+ fi
+ assert_not_contains $'helm\tuninstall'
+ assert_not_contains $'delete\tnamespace'
+
+ reset_log
+ if OR_STACK_TEST_NAMESPACE_EXISTS=true OR_STACK_TEST_SECRET_POLICY= run_stack uninstall \
+ --name stack-a --kube-context test-context >/dev/null 2>&1; then
+ fail "Uninstall accepted a Secret without the keep policy"
+ fi
+ assert_not_contains $'helm\tuninstall'
+
+ reset_log
+ if OR_STACK_TEST_NAMESPACE_EXISTS=true \
+ OR_STACK_TEST_PVC_METADATA=$'manager|manager|\npostgresql|postgresql|keep' \
+ run_stack uninstall --name stack-a --kube-context test-context >/dev/null 2>&1; then
+ fail "Uninstall accepted a chart-managed PVC without the keep policy"
+ fi
+ assert_not_contains $'helm\tuninstall'
+
+ reset_log
+ if OR_STACK_TEST_NAMESPACE_EXISTS=true OR_STACK_TEST_PV_WAIT_SUCCESS=false run_stack destroy \
+ --name stack-a --kube-context test-context --confirm stack-a >/dev/null 2>&1; then
+ fail "Destroy reported success while PersistentVolume deletion failed"
+ fi
+ assert_contains $'delete\tnamespace\tstack-a'
+
+ reset_log
+ if OR_STACK_TEST_NAMESPACE_EXISTS=true OR_STACK_TEST_SECRET_EXISTS=false run_stack credentials \
+ --name stack-a --kube-context test-context >/dev/null 2>&1; then
+ fail "Credentials succeeded without the stack Secret"
+ fi
+
+ reset_log
+ if OR_STACK_TEST_NAMESPACE_EXISTS=true OR_STACK_TEST_NAMESPACE_LABEL= run_stack credentials \
+ --name stack-a --kube-context test-context >/dev/null 2>&1; then
+ fail "Credentials accepted an unlabeled namespace"
+ fi
+ assert_not_contains $'get\tsecret\topenremote-secret'
+
+ reset_log
+ if OR_STACK_TEST_NAMESPACE_EXISTS=true \
+ OR_STACK_TEST_TARGET_LABEL=local \
+ OR_STACK_TEST_EXPOSURE_LABEL=ingress \
+ run_stack apply --name stack-a --kube-context test-context --target local \
+ >/dev/null 2>&1; then
+ fail "Apply changed an existing stack's exposure mode"
+ fi
+ assert_not_contains $'label\tnamespace\tstack-a'
+ assert_not_contains $'helm\tupgrade'
+
+ reset_log
+ if OR_STACK_TEST_NAMESPACE_EXISTS=true \
+ OR_STACK_TEST_TARGET_LABEL=eks \
+ OR_STACK_TEST_EXPOSURE_LABEL=haproxy \
+ run_stack apply --name stack-a --kube-context test-context --target local \
+ >/dev/null 2>&1; then
+ fail "Apply changed an existing stack's target"
+ fi
+ assert_not_contains $'label\tnamespace\tstack-a'
+ assert_not_contains $'helm\tupgrade'
+
+ reset_log
+ if OR_STACK_TEST_NAMESPACE_EXISTS=true \
+ OR_STACK_TEST_TARGET_LABEL= \
+ OR_STACK_TEST_EXPOSURE_LABEL=haproxy \
+ run_stack apply --name stack-a --kube-context test-context --target eks \
+ --hostname stack-a.example.com >/dev/null 2>&1; then
+ fail "Apply accepted a stack with missing target metadata"
+ fi
+ assert_not_contains $'label\tnamespace\tstack-a'
+ assert_not_contains $'helm\tupgrade'
+
+ reset_log
+ if OR_STACK_TEST_NAMESPACE_EXISTS=true \
+ OR_STACK_TEST_TARGET_LABEL=eks \
+ OR_STACK_TEST_EXPOSURE_LABEL= \
+ run_stack apply --name stack-a --kube-context test-context --target eks \
+ --exposure ingress --hostname stack-a.example.com \
+ --certificate-arn "$TEST_CERTIFICATE_ARN" >/dev/null 2>&1; then
+ fail "Apply accepted a stack with missing exposure metadata"
+ fi
+ assert_not_contains $'label\tnamespace\tstack-a'
+ assert_not_contains $'helm\tupgrade'
+}
+
+test_validation() {
+ if run_stack prepare --name stack-a --kube-context test-context >/dev/null 2>&1; then
+ fail "Prepare accepted a missing target"
+ fi
+
+ if run_stack prepare --name stack-a --kube-context test-context --target local \
+ --exposure none --dns external-dns >/dev/null 2>&1; then
+ fail "Prepare accepted ExternalDNS ownership without a public endpoint"
+ fi
+
+ if run_stack apply --name default --kube-context test-context --target local >/dev/null 2>&1; then
+ fail "Reserved namespace was accepted"
+ fi
+
+ if run_stack apply --name Stack_A --kube-context test-context --target local >/dev/null 2>&1; then
+ fail "Invalid namespace name was accepted"
+ fi
+
+ if run_stack apply --name stack-a --kube-context test-context --target unknown >/dev/null 2>&1; then
+ fail "Unknown target was accepted"
+ fi
+
+ if run_stack apply --name stack-a --kube-context test-context --target local \
+ --exposure unknown >/dev/null 2>&1; then
+ fail "Unknown exposure was accepted"
+ fi
+
+ if run_stack apply --name stack-a --kube-context test-context --target local \
+ --dns unknown >/dev/null 2>&1; then
+ fail "Unknown DNS mode was accepted"
+ fi
+
+ if run_stack apply --name stack-a --kube-context test-context --target local \
+ --exposure none --dns external-dns --hostname stack-a.example.com >/dev/null 2>&1; then
+ fail "ExternalDNS ownership was accepted without a public endpoint"
+ fi
+
+ if run_stack apply --name stack-a --kube-context test-context --target local \
+ --dns external-dns >/dev/null 2>&1; then
+ fail "ExternalDNS ownership accepted the default localhost hostname"
+ fi
+
+ if run_stack apply --name stack-a --kube-context test-context --target local \
+ --hostname 'invalid host' >/dev/null 2>&1; then
+ fail "Invalid hostname was accepted"
+ fi
+
+ if OR_STACK_TEST_EKS_STORAGE=false run_stack apply \
+ --name stack-a --kube-context test-context --target eks \
+ --hostname stack-a.example.com >/dev/null 2>&1; then
+ fail "EKS target accepted a missing StorageClass"
+ fi
+
+ if OR_STACK_TEST_EKS_NETWORK_POLICY=false run_stack apply \
+ --name stack-a --kube-context test-context --target eks \
+ --hostname stack-a.example.com >/dev/null 2>&1; then
+ fail "EKS target accepted disabled NetworkPolicy enforcement"
+ fi
+
+ if OR_STACK_TEST_EKS_CNI_CONTAINERS=aws-node run_stack apply \
+ --name stack-a --kube-context test-context --target eks \
+ --hostname stack-a.example.com >/dev/null 2>&1; then
+ fail "EKS target accepted a missing NetworkPolicy agent"
+ fi
+
+ if run_stack apply --name stack-a --kube-context test-context --target eks \
+ >/dev/null 2>&1; then
+ fail "EKS target accepted a missing public hostname"
+ fi
+
+ if run_stack apply --name stack-a --kube-context test-context --target eks \
+ --exposure ingress --hostname stack-a.example.com >/dev/null 2>&1; then
+ fail "EKS ingress exposure accepted a missing ACM certificate ARN"
+ fi
+
+ if run_stack apply --name stack-a --kube-context test-context --target eks \
+ --exposure ingress --hostname stack-a.example.com \
+ --certificate-arn not-an-arn >/dev/null 2>&1; then
+ fail "EKS ingress exposure accepted an invalid ACM certificate ARN"
+ fi
+
+ if run_stack apply --name stack-a --kube-context test-context --target eks \
+ --hostname stack-a.example.com \
+ --certificate-arn "$TEST_CERTIFICATE_ARN" >/dev/null 2>&1; then
+ fail "HAProxy exposure accepted an ACM certificate ARN"
+ fi
+
+ if run_stack apply --name stack-a --kube-context test-context --target local \
+ --exposure none --http-port 8080 >/dev/null 2>&1; then
+ fail "Non-HAProxy exposure accepted an HTTP port"
+ fi
+
+ if run_stack apply --name stack-a --kube-context test-context --target local \
+ --http-port 8443 --https-port 8443 >/dev/null 2>&1; then
+ fail "HAProxy exposure accepted identical HTTP and HTTPS ports"
+ fi
+
+ if run_stack apply --name stack-a --kube-context test-context --target local \
+ --mqtts-port 18883 >/dev/null 2>&1; then
+ fail "A custom MQTTS port was accepted without --mqtts"
+ fi
+
+ if run_stack apply --name stack-a --kube-context test-context --target local \
+ --exposure ingress --hostname stack-a.example.com --mqtts \
+ --mqtts-hostname mqtt-stack-a.example.com \
+ --mqtts-certificate-arn "$TEST_MQTTS_CERTIFICATE_ARN" >/dev/null 2>&1; then
+ fail "Local Ingress exposure accepted EKS-specific MQTTS"
+ fi
+
+ if run_stack apply --name stack-a --kube-context test-context --target local \
+ --exposure none --mqtts >/dev/null 2>&1; then
+ fail "Internal-only exposure accepted MQTTS"
+ fi
+
+ if run_stack apply --name stack-a --kube-context test-context --target local \
+ --mqtts --mqtts-port 443 >/dev/null 2>&1; then
+ fail "HAProxy exposure accepted an MQTTS port that collides with HTTPS"
+ fi
+
+ if run_stack apply --name stack-a --kube-context test-context --target eks \
+ --exposure ingress --hostname stack-a.example.com \
+ --certificate-arn "$TEST_CERTIFICATE_ARN" --mqtts >/dev/null 2>&1; then
+ fail "EKS Ingress MQTTS accepted a missing hostname and certificate"
+ fi
+
+ if run_stack apply --name stack-a --kube-context test-context --target eks \
+ --exposure ingress --hostname stack-a.example.com \
+ --certificate-arn "$TEST_CERTIFICATE_ARN" --mqtts \
+ --mqtts-hostname stack-a.example.com \
+ --mqtts-certificate-arn "$TEST_MQTTS_CERTIFICATE_ARN" >/dev/null 2>&1; then
+ fail "EKS Ingress MQTTS accepted the web hostname"
+ fi
+
+ if run_stack apply --name stack-a --kube-context test-context --target eks \
+ --exposure ingress --hostname stack-a.example.com \
+ --certificate-arn "$TEST_CERTIFICATE_ARN" --mqtts \
+ --mqtts-hostname mqtt-stack-a.example.com \
+ --mqtts-certificate-arn invalid >/dev/null 2>&1; then
+ fail "EKS Ingress MQTTS accepted an invalid ACM certificate ARN"
+ fi
+
+ if run_stack apply --name stack-a --kube-context test-context --target local \
+ --mqtts --mqtts-hostname mqtt-stack-a.example.com >/dev/null 2>&1; then
+ fail "HAProxy MQTTS accepted a separate MQTTS hostname"
+ fi
+
+ if OR_STACK_TEST_KUBE_DNS_EXISTS=false run_stack apply \
+ --name stack-a --kube-context test-context --target local >/dev/null 2>&1; then
+ fail "HAProxy exposure accepted a missing kube-dns Service"
+ fi
+
+ if OR_STACK_TEST_EKS_LOAD_BALANCER_CONTROLLER=false run_stack apply \
+ --name stack-a --kube-context test-context --target eks \
+ --hostname stack-a.example.com >/dev/null 2>&1; then
+ fail "Public EKS exposure accepted a missing AWS Load Balancer Controller"
+ fi
+
+ if OR_STACK_TEST_EKS_INGRESS_CLASS=false run_stack apply \
+ --name stack-a --kube-context test-context --target eks \
+ --exposure ingress --hostname stack-a.example.com \
+ --certificate-arn "$TEST_CERTIFICATE_ARN" >/dev/null 2>&1; then
+ fail "EKS target accepted a missing shared IngressClass"
+ fi
+
+ if OR_STACK_TEST_EKS_INGRESS_CLASS_PARAMS=false run_stack apply \
+ --name stack-a --kube-context test-context --target eks \
+ --exposure ingress --hostname stack-a.example.com \
+ --certificate-arn "$TEST_CERTIFICATE_ARN" >/dev/null 2>&1; then
+ fail "EKS target accepted missing shared IngressClass parameters"
+ fi
+
+ if OR_STACK_TEST_EKS_SSL_REDIRECT_PORT=80 run_stack apply \
+ --name stack-a --kube-context test-context --target eks \
+ --exposure ingress --hostname stack-a.example.com \
+ --certificate-arn "$TEST_CERTIFICATE_ARN" >/dev/null 2>&1; then
+ fail "EKS target accepted an IngressClass without HTTPS redirection"
+ fi
+
+ if OR_STACK_TEST_NAMESPACE_EXISTS=true \
+ OR_STACK_TEST_TARGET_LABEL=eks \
+ OR_STACK_TEST_EXPOSURE_LABEL=haproxy \
+ OR_STACK_TEST_EKS_CNI_CONTAINERS='aws-node aws-network-policy-agent' \
+ run_stack apply --name stack-alternate-agent --kube-context test-context --target eks \
+ --hostname stack-alternate-agent.example.com >/dev/null 2>&1; then
+ fail "EKS target accepted the image name instead of the aws-eks-nodeagent container"
+ fi
+ assert_not_contains $'helm\tupgrade'
+}
+
+test_prepare
+test_local_apply
+test_local_public_origins
+test_local_ingress_apply
+test_eks_apply
+test_eks_ingress_mqtts_apply
+test_eks_haproxy_apply
+test_eks_none_apply
+test_status
+test_uninstall
+test_credentials
+test_destroy
+test_lifecycle_safety
+test_validation
+
+printf 'or-stack tests passed\n'
diff --git a/test/README-eks-load.md b/test/README-eks-load.md
new file mode 100644
index 000000000..e1d08321b
--- /dev/null
+++ b/test/README-eks-load.md
@@ -0,0 +1,329 @@
+# Deploying EKS load-test stacks
+
+`test/or-eks-load` is a Bash helper for deploying a load1 or load2 profile. Save
+your settings in one file, then use `up` to configure DNS access, create or
+reconcile the cluster, and deploy the stack. The default deployment provides
+HTTPS and MQTTS through HAProxy, with a separate namespace and storage per stack.
+
+Run the commands below from the **repository root**.
+
+## Deploy and run a test
+
+### 1. Prepare credentials and the Manager image
+
+Install the tools required by the [EKS deployment guide](../kubernetes/README-AWS.md):
+AWS CLI, eksctl, kubectl, Helm, jq, envsubst, dig, curl, and Python 3. The helper
+itself uses Bash, including macOS's built-in Bash; the underlying EKS CLI uses
+Python for its MQTTS TLS check.
+
+Authenticate to the cluster account using your normal AWS credential chain.
+For pasted temporary credentials, export `AWS_ACCESS_KEY_ID`,
+`AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN`, and **unset** `AWS_PROFILE`
+and `AWS_DEFAULT_PROFILE` if previously set. Do not set a profile to an empty
+string. A named cluster profile can instead be selected in the config below.
+
+Build and push the custom Manager image using the [load1](load1-eks/README.md)
+or [load2](load2-eks/README.md) instructions. Ensure the cluster nodes can pull
+it, including ECR repository permissions for cross-account pulls. The supplied
+clusters use ARM nodes; the image examples publish both ARM64 and AMD64 images.
+The helper does not build images or change registry permissions.
+
+### 2. Save your deployment settings
+
+For load2:
+
+```bash
+mkdir -p .local
+cp test/load2-eks/deployment.env.example .local/loadtest.env
+```
+
+For load1, copy `test/load1-eks/deployment.env.example` instead.
+Edit `.local/loadtest.env` for your deployment. For example:
+
+```bash
+LOAD_TEST=load2
+LOAD_PROFILE=xlarge
+CLUSTER_NAME=loadtest
+STACK_NAME=load2
+AWS_REGION=eu-west-1
+LOAD_HOSTNAME=load2.openremote.app
+LOAD_MANAGER_REPOSITORY=134517981306.dkr.ecr.eu-west-1.amazonaws.com/openremote/manager
+DNS_AWS_PROFILE=134517981306_AWSAdministratorAccess
+```
+
+`LOAD_MANAGER_REPOSITORY` includes `/openremote/manager`, without an image tag;
+the profile selects `load1` or `load2`. `LOAD_HOSTNAME` is a bare hostname,
+without `https://` or a path, and must be unused by another stack or cluster.
+
+Use `DNS_AWS_PROFILE` for the account containing your public Route 53 zone.
+Leave it empty to use the cluster credentials. If using an SSO profile,
+authenticate it before deployment. Set `CLUSTER_AWS_PROFILE` only if you want
+a specific named profile for cluster operations.
+
+You do not normally need a zone ID, role ARN, external ID, or ownership ID.
+The helper discovers the zone and configures a dedicated role for this cluster.
+If multiple public zones match the same domain, it lists their IDs and stops;
+add `DNS_ZONE_ID=...` to choose one explicitly.
+
+The config is **sourced as Bash code**, so use only a file you trust. It needs
+no `export` statements and must not contain access keys. `.local/` is ignored
+by Git. Helper settings come from this file, replacing the old environment
+variable and `--` forwarding interface.
+
+### 3. Deploy
+
+```bash
+test/or-eks-load up --config .local/loadtest.env
+```
+
+This checks your credentials, configures DNS IAM, creates the cluster if absent
+or reconciles its shared add-ons if present, and applies the selected profile.
+It tests the ExternalDNS service account's actual IAM credential chain and
+Route 53 read access before starting the stack deployment. It then waits for
+DNS, trusted HTTPS, and MQTTS, and checks Keycloak's advertised login URL.
+
+Initial dataset creation can take time; the default deployment timeout is
+90 minutes. If credentials expire or a later deployment step fails, refresh
+your credentials, fix the reported issue, and rerun the **same command**.
+Reapplying retains stack data and credentials. Changing the selected profile
+does not resize existing cluster nodes or upgrade Kubernetes.
+
+### 4. Retrieve credentials and run the test
+
+```bash
+test/or-eks-load credentials --config .local/loadtest.env
+test/or-eks-load status --config .local/loadtest.env
+```
+
+Open `https://load2.openremote.app/manager/` (substitute your configured hostname)
+and sign in with the returned administrator credentials. Before running the
+load scenario:
+
+1. Check that the page is styled and login redirects stay on the correct host.
+2. Verify the expected test users and assets, and asset creation/editing.
+3. Verify an authenticated MQTTS publish using a provisioned test account.
+
+Configure the test client with `MANAGER_HOSTNAME` set to `LOAD_HOSTNAME` and
+MQTTS port **8883**. Plaintext MQTT is not exposed. Use the clients documented
+in [load1](load1/README.md); for load2, use its
+[scenarios and parameters](load2-eks/README.md#running-tests).
+
+## Manage the deployment
+
+All helper commands use the same `--config` file.
+
+| Command | What it does |
+| ------------------------------------------ | ------------------------------------------------------------------------------------------------------ |
+| `up` | Reconcile DNS IAM and cluster add-ons, then deploy and verify the stack |
+| `cluster-up` | Configure DNS IAM and the cluster only, then check DNS access |
+| `deploy` | Deploy and verify a stack on an already configured cluster, without changing DNS IAM or shared add-ons |
+| `status` | Show stack status |
+| `credentials` | Retrieve stack credentials |
+| `uninstall` | Stop the stack while retaining data, credentials, and certificate state |
+| `destroy-stack --confirm <stack-name>` | Delete one stack, its data, and its owned external resources |
+| `destroy-cluster --confirm <cluster-name>` | Delete a cluster after all its stacks have been destroyed |
+| `prepare --values-dir <new-directory>` | Write editable profile values locally, without contacting AWS |
+
+Status, credentials, uninstall, and destruction use only cluster credentials.
+`deploy` also uses only cluster credentials: it reads the installed ExternalDNS
+configuration and tests its role access. `up` and `cluster-up` need permissions
+to manage the dedicated IAM CloudFormation stack in the DNS account as well.
+
+### Reuse a cluster or deploy independently
+
+To deploy another stack in an already configured cluster, copy the deployment
+config and choose a new `STACK_NAME` and `LOAD_HOSTNAME`. Keep the same
+`CLUSTER_NAME` and region, check available capacity, then run:
+
+```bash
+test/or-eks-load deploy --config .local/another-stack.env
+```
+
+The existing cluster must have the OpenRemote add-ons and managed ExternalDNS.
+For an independently sized cluster, choose a new `CLUSTER_NAME` too and use
+`up`; the helper derives a separate DNS role and ownership ID.
+
+### Customise component values
+
+The standard profile requires no YAML copying. To change dataset parameters,
+resource limits, or image tags, prepare an editable copy:
+
+```bash
+test/or-eks-load prepare --config .local/loadtest.env --values-dir .local/loadtest-values
+```
+
+This refuses to overwrite an existing directory. Edit the generated component
+files, then add this line to `.local/loadtest.env`:
+
+```bash
+LOAD_VALUES_DIR=loadtest-values
+```
+
+`LOAD_VALUES_DIR` is relative to the **config file**, whereas `prepare`'s
+`--values-dir` is relative to the current directory. Absolute paths also work.
+With this setting, subsequent `up` and `deploy` calls use your files unchanged.
+The image repository has already been rendered; edit `manager.yaml` to change
+it. Remove the setting to return to the checked-in profile.
+
+```bash
+test/or-eks-load deploy --config .local/loadtest.env
+```
+
+An optional `or-setup.yaml` can supply further overrides. Prepared values can
+also be passed directly to `or-eks-stack` or `or-stack` with `--values-dir`.
+
+### Update the Manager image or inspect logs
+
+Prefer a unique image tag per build and update `image.tag` in prepared
+`manager.yaml`, then run `deploy`. If you replace the image under the same tag,
+reapplying unchanged values does not restart Pods. Load the trusted config into
+your shell and explicitly restart Manager:
+
+```bash
+source .local/loadtest.env
+kubectl --context "$CLUSTER_NAME@$AWS_REGION" --namespace "$STACK_NAME" rollout restart deployment/manager
+kubectl --context "$CLUSTER_NAME@$AWS_REGION" --namespace "$STACK_NAME" rollout status deployment/manager --timeout=90m
+```
+
+Use the same context and namespace to inspect setup logs:
+
+```bash
+kubectl --context "$CLUSTER_NAME@$AWS_REGION" --namespace "$STACK_NAME" logs deployment/manager --tail=100
+```
+
+On clean initialization, logs should identify `org.openremote.setup.load1.SetupTasks`
+or `org.openremote.setup.load2.SetupTasks`. Load2 also provides internal JMX:
+
+```bash
+kubectl --context "$CLUSTER_NAME@$AWS_REGION" --namespace "$STACK_NAME" port-forward service/manager-jmx 8085:8085
+```
+
+### Reset the test dataset
+
+Use prepared values and set `or.setupRunOnRestart: true` in `manager.yaml`.
+Run `deploy` and wait for setup to finish, then set it back to `false` and run
+`deploy` again before testing. **While enabled, this setting wipes the stack's
+database on every Manager startup.** A normal reapply does not reset data.
+
+### Delete resources
+
+To stop the stack while retaining its data:
+
+```bash
+test/or-eks-load uninstall --config .local/loadtest.env
+```
+
+To permanently delete the example stack, then its now-empty cluster:
+
+```bash
+test/or-eks-load destroy-stack --config .local/loadtest.env --confirm load2
+test/or-eks-load destroy-cluster --config .local/loadtest.env --confirm loadtest
+```
+
+Use the exact names from your config. Destroy **every** stack before deleting
+a cluster; uninstall alone retains namespaces and is not enough. Cluster
+destruction refuses remaining stacks/endpoints.
+
+The DNS-account IAM CloudFormation stack is retained for reuse. Once the
+cluster is gone and its DNS cleanup has completed, you can separately delete
+`or-load-<cluster-account-id>-<cluster-name>-<region>-dns` in the DNS account.
+
+## Configuration and deployment reference
+
+### Optional settings
+
+| Setting | Default / purpose |
+| ----------------------- | --------------------------------------------------------------------------- |
+| `CLUSTER_AWS_PROFILE` | Empty: use the normal AWS credential chain |
+| `DNS_AWS_PROFILE` | Empty: use cluster credentials |
+| `DNS_ZONE_ID` | Discover the most specific matching public hosted zone; set to disambiguate |
+| `DNS_DOMAIN` | Discovered zone name; set to constrain discovery |
+| `LOAD_VALUES_DIR` | Render the selected profile into a temporary directory |
+| `LOAD_TIMEOUT` | `90m`; increase for longer initialization |
+| `LOAD_EXPOSURE` | `haproxy`; `ingress` also supported |
+| `LOAD_MQTTS_HOSTNAME` | Required for ingress; a separate hostname in the same managed domain |
+| `CERTIFICATE_ARN` | Ingress only: use an existing web ACM certificate |
+| `MQTTS_CERTIFICATE_ARN` | Ingress only: use an existing MQTTS ACM certificate |
+
+Load1 has one fixed profile; omit `LOAD_PROFILE`. Load2 supports `large`
+(the fallback when omitted), `xlarge-minimal`, `xlarge`, `2xlarge`, `4xlarge`,
+and `8xlarge`. The example config selects `xlarge`.
+
+### DNS configuration and retries
+
+The DNS role grants access to the selected zone/domain and trusts the cluster's
+exact ExternalDNS IRSA role. The helper uses a stable, account-qualified external
+ID identifying that relationship; it is not an access key. No manually copied
+role ARN or generated local state file is needed to resume on another machine.
+
+On an existing cluster, `up` preserves ExternalDNS's TXT ownership ID and
+refuses to change its managed zone/domain. It configures ExternalDNS to use the
+helper's dedicated role, leaving any previously used DNS-account role intact.
+Use `deploy` when the cluster's DNS IAM and add-ons should remain as configured.
+
+The preflight requests a short-lived token for the `external-dns` service
+account and tests both STS role assumptions and Route 53 reads. Your Kubernetes
+identity needs permission to create that token. This catches trust/external-ID
+mismatches early; DNS record writes and propagation are checked by deployment
+readiness. Tokens and rendered temporary values are removed on exit, including
+failure. Configuration and prepared values remain available for retries.
+
+### Ingress with ACM
+
+For a **new** stack, add these settings before running `up` or `deploy`:
+
+```bash
+LOAD_EXPOSURE=ingress
+LOAD_MQTTS_HOSTNAME=mqtt-load2.openremote.app
+```
+
+The web endpoint uses a shared ALB; MQTTS uses a dedicated NLB and the separate
+hostname. Set the MQTT scenarios' `MANAGER_HOSTNAME` to `LOAD_MQTTS_HOSTNAME`;
+browser access still uses `LOAD_HOSTNAME`. The CLI uses a configured shared ACM
+certificate if available, otherwise manages certificates for the stack. The
+optional ARN settings select existing certificates instead.
+
+Exposure and hostname changes require a new stack. This topology differs from
+the legacy experiment, which puts both protocols behind one ACM-backed NLB.
+For deployment modes outside this helper, use the
+[EKS CLI](../kubernetes/README-AWS.md) or [or-stack](../kubernetes/README.md)
+directly.
+
+### Startup and capacity
+
+Profiles contain the cluster template and component resource budgets. `up`
+uses the template when creating a cluster; it does not resize existing nodes.
+Manager and Keycloak use `Recreate` updates to avoid overlapping Pods on a
+single-node cluster, so upgrades interrupt service.
+
+If dataset initialization exceeds the Manager startup probe allowance, increase
+that allowance in prepared `manager.yaml` as well as `LOAD_TIMEOUT`. A shared
+cluster needs capacity for all stacks plus Kubernetes and shared controllers.
+Namespace isolation does not remove CPU, disk, or network contention; use
+separate clusters for independent benchmarks.
+
+## Legacy scripts
+
+The old load1 scripts and load2's standard `eks-setup-load.sh` remain in the
+repository but stop with guidance directing users to the CLIs.
+The [load2 ACM/NLB experiment](load2-eks/README.md#legacy-acmnlb-multi-proxy-experiment)
+remains deployable using its existing setup, redeployment, and cleanup scripts.
+Use a dedicated cluster for that workflow; do not mix it with CLI-managed stacks.
+
+## Offline validation
+
+```bash
+python3 kubernetes/test/load-helper-test
+python3 kubernetes/test/keycloak-url-test
+python3 kubernetes/test/load-eks-test
+```
+
+The helper tests require Python 3, Bash, jq, and envsubst. They stub AWS,
+Kubernetes, Helm, and the EKS CLIs to check all profiles, DNS discovery,
+role-access failures, retries, config handling, and separate cleanup operations.
+The Keycloak URL tests and profile tests also require Helm. The profile tests
+render all seven profiles through the real `or-stack` and cover the retained
+legacy ACM/NLB scripts. No AWS credentials or cluster are used.
+
+Real EKS acceptance still requires login, dataset initialization, MQTTS traffic,
+persistence, and independent stack cleanup against a deployed environment.
diff --git a/test/load1-eks/README.md b/test/load1-eks/README.md
index ea790dc81..74d03f233 100644
--- a/test/load1-eks/README.md
+++ b/test/load1-eks/README.md
@@ -1,13 +1,13 @@
-# Load Tests
+# Load1 on EKS
-Scripts and configuration files to setup an OpenRemote stack running in an EKS cluster appropriate for load testing.
-This includes configuring the cluster to use a more powerful machine
-and adapting the values files to configure the memory usage on the different pods.
-For the manager, JVM parameters are used to make use of the extra memory available to the container.
-It also uses a custom manager image (available in the ECR) that includes setup code to create assets and users used during the tests.
+Deploy load1 with `test/or-eks-load`, which selects its profile and calls
+`kubernetes/or-eks-cluster` and `kubernetes/or-eks-stack`.
+Cluster creation and stack deployment are independent: use a dedicated load-test
+cluster, or deploy another named stack into an existing compatible cluster.
+The cluster template uses one `m8g.2xlarge` node. Component values preserve the
+load1 resource budgets and provision separate 16 GiB Manager and PostgreSQL PVCs.
-Once deployed, running the load tests can be done using the same clients as for load testing a VM,
-see `load1` folder for the tools, scripts and documentation.
+Run the commands below from the **repository root**.
## Creating load test manager image
@@ -38,3 +38,40 @@ The JAR copy above is required: without it, the image can initialize Keycloak
and allow login, but the custom load-test setup provider is absent and no test
users or assets are created. During a clean initialization, Manager logs should
identify `org.openremote.setup.load1.SetupTasks` as a custom setup provider.
+
+## Deploy the cluster and stack
+
+Copy the example configuration and edit its hostname, image repository, and
+AWS profile settings:
+
+```bash
+mkdir -p .local
+cp test/load1-eks/deployment.env.example .local/loadtest.env
+```
+
+Then deploy and retrieve credentials:
+
+```bash
+test/or-eks-load up --config .local/loadtest.env
+test/or-eks-load credentials --config .local/loadtest.env
+```
+
+The [shared deployment guide](../README-eks-load.md) explains prerequisites,
+configuration, verification, and cleanup. The Bash helper discovers the DNS
+zone, configures the cluster's DNS role, and prepares component values
+automatically. Use `cluster-up` and `deploy` separately when managing an
+existing cluster. Selecting another profile does not resize existing nodes.
+
+## Running tests
+
+Use the clients and scripts in [load1](../load1/README.md), with
+`MANAGER_HOSTNAME` set to this stack's hostname. The default deployment exposes
+HTTPS on 443 and MQTTS on 8883 through the same HAProxy/NLB.
+
+Before a load run, verify browser login, asset creation/editing, and an
+authenticated MQTTS publish using a provisioned test account. On clean startup,
+Manager logs should identify `org.openremote.setup.load1.SetupTasks`.
+
+The `eks-*.sh` entry points now stop with migration guidance. They no longer
+configure AWS credentials, provision static EBS volumes, or operate on the
+current context's default namespace.
diff --git a/test/load1-eks/cluster.yaml b/test/load1-eks/cluster.yaml
index 4768a0c6c..c91a1630b 100644
--- a/test/load1-eks/cluster.yaml
+++ b/test/load1-eks/cluster.yaml
@@ -29,6 +29,12 @@ nodeGroups:
- ${AWS_REGION}a
addons:
+ - name: vpc-cni
+ version: latest
+ attachPolicyARNs:
+ - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
+ configurationValues: '{"enableNetworkPolicy":"true"}'
+ resolveConflicts: overwrite
- name: aws-ebs-csi-driver
version: latest
wellKnownPolicies:
diff --git a/test/load1-eks/deployment.env.example b/test/load1-eks/deployment.env.example
new file mode 100644
index 000000000..28c343311
--- /dev/null
+++ b/test/load1-eks/deployment.env.example
@@ -0,0 +1,13 @@
+# Copy to .local/loadtest.env and edit before running test/or-eks-load.
+# This is a trusted Bash file. Keep AWS access keys out of it.
+LOAD_TEST=load1
+CLUSTER_NAME=loadtest
+STACK_NAME=load1
+AWS_REGION=eu-west-1
+LOAD_HOSTNAME=load1.example.com
+LOAD_MANAGER_REPOSITORY=123456789012.dkr.ecr.eu-west-1.amazonaws.com/openremote/manager
+
+# Leave empty to use the cluster credentials, or name the DNS account's profile.
+DNS_AWS_PROFILE=""
+# Cluster commands use the normal AWS credential chain by default.
+# CLUSTER_AWS_PROFILE=my-cluster-account
diff --git a/test/load1-eks/eks-cleanup-load.sh b/test/load1-eks/eks-cleanup-load.sh
index e0b2f811c..217b771ab 100755
--- a/test/load1-eks/eks-cleanup-load.sh
+++ b/test/load1-eks/eks-cleanup-load.sh
@@ -1,66 +1,8 @@
-#!/bin/bash
-
-. ./eks-common.sh
-
-CLUSTER_VPC_ID=$(aws eks describe-cluster --name $CLUSTER_NAME --query 'cluster.resourcesVpcConfig.vpcId' --output text --profile or)
-if [ -z "$CLUSTER_VPC_ID" ] || [ "$CLUSTER_VPC_ID" = "None" ]; then
- echo "Error: Failed to retrieve VPC ID for cluster '$CLUSTER_NAME'. Aborting."
- exit 1
-fi
-
-DNS_NAME=$(aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?VpcId=='$CLUSTER_VPC_ID' && Type=='network' && Scheme=='internet-facing'].DNSName | [0]")
-HOSTED_ZONE_ID=$(aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?VpcId=='$CLUSTER_VPC_ID' && Type=='network' && Scheme=='internet-facing'].CanonicalHostedZoneId | [0]")
-
-echo "Delete DNS record $FQDN"
-
-aws route53 change-resource-record-sets \
- --hosted-zone-id /hostedzone/Z08751721JH0NB6LLCB4V \
- --change-batch \
- '{"Changes": [ { "Action": "DELETE", "ResourceRecordSet": { "Name": "'$FQDN'", "Type": "A", "AliasTarget":{ "HostedZoneId": '$HOSTED_ZONE_ID',"DNSName": '$DNS_NAME',"EvaluateTargetHealth": false} } } ]}' \
- --profile dnschg
-
-helm uninstall manager
-helm uninstall keycloak
-helm uninstall postgresql
-helm uninstall proxy
-
-echo "Delete VPC peerings"
-
-VPC_PEERING_IDS=$(aws ec2 describe-vpc-peering-connections \
- --filters "Name=accepter-vpc-info.vpc-id,Values=$CLUSTER_VPC_ID" "Name=status-code,Values=active" \
- --query 'VpcPeeringConnections[*].VpcPeeringConnectionId' \
- --profile or \
- --output text)
-
-for peering_id in $VPC_PEERING_IDS; do
- echo "Deleting peering $peering_id"
- aws ec2 delete-vpc-peering-connection --vpc-peering-connection-id $peering_id --profile or
-done
-
-# Give the AWS LB Controller time to delete the NLBs after the Service is deleted
-while aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?VpcId=='$CLUSTER_VPC_ID' && Type=='network']" 2>/dev/null | grep '"Code": "active"'; do
- echo "Waiting for load balancers to be deleted..."
- sleep 10
-done
-helm uninstall aws-load-balancer-controller -n kube-system
-
-MANAGER_VOLUMEID=$(kubectl get pv manager-data-pv -o=jsonpath='{.spec.awsElasticBlockStore.volumeID}')
-PSQL_VOLUMEID=$(kubectl get pv postgresql-data-pv -o=jsonpath='{.spec.awsElasticBlockStore.volumeID}')
-kubectl delete pv manager-data-pv
-kubectl delete pv postgresql-data-pv
-aws ec2 delete-volume --volume-id $MANAGER_VOLUMEID
-aws ec2 delete-volume --volume-id $PSQL_VOLUMEID
-helm uninstall or-setup
-
-# Manually deleting all addons, this should not be required but otherwise the delete cluster fails
-# with "2 pods are unevictable from node ..." error messages
-eksctl delete addon --cluster $CLUSTER_NAME --name aws-ebs-csi-driver
-eksctl delete addon --cluster $CLUSTER_NAME --name vpc-cni
-eksctl delete addon --cluster $CLUSTER_NAME --name metrics-server
-eksctl delete addon --cluster $CLUSTER_NAME --name coredns
-eksctl delete addon --cluster $CLUSTER_NAME --name kube-proxy
-
-eksctl delete cluster --wait --name $CLUSTER_NAME || {
- echo "Clean-up has failed, please check the status in AWS CloudFormation console at https://eu-west-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks?filteringText=&filteringStatus=active&viewNested=true"
- echo "Failed clean-ups are often caused by a dangling VPC. Manually delete the VPC and then retry the cluster stack delete operation."
-}
+#!/usr/bin/env bash
+
+# Retained legacy entry point. See README.md for the separate CLI lifecycles.
+printf '%s\n' \
+ 'This legacy load-test script is no longer supported.' \
+ 'Use kubernetes/or-eks-cluster for the cluster and kubernetes/or-eks-stack for each stack.' \
+ "See the README.md in this script's directory for image, profile, DNS, and cleanup configuration." >&2
+exit 1
diff --git a/test/load1-eks/eks-common.sh b/test/load1-eks/eks-common.sh
index 153a513cd..217b771ab 100755
--- a/test/load1-eks/eks-common.sh
+++ b/test/load1-eks/eks-common.sh
@@ -1,47 +1,8 @@
-#!/bin/bash
-
-set -eo pipefail
-
-if [ -z "$AWS_ACCOUNT_ID" ]; then
- echo "Error: AWS_ACCOUNT_ID environment variable is not set"
- echo "Please set it to the ID of the account you want to deploy the cluster into with: export AWS_ACCOUNT_ID=your-account-id"
- exit 1
-fi
-if [ -z "$AWS_DEVELOPERS_ACCOUNT_ID" ]; then
- echo "Error: $AWS_DEVELOPERS_ACCOUNT_ID environment variable is not set"
- echo "Please set it to the ID of the OpenRemote developers root account with: export AWS_DEVELOPERS_ACCOUNT_ID=dev-account-id"
- exit 1
-fi
-
-OR_KUBERNETES_PATH=../../kubernetes
-
-# Name of cluster, not exposed but must be unique within account
-export CLUSTER_NAME=loadtest-cluster
-
-# Hostname to use for public access to this instance, always under the openremote.app domain
-HOSTNAME=load1
-FQDN=$HOSTNAME.openremote.app
-
-export AWS_REGION="eu-west-1"
-
-DNSCHG_ROLE_ARN="arn:aws:iam::$AWS_DEVELOPERS_ACCOUNT_ID:role/route53-full-access"
-
-aws configure set region $AWS_REGION
-aws configure --profile or set region $AWS_REGION
-aws configure --profile or set output json
-aws configure --profile dnschg set source_profile or
-aws configure --profile dnschg set role_arn $DNSCHG_ROLE_ARN
-
-if [ -n "$AWS_ACCESS_KEY_ID" ]; then
- aws configure --profile or set aws_access_key_id $AWS_ACCESS_KEY_ID
-fi
-
-if [ -n "$AWS_SECRET_ACCESS_KEY" ]; then
- aws configure --profile or set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
-fi
-
-if [ -n "$AWS_SESSION_TOKEN" ]; then
- aws configure --profile or set aws_session_token $AWS_SESSION_TOKEN
-fi
-
-aws sts assume-role --role-arn $DNSCHG_ROLE_ARN --role-session-name dnschg --profile or --query "Credentials.[AccessKeyId, SecretAccessKey, SessionToken]" --output text | awk -F'\t' '{print "aws_access_key_id "$1"\naws_secret_access_key "$2"\naws_session_token "$3 }' | xargs -L 1 aws configure --profile dnschg set
+#!/usr/bin/env bash
+
+# Retained legacy entry point. See README.md for the separate CLI lifecycles.
+printf '%s\n' \
+ 'This legacy load-test script is no longer supported.' \
+ 'Use kubernetes/or-eks-cluster for the cluster and kubernetes/or-eks-stack for each stack.' \
+ "See the README.md in this script's directory for image, profile, DNS, and cleanup configuration." >&2
+exit 1
diff --git a/test/load1-eks/eks-setup-load.sh b/test/load1-eks/eks-setup-load.sh
index 972aa43b7..217b771ab 100755
--- a/test/load1-eks/eks-setup-load.sh
+++ b/test/load1-eks/eks-setup-load.sh
@@ -1,72 +1,8 @@
-#!/bin/bash
-
-. ./eks-common.sh
-
-envsubst < cluster.yaml | eksctl create cluster -f - --profile or
-
-# [Installation Guide - AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/deploy/installation/)
-
-helm repo add eks https://aws.github.io/eks-charts
-helm repo update eks
-
-helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
- -n kube-system \
- --set clusterName=$CLUSTER_NAME \
- --set serviceAccount.create=false \
- --set serviceAccount.name=aws-load-balancer-controller
-
-PSQL_VOLUMEID=$(aws ec2 create-volume --size 1 --availability-zone eu-west-1a --tag-specifications "ResourceType=volume,Tags=[{Key=Name,Value=psql-data}]" --query VolumeId)
-MANAGER_VOLUMEID=$(aws ec2 create-volume --size 1 --availability-zone eu-west-1a --tag-specifications "ResourceType=volume,Tags=[{Key=Name,Value=manager-data}]" --query VolumeId)
-
-# Wait for AWS LB ctrl to be ready
-kubectl rollout status deployment aws-load-balancer-controller -n kube-system --timeout=300s
-
-CLUSTER_DNS=$(kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}:{.spec.ports[?(@.name=="dns")].port}')
-
-helm install or-setup $OR_KUBERNETES_PATH/or-setup -f values-or-setup-eks-load.yaml --set aws.enabled=true --set aws.managerVolumeId=$MANAGER_VOLUMEID --set aws.psqlVolumeId=$PSQL_VOLUMEID
-
-helm install proxy $OR_KUBERNETES_PATH/proxy -f $OR_KUBERNETES_PATH/proxy/values-eks.yaml -f values-proxy-eks-load.yaml --set or.nameserver=$CLUSTER_DNS --set or.hostname=$FQDN
-
-helm install postgresql $OR_KUBERNETES_PATH/postgresql -f $OR_KUBERNETES_PATH/postgresql/values-eks.yaml -f values-postgresql-eks-load.yaml
-
-# Waiting for the LB to be created
-# AWS LB Controller only creates an Network LB if there's a service
-
-CLUSTER_VPC_ID=$(aws eks describe-cluster --profile or --name $CLUSTER_NAME --query 'cluster.resourcesVpcConfig.vpcId' --output text)
-if [ -z "$CLUSTER_VPC_ID" ] || [ "$CLUSTER_VPC_ID" = "None" ]; then
- echo "Error: Failed to retrieve VPC ID for cluster '$CLUSTER_NAME'. Aborting."
- exit 1
-fi
-
-while ! aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?VpcId=='$CLUSTER_VPC_ID' && Type=='network' && Scheme=='internet-facing']" 2>/dev/null | grep '"Code": "active"'; do
- echo "Waiting for load balancer to be created..."
- sleep 10
-done
-
-DNS_NAME=$(aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?VpcId=='$CLUSTER_VPC_ID' && Type=='network' && Scheme=='internet-facing'].DNSName | [0]")
-HOSTED_ZONE_ID=$(aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?VpcId=='$CLUSTER_VPC_ID' && Type=='network' && Scheme=='internet-facing'].CanonicalHostedZoneId | [0]")
-
-aws route53 change-resource-record-sets \
- --hosted-zone-id /hostedzone/Z08751721JH0NB6LLCB4V \
- --change-batch \
- '{"Changes": [ { "Action": "UPSERT", "ResourceRecordSet": { "Name": "'$FQDN'", "Type": "A", "AliasTarget":{ "HostedZoneId": '$HOSTED_ZONE_ID',"DNSName": '$DNS_NAME',"EvaluateTargetHealth": false} } } ]}' \
- --profile dnschg
-
-helm install keycloak $OR_KUBERNETES_PATH/keycloak -f $OR_KUBERNETES_PATH/keycloak/values-haproxy.yaml \
- -f values-keycloak-eks-load.yaml --set-string or.hostname=$FQDN
-helm install manager $OR_KUBERNETES_PATH/manager -f $OR_KUBERNETES_PATH/manager/values-haproxy-eks.yaml \
- -f values-manager-eks-load.yaml --set-string or.hostname=$FQDN \
- --set-string image.repository=$AWS_DEVELOPERS_ACCOUNT_ID.dkr.ecr.eu-west-1.amazonaws.com/openremote/manager
-
-while ! dig +short $FQDN | grep -qE '^[0-9]'; do
- echo "Waiting for DNS resolution..."
- sleep 5
-done
-
-# Now that DNS is in place, HAProxy can properly create the certificate
-kubectl exec $(kubectl get pod -l "app.kubernetes.io/name=proxy" -o name) -- sh -c "/entrypoint.sh add $FQDN" || {
- echo "Certificate generation failed, please wait a moment and manually execute the following command:"
- echo "kubectl exec \$(kubectl get pod -l \"app.kubernetes.io/name=proxy\" -o name) -- sh -c \"/entrypoint.sh add $FQDN\""
-}
-
-echo "Access the manager at https://$FQDN"
+#!/usr/bin/env bash
+
+# Retained legacy entry point. See README.md for the separate CLI lifecycles.
+printf '%s\n' \
+ 'This legacy load-test script is no longer supported.' \
+ 'Use kubernetes/or-eks-cluster for the cluster and kubernetes/or-eks-stack for each stack.' \
+ "See the README.md in this script's directory for image, profile, DNS, and cleanup configuration." >&2
+exit 1
diff --git a/test/load1-eks/keycloak.yaml b/test/load1-eks/keycloak.yaml
new file mode 100644
index 000000000..f4dc0a8c0
--- /dev/null
+++ b/test/load1-eks/keycloak.yaml
@@ -0,0 +1,11 @@
+# Single-node load profiles cannot assume spare capacity for surge Pods.
+strategy:
+ type: Recreate
+
+resources:
+ limits:
+ cpu: 1500m
+ memory: 5Gi
+ requests:
+ cpu: 1200m
+ memory: 5Gi
diff --git a/test/load1-eks/values-manager-eks-load.yaml b/test/load1-eks/manager.yaml
similarity index 71%
rename from test/load1-eks/values-manager-eks-load.yaml
rename to test/load1-eks/manager.yaml
index 1abfed0a5..dca0126ef 100644
--- a/test/load1-eks/values-manager-eks-load.yaml
+++ b/test/load1-eks/manager.yaml
@@ -1,4 +1,10 @@
+# Single-node load profiles cannot assume spare capacity for surge Pods.
+strategy:
+ type: Recreate
+
+# Render this template with LOAD_MANAGER_REPOSITORY before applying; see README.
image:
+ repository: "${LOAD_MANAGER_REPOSITORY}"
pullPolicy: Always
tag: "load1"
@@ -27,7 +33,8 @@ resources:
cpu: 3000m
memory: 13Gi
-volumeSize: "16Gi"
+persistence:
+ size: "16Gi"
or:
env:
diff --git a/test/load2-eks/profiles/2xlarge/values-postgresql-eks-load.yaml b/test/load1-eks/postgresql.yaml
similarity index 77%
rename from test/load2-eks/profiles/2xlarge/values-postgresql-eks-load.yaml
rename to test/load1-eks/postgresql.yaml
index 7b8df849e..c827dc8cd 100644
--- a/test/load2-eks/profiles/2xlarge/values-postgresql-eks-load.yaml
+++ b/test/load1-eks/postgresql.yaml
@@ -1,4 +1,5 @@
-volumeSize: "16Gi"
+persistence:
+ size: "16Gi"
resources:
limits:
diff --git a/test/load1-eks/values-proxy-eks-load.yaml b/test/load1-eks/proxy.yaml
similarity index 100%
rename from test/load1-eks/values-proxy-eks-load.yaml
rename to test/load1-eks/proxy.yaml
diff --git a/test/load1-eks/values-keycloak-eks-load.yaml b/test/load1-eks/values-keycloak-eks-load.yaml
deleted file mode 100644
index ed0ca53cc..000000000
--- a/test/load1-eks/values-keycloak-eks-load.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-resources:
- limits:
- cpu: 1500m
- memory: 5Gi
- requests:
- cpu: 1200m
- memory: 5Gi
diff --git a/test/load1-eks/values-or-setup-eks-load.yaml b/test/load1-eks/values-or-setup-eks-load.yaml
deleted file mode 100644
index 8dcadffa0..000000000
--- a/test/load1-eks/values-or-setup-eks-load.yaml
+++ /dev/null
@@ -1,6 +0,0 @@
-managerVolumeSize: "16Gi"
-psqlVolumeSize: "16Gi"
-
-aws:
- # Set enabled to true if deployed under AWS
- enabled: true
diff --git a/test/load2-eks/README.md b/test/load2-eks/README.md
index 01d968f55..64695fce0 100644
--- a/test/load2-eks/README.md
+++ b/test/load2-eks/README.md
@@ -1,10 +1,12 @@
# Load Tests
-Scripts and configuration files to setup an OpenRemote stack running in an EKS cluster appropriate for load testing.
-This includes configuring the cluster to use a more powerful machine
-and adapting the values files to configure the memory usage on the different pods.
-For the manager, JVM parameters are used to make use of the extra memory available to the container.
-There are different "profiles" available depending on the power required and the tests to be run:
+Deploy load2 with `test/or-eks-load`, which selects its profile and calls
+`kubernetes/or-eks-cluster` and `kubernetes/or-eks-stack`.
+Create a dedicated cluster or deploy independently into an existing compatible
+cluster. Each stack has its own namespace, credentials, storage, and endpoints.
+Run the commands below from the **repository root**.
+
+Profiles select the node size and per-component CPU/memory budgets:
- large: minimal set-up useful to test memory leaks and pressure condition
- xlarge-minimal: gives Manager 2Gi more memory than large (3840Mi vs 1792Mi), with room for further increases
@@ -13,7 +15,13 @@ There are different "profiles" available depending on the power required and the
- 4xlarge: bigger setup using all capacity on 16 CPU/64 GB cluster
- 8xlarge: bigger setup using all capacity on 32 CPU/128 GB cluster
-The profile is selected by setting the OR_PROFILE environment variable in the `eks-common.sh` script.
+Select a profile through `LOAD_PROFILE` in the deployment config. Each directory contains an
+`or-eks-cluster --config` template and the four shared component values files:
+`manager.yaml`, `postgresql.yaml`, `keycloak.yaml`, and `proxy.yaml`. Both the
+CLI workflow and the legacy ACM/NLB scripts use these files.
+All profiles provision a 32 GiB Manager PVC and a 256 GiB PostgreSQL PVC,
+preserving the intended sizes from the old provisioning configuration. The old
+per-chart `volumeSize: 16Gi` keys were no longer consumed by the charts.
The `large` and `xlarge-minimal` profiles now use 768Mi for Keycloak.
The old 512Mi allocation caused Keycloak 26 to be OOMKilled
@@ -47,6 +55,8 @@ The setup creates:
- OR_SETUP_USERS restricted service users, linked to the corresponding Building asset,
with appropriate permission to push attribute values
+## Creating the load test Manager image
+
Build and push the custom Manager image from the repository root using the
checkout you want to test. All profiles use the custom `load2` tag, so upgrading
the Manager chart's `appVersion` does not upgrade this image. Rebuild and push
@@ -74,17 +84,28 @@ and allow login, but the custom load-test setup provider is absent and no test
users or assets are created. During a clean initialization, Manager logs should
identify `org.openremote.setup.load2.SetupTasks` as a custom setup provider.
-You might want to update the stack without re-creating the whole cluster, as re-creating the cluster takes time and
-requires re-generating a new certificate (on which we have some limits).
-To do so, keep the proxy running and uninstall the other charts
+## Deploy the cluster and stack
-```shell
-helm uninstall manager postgresql keycloak
+Copy the example configuration and edit its hostname, image repository, and
+AWS profile settings:
+
+```bash
+mkdir -p .local
+cp test/load2-eks/deployment.env.example .local/loadtest.env
```
-then use the `eks-deploy-load.sh` script to properly re-deploy those charts.
-That script sets `or.setupRunOnRestart=true` to reset and rebuild the test
-dataset; this procedure does not preserve your existing test data.
+Then deploy and retrieve credentials:
+
+```bash
+test/or-eks-load up --config .local/loadtest.env
+test/or-eks-load credentials --config .local/loadtest.env
+```
+
+The [shared deployment guide](../README-eks-load.md) explains prerequisites,
+configuration, verification, and cleanup. The Bash helper discovers the DNS
+zone, configures the cluster's DNS role, and prepares component values
+automatically. Use `cluster-up` and `deploy` separately when managing an
+existing cluster. Selecting another profile does not resize existing nodes.
## Switching an existing Deployment to Recreate
@@ -142,7 +163,11 @@ still encounter the issue if it starts with `RollingUpdate` and switches later.
See the Kubernetes documentation on [patching API objects](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/)
and [server-side apply](https://kubernetes.io/docs/reference/using-api/server-side-apply/).
-## Running load tests
+## Running tests
+
+Before starting a load run, verify browser login, asset creation/editing, and
+an authenticated MQTTS publish using a provisioned test account. On clean
+startup, Manager logs should identify `org.openremote.setup.load2.SetupTasks`.
Once deployed, running the load tests can be done using the same clients as for load testing a VM,
but using the scenarios present under `scenarios` in this folder instead.
@@ -179,7 +204,7 @@ THREAD_COUNT: Number of parallel accounts that will connect and publish in paral
RAMP_RATE: Number of thread to add per second during ramp-up. Default is 50.
MILLIS_BETWEEN_PUBLISHES: Delay between each publishing iteration. Default is 30000.
-## Multi-proxy test
+## Legacy ACM/NLB multi-proxy experiment
As part of [Test using multiple HAProxy replicas under EKS · Issue #2568 · openremote/core](https://github.com/openremote/core/issues/2568),
a setup terminating TLS at the AWS NLB level and using ACM to manage certificates, while still using HAProxy pod(s) has been implemented.
diff --git a/test/load2-eks/deployment.env.example b/test/load2-eks/deployment.env.example
new file mode 100644
index 000000000..c07ef23b4
--- /dev/null
+++ b/test/load2-eks/deployment.env.example
@@ -0,0 +1,14 @@
+# Copy to .local/loadtest.env and edit before running test/or-eks-load.
+# This is a trusted Bash file. Keep AWS access keys out of it.
+LOAD_TEST=load2
+LOAD_PROFILE=xlarge
+CLUSTER_NAME=loadtest
+STACK_NAME=load2
+AWS_REGION=eu-west-1
+LOAD_HOSTNAME=load2.example.com
+LOAD_MANAGER_REPOSITORY=123456789012.dkr.ecr.eu-west-1.amazonaws.com/openremote/manager
+
+# Leave empty to use the cluster credentials, or name the DNS account's profile.
+DNS_AWS_PROFILE=""
+# Cluster commands use the normal AWS credential chain by default.
+# CLUSTER_AWS_PROFILE=my-cluster-account
diff --git a/test/load2-eks/eks-cleanup-load.sh b/test/load2-eks/eks-cleanup-load.sh
index 5b11f85ae..ecd20b49e 100755
--- a/test/load2-eks/eks-cleanup-load.sh
+++ b/test/load2-eks/eks-cleanup-load.sh
@@ -1,11 +1,20 @@
#!/bin/bash
+set -eo pipefail
+
if [ -z "$K_CONTEXT" ]; then
echo "Error: K_CONTEXT environment variable is not set"
echo "Please set it to the kubernetes context to use with: export K_CONTEXT=context"
exit 1
fi
+# Never let this single-stack cleanup remove a namespaced CLI deployment.
+CLI_NAMESPACES=$(kubectl --context "$K_CONTEXT" get namespaces -l openremote.io/stack -o name)
+if [ -n "$CLI_NAMESPACES" ]; then
+ echo "Refusing legacy cleanup: cluster contains CLI-managed stacks." >&2
+ exit 1
+fi
+
. ./eks-common.sh
CLUSTER_VPC_ID=$(aws eks describe-cluster --name $CLUSTER_NAME --query 'cluster.resourcesVpcConfig.vpcId' --output text --profile or)
@@ -25,10 +34,10 @@ aws route53 change-resource-record-sets \
'{"Changes": [ { "Action": "DELETE", "ResourceRecordSet": { "Name": "'$FQDN'", "Type": "A", "AliasTarget":{ "HostedZoneId": '$HOSTED_ZONE_ID',"DNSName": '$DNS_NAME',"EvaluateTargetHealth": false} } } ]}' \
--profile dnschg
-helm uninstall --kube-context=$K_CONTEXT manager
-helm uninstall --kube-context=$K_CONTEXT keycloak
-helm uninstall --kube-context=$K_CONTEXT postgresql
-helm uninstall --kube-context=$K_CONTEXT proxy
+helm uninstall --kube-context="$K_CONTEXT" --namespace default manager
+helm uninstall --kube-context="$K_CONTEXT" --namespace default keycloak
+helm uninstall --kube-context="$K_CONTEXT" --namespace default postgresql
+helm uninstall --kube-context="$K_CONTEXT" --namespace default proxy
echo "Delete VPC peerings"
@@ -48,25 +57,27 @@ while aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?Vp
echo "Waiting for load balancers to be deleted..."
sleep 10
done
-helm uninstall --kube-context=$K_CONTEXT aws-load-balancer-controller -n kube-system
+helm uninstall --kube-context="$K_CONTEXT" aws-load-balancer-controller -n kube-system
-MANAGER_VOLUMEID=$(kubectl get pv manager-data-pv -o=jsonpath='{.spec.awsElasticBlockStore.volumeID}')
-PSQL_VOLUMEID=$(kubectl get pv postgresql-data-pv -o=jsonpath='{.spec.awsElasticBlockStore.volumeID}')
-kubectl --context $K_CONTEXT delete pv manager-data-pv
-kubectl --context $K_CONTEXT delete pv postgresql-data-pv
-aws ec2 delete-volume --volume-id $MANAGER_VOLUMEID
-aws ec2 delete-volume --volume-id $PSQL_VOLUMEID
-helm uninstall --kube-context=$K_CONTEXT or-setup
+# Current charts retain namespaced PVCs and the generated Secret on uninstall.
+# This cleanup is only for the dedicated legacy cluster, never a CLI cluster.
+VOLUME_NAMES=$(kubectl --context "$K_CONTEXT" --namespace default get pvc manager postgresql proxy --ignore-not-found -o jsonpath='{.items[*].spec.volumeName}')
+kubectl --context "$K_CONTEXT" --namespace default delete pvc manager postgresql proxy --ignore-not-found
+for volume in $VOLUME_NAMES; do
+ kubectl --context "$K_CONTEXT" wait --for=delete "pv/$volume" --timeout=10m
+done
+helm uninstall --kube-context="$K_CONTEXT" --namespace default or-setup
+kubectl --context "$K_CONTEXT" --namespace default delete secret openremote-secret --ignore-not-found
# Manually deleting all addons, this should not be required but otherwise the delete cluster fails
# with "2 pods are unevictable from node ..." error messages
-eksctl delete addon --cluster $CLUSTER_NAME --name aws-ebs-csi-driver
-eksctl delete addon --cluster $CLUSTER_NAME --name vpc-cni
-eksctl delete addon --cluster $CLUSTER_NAME --name metrics-server
-eksctl delete addon --cluster $CLUSTER_NAME --name coredns
-eksctl delete addon --cluster $CLUSTER_NAME --name kube-proxy
+eksctl delete addon --profile or --region "$AWS_REGION" --cluster $CLUSTER_NAME --name aws-ebs-csi-driver
+eksctl delete addon --profile or --region "$AWS_REGION" --cluster $CLUSTER_NAME --name vpc-cni
+eksctl delete addon --profile or --region "$AWS_REGION" --cluster $CLUSTER_NAME --name metrics-server
+eksctl delete addon --profile or --region "$AWS_REGION" --cluster $CLUSTER_NAME --name coredns
+eksctl delete addon --profile or --region "$AWS_REGION" --cluster $CLUSTER_NAME --name kube-proxy
-eksctl delete cluster --wait --name $CLUSTER_NAME || {
+eksctl delete cluster --profile or --region "$AWS_REGION" --wait --name $CLUSTER_NAME || {
echo "Clean-up has failed, please check the status in AWS CloudFormation console at https://eu-west-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks?filteringText=&filteringStatus=active&viewNested=true"
echo "Failed clean-ups are often caused by a dangling VPC. Manually delete the VPC and then retry the cluster stack delete operation."
}
diff --git a/test/load2-eks/eks-common.sh b/test/load2-eks/eks-common.sh
index eb316b4d1..102384d02 100755
--- a/test/load2-eks/eks-common.sh
+++ b/test/load2-eks/eks-common.sh
@@ -2,6 +2,8 @@
set -eo pipefail
+# Legacy ACM/NLB experiment only. Use the CLIs for normal deployments.
+
if [ -z "$AWS_ACCOUNT_ID" ]; then
echo "Error: AWS_ACCOUNT_ID environment variable is not set"
echo "Please set it to the ID of the account you want to deploy the cluster into with: export AWS_ACCOUNT_ID=your-account-id"
@@ -14,16 +16,18 @@ if [ -z "$AWS_DEVELOPERS_ACCOUNT_ID" ]; then
fi
OR_KUBERNETES_PATH=../../kubernetes
-export OR_PROFILE=large
+export OR_PROFILE=${OR_PROFILE:-large}
# Name of cluster, not exposed but must be unique within account
-export CLUSTER_NAME=loadtest-cluster
+export CLUSTER_NAME=${CLUSTER_NAME:-load2-legacy-cluster}
# Hostname to use for public access to this instance, always under the openremote.app domain
-HOSTNAME=load2
-FQDN=$HOSTNAME.openremote.app
+FQDN=${FQDN:-load2-legacy.openremote.app}
+
+export AWS_REGION="${AWS_REGION:-eu-west-1}"
+K_CONTEXT=${K_CONTEXT:-$CLUSTER_NAME@$AWS_REGION}
-export AWS_REGION="eu-west-1"
+printf '%s\n' "Legacy load2 workflow: dedicated cluster $CLUSTER_NAME, default namespace." >&2
DNSCHG_ROLE_ARN="arn:aws:iam::$AWS_DEVELOPERS_ACCOUNT_ID:role/route53-full-access"
diff --git a/test/load2-eks/eks-deploy-load.sh b/test/load2-eks/eks-deploy-load.sh
index 7c802f2ed..7a6f62d14 100755
--- a/test/load2-eks/eks-deploy-load.sh
+++ b/test/load2-eks/eks-deploy-load.sh
@@ -8,14 +8,14 @@ fi
. ./eks-common.sh
-kubectl --context $K_CONTEXT patch pv manager-data-pv -p '{"spec":{"claimRef": null}}'
-kubectl --context $K_CONTEXT patch pv postgresql-data-pv -p '{"spec":{"claimRef": null}}'
+# Current charts retain and reuse their PVCs; no static PV rebinding is needed.
+helm upgrade --install --kube-context="$K_CONTEXT" --namespace default --wait --timeout 90m postgresql $OR_KUBERNETES_PATH/postgresql -f $OR_KUBERNETES_PATH/postgresql/values-eks.yaml -f "profiles/$OR_PROFILE/postgresql.yaml"
-helm install --kube-context=$K_CONTEXT postgresql $OR_KUBERNETES_PATH/postgresql -f $OR_KUBERNETES_PATH/postgresql/values-eks.yaml -f profiles/$OR_PROFILE/values-postgresql-eks-load.yaml
+# The Keycloak chart constructs the public issuer from the hostname.
+helm upgrade --install --kube-context="$K_CONTEXT" --namespace default --wait --timeout 90m keycloak $OR_KUBERNETES_PATH/keycloak -f $OR_KUBERNETES_PATH/keycloak/values-haproxy.yaml \
+ -f "profiles/$OR_PROFILE/keycloak.yaml" --set-string "or.hostname=$FQDN"
-helm install --kube-context=$K_CONTEXT keycloak $OR_KUBERNETES_PATH/keycloak -f $OR_KUBERNETES_PATH/keycloak/values-haproxy.yaml \
- -f profiles/$OR_PROFILE/values-keycloak-eks-load.yaml --set-string or.hostname=$FQDN
-
-helm install --kube-context=$K_CONTEXT manager $OR_KUBERNETES_PATH/manager -f $OR_KUBERNETES_PATH/manager/values-haproxy-eks.yaml \
- -f profiles/$OR_PROFILE/values-manager-eks-load.yaml --set-string or.hostname=$FQDN --set or.setupRunOnRestart=true \
+# Override the shared Manager template's image repository directly.
+helm upgrade --install --kube-context="$K_CONTEXT" --namespace default --wait --timeout 90m manager $OR_KUBERNETES_PATH/manager -f $OR_KUBERNETES_PATH/manager/values-haproxy-eks.yaml \
+ -f "profiles/$OR_PROFILE/manager.yaml" --set-string "or.hostname=$FQDN" --set or.setupRunOnRestart=true \
--set-string image.repository=$AWS_DEVELOPERS_ACCOUNT_ID.dkr.ecr.eu-west-1.amazonaws.com/openremote/manager
diff --git a/test/load2-eks/eks-setup-load-acm.sh b/test/load2-eks/eks-setup-load-acm.sh
index 4e5ea0d0a..34cf46439 100755
--- a/test/load2-eks/eks-setup-load-acm.sh
+++ b/test/load2-eks/eks-setup-load-acm.sh
@@ -1,31 +1,32 @@
#!/bin/bash
+# Legacy topology: ACM terminates HTTPS and MQTTS at one public NLB.
+# Run from this directory, on a dedicated disposable cluster only.
. ./eks-common.sh
-envsubst < profiles/$OR_PROFILE/cluster.yaml | eksctl create cluster -f - --profile or
+envsubst '${CLUSTER_NAME} ${AWS_REGION}' < profiles/$OR_PROFILE/cluster.yaml | eksctl create cluster -f - --profile or
+aws eks update-kubeconfig --name "$CLUSTER_NAME" --region "$AWS_REGION" --profile or --alias "$K_CONTEXT"
+
+# Current component charts dynamically provision their own PVCs.
+kubectl --context "$K_CONTEXT" apply -f "$OR_KUBERNETES_PATH/cluster/eks/storage-class.yaml"
# [Installation Guide - AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/deploy/installation/)
helm repo add eks https://aws.github.io/eks-charts
helm repo update eks
-helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
- -n kube-system \
+helm install --kube-context "$K_CONTEXT" aws-load-balancer-controller eks/aws-load-balancer-controller \
+ -n kube-system --version 1.14.0 \
--set clusterName=$CLUSTER_NAME \
--set serviceAccount.create=false \
--set serviceAccount.name=aws-load-balancer-controller
-PSQL_VOLUMEID=$(aws ec2 create-volume --size $(grep "psqlVolumeSize:" values-or-setup-eks-load.yaml | awk '{print $2}' | tr -d '"Gi') \
- --availability-zone eu-west-1a --tag-specifications "ResourceType=volume,Tags=[{Key=Name,Value=psql-data}]" --query VolumeId)
-MANAGER_VOLUMEID=$(aws ec2 create-volume --size $(grep "managerVolumeSize:" values-or-setup-eks-load.yaml | awk '{print $2}' | tr -d '"Gi') \
- --availability-zone eu-west-1a --tag-specifications "ResourceType=volume,Tags=[{Key=Name,Value=manager-data}]" --query VolumeId)
-
# Wait for AWS LB ctrl to be ready
-kubectl rollout status deployment aws-load-balancer-controller -n kube-system --timeout=300s
+kubectl --context "$K_CONTEXT" rollout status deployment aws-load-balancer-controller -n kube-system --timeout=300s
-CLUSTER_DNS=$(kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}:{.spec.ports[?(@.name=="dns")].port}')
+CLUSTER_DNS=$(kubectl --context "$K_CONTEXT" get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}:{.spec.ports[?(@.name=="dns")].port}')
-helm install or-setup $OR_KUBERNETES_PATH/or-setup -f values-or-setup-eks-load.yaml --set aws.enabled=true --set aws.managerVolumeId=$MANAGER_VOLUMEID --set aws.psqlVolumeId=$PSQL_VOLUMEID
+helm install --kube-context "$K_CONTEXT" --namespace default or-setup "$OR_KUBERNETES_PATH/or-setup"
CERTIFICATE_ARN=$(aws acm request-certificate --domain-name $FQDN --validation-method DNS --profile or --query "CertificateArn" --output text)
@@ -45,13 +46,15 @@ aws route53 change-resource-record-sets \
'{"Changes": [ { "Action": "UPSERT", "ResourceRecordSet": { "Name": "'$DNS_RECORD_NAME'", "Type": "CNAME", "TTL": 300, "ResourceRecords" : [ { "Value": "'$DNS_RECORD_VALUE'" } ] } } ]}' \
--profile dnschg
-helm install proxy $OR_KUBERNETES_PATH/proxy \
- -f $OR_KUBERNETES_PATH/proxy/values-eks.yaml -f profiles/$OR_PROFILE/values-proxy-eks-load.yaml -f values-proxy-acm-load.yaml \
+aws acm wait certificate-validated --certificate-arn "$CERTIFICATE_ARN" --profile or
+
+helm install --kube-context "$K_CONTEXT" --namespace default proxy $OR_KUBERNETES_PATH/proxy \
+ -f $OR_KUBERNETES_PATH/proxy/values-eks.yaml -f "profiles/$OR_PROFILE/proxy.yaml" -f values-proxy-acm-load.yaml \
--set or.nameserver=$CLUSTER_DNS --set or.hostname=$FQDN \
--set-string 'service.http.annotations.service\.beta\.kubernetes\.io\/aws-load-balancer-ssl-cert'=$CERTIFICATE_ARN \
--set-string image.repository=$AWS_DEVELOPERS_ACCOUNT_ID.dkr.ecr.eu-west-1.amazonaws.com/openremote/proxy
-helm install postgresql $OR_KUBERNETES_PATH/postgresql -f $OR_KUBERNETES_PATH/postgresql/values-eks.yaml -f profiles/$OR_PROFILE/values-postgresql-eks-load.yaml
+helm install --kube-context "$K_CONTEXT" --namespace default postgresql $OR_KUBERNETES_PATH/postgresql -f $OR_KUBERNETES_PATH/postgresql/values-eks.yaml -f "profiles/$OR_PROFILE/postgresql.yaml"
# Waiting for the LB to be created
# AWS LB Controller only creates an Network LB if there's a service
@@ -76,10 +79,12 @@ aws route53 change-resource-record-sets \
'{"Changes": [ { "Action": "UPSERT", "ResourceRecordSet": { "Name": "'$FQDN'", "Type": "A", "AliasTarget":{ "HostedZoneId": '$HOSTED_ZONE_ID',"DNSName": '$DNS_NAME',"EvaluateTargetHealth": false} } } ]}' \
--profile dnschg
-helm install keycloak $OR_KUBERNETES_PATH/keycloak -f $OR_KUBERNETES_PATH/keycloak/values-haproxy.yaml \
- -f profiles/$OR_PROFILE/values-keycloak-eks-load.yaml --set-string or.hostname=$FQDN
-helm install manager $OR_KUBERNETES_PATH/manager -f $OR_KUBERNETES_PATH/manager/values-haproxy-eks.yaml \
- -f profiles/$OR_PROFILE/values-manager-eks-load.yaml --set-string or.hostname=$FQDN \
+# The Keycloak chart constructs the public issuer from the hostname.
+helm install --kube-context "$K_CONTEXT" --namespace default --wait --timeout 90m keycloak $OR_KUBERNETES_PATH/keycloak -f $OR_KUBERNETES_PATH/keycloak/values-haproxy.yaml \
+ -f "profiles/$OR_PROFILE/keycloak.yaml" --set-string "or.hostname=$FQDN"
+# Override the shared Manager template's image repository directly.
+helm install --kube-context "$K_CONTEXT" --namespace default --wait --timeout 90m manager $OR_KUBERNETES_PATH/manager -f $OR_KUBERNETES_PATH/manager/values-haproxy-eks.yaml \
+ -f "profiles/$OR_PROFILE/manager.yaml" --set-string "or.hostname=$FQDN" \
--set-string image.repository=$AWS_DEVELOPERS_ACCOUNT_ID.dkr.ecr.eu-west-1.amazonaws.com/openremote/manager
while ! dig +short $FQDN | grep -qE '^[0-9]'; do
@@ -88,3 +93,6 @@ while ! dig +short $FQDN | grep -qE '^[0-9]'; do
done
echo "Access the manager at https://$FQDN"
+
+echo "Retrieve the generated admin password (username: admin) with:"
+echo "kubectl --context $K_CONTEXT --namespace default get secret openremote-secret -o jsonpath='{.data.admin-password}' | base64 --decode"
diff --git a/test/load2-eks/eks-setup-load.sh b/test/load2-eks/eks-setup-load.sh
index e53cef411..217b771ab 100755
--- a/test/load2-eks/eks-setup-load.sh
+++ b/test/load2-eks/eks-setup-load.sh
@@ -1,74 +1,8 @@
-#!/bin/bash
-
-. ./eks-common.sh
-
-envsubst < profiles/$OR_PROFILE/cluster.yaml | eksctl create cluster -f - --profile or
-
-# [Installation Guide - AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/deploy/installation/)
-
-helm repo add eks https://aws.github.io/eks-charts
-helm repo update eks
-
-helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
- -n kube-system \
- --set clusterName=$CLUSTER_NAME \
- --set serviceAccount.create=false \
- --set serviceAccount.name=aws-load-balancer-controller
-
-PSQL_VOLUMEID=$(aws ec2 create-volume --size $(grep "psqlVolumeSize:" values-or-setup-eks-load.yaml | awk '{print $2}' | tr -d '"Gi') \
- --availability-zone eu-west-1a --tag-specifications "ResourceType=volume,Tags=[{Key=Name,Value=psql-data}]" --query VolumeId)
-MANAGER_VOLUMEID=$(aws ec2 create-volume --size $(grep "managerVolumeSize:" values-or-setup-eks-load.yaml | awk '{print $2}' | tr -d '"Gi') \
- --availability-zone eu-west-1a --tag-specifications "ResourceType=volume,Tags=[{Key=Name,Value=manager-data}]" --query VolumeId)
-
-# Wait for AWS LB ctrl to be ready
-kubectl rollout status deployment aws-load-balancer-controller -n kube-system --timeout=300s
-
-CLUSTER_DNS=$(kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}:{.spec.ports[?(@.name=="dns")].port}')
-
-helm install or-setup $OR_KUBERNETES_PATH/or-setup -f values-or-setup-eks-load.yaml --set aws.enabled=true --set aws.managerVolumeId=$MANAGER_VOLUMEID --set aws.psqlVolumeId=$PSQL_VOLUMEID
-
-helm install proxy $OR_KUBERNETES_PATH/proxy -f $OR_KUBERNETES_PATH/proxy/values-eks.yaml -f profiles/$OR_PROFILE/values-proxy-eks-load.yaml --set or.nameserver=$CLUSTER_DNS --set or.hostname=$FQDN
-
-helm install postgresql $OR_KUBERNETES_PATH/postgresql -f $OR_KUBERNETES_PATH/postgresql/values-eks.yaml -f profiles/$OR_PROFILE/values-postgresql-eks-load.yaml
-
-# Waiting for the LB to be created
-# AWS LB Controller only creates an Network LB if there's a service
-
-CLUSTER_VPC_ID=$(aws eks describe-cluster --profile or --name $CLUSTER_NAME --query 'cluster.resourcesVpcConfig.vpcId' --output text)
-if [ -z "$CLUSTER_VPC_ID" ] || [ "$CLUSTER_VPC_ID" = "None" ]; then
- echo "Error: Failed to retrieve VPC ID for cluster '$CLUSTER_NAME'. Aborting."
- exit 1
-fi
-
-while ! aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?VpcId=='$CLUSTER_VPC_ID' && Type=='network' && Scheme=='internet-facing']" 2>/dev/null | grep '"Code": "active"'; do
- echo "Waiting for load balancer to be created..."
- sleep 10
-done
-
-DNS_NAME=$(aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?VpcId=='$CLUSTER_VPC_ID' && Type=='network' && Scheme=='internet-facing'].DNSName | [0]")
-HOSTED_ZONE_ID=$(aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?VpcId=='$CLUSTER_VPC_ID' && Type=='network' && Scheme=='internet-facing'].CanonicalHostedZoneId | [0]")
-
-aws route53 change-resource-record-sets \
- --hosted-zone-id /hostedzone/Z08751721JH0NB6LLCB4V \
- --change-batch \
- '{"Changes": [ { "Action": "UPSERT", "ResourceRecordSet": { "Name": "'$FQDN'", "Type": "A", "AliasTarget":{ "HostedZoneId": '$HOSTED_ZONE_ID',"DNSName": '$DNS_NAME',"EvaluateTargetHealth": false} } } ]}' \
- --profile dnschg
-
-helm install keycloak $OR_KUBERNETES_PATH/keycloak -f $OR_KUBERNETES_PATH/keycloak/values-haproxy.yaml \
- -f profiles/$OR_PROFILE/values-keycloak-eks-load.yaml --set-string or.hostname=$FQDN
-helm install manager $OR_KUBERNETES_PATH/manager -f $OR_KUBERNETES_PATH/manager/values-haproxy-eks.yaml \
- -f profiles/$OR_PROFILE/values-manager-eks-load.yaml --set-string or.hostname=$FQDN \
- --set-string image.repository=$AWS_DEVELOPERS_ACCOUNT_ID.dkr.ecr.eu-west-1.amazonaws.com/openremote/manager
-
-while ! dig +short $FQDN | grep -qE '^[0-9]'; do
- echo "Waiting for DNS resolution..."
- sleep 5
-done
-
-# Now that DNS is in place, HAProxy can properly create the certificate
-kubectl exec $(kubectl get pod -l "app.kubernetes.io/name=proxy" -o name) -- sh -c "/entrypoint.sh add $FQDN" || {
- echo "Certificate generation failed, please wait a moment and manually execute the following command:"
- echo "kubectl exec \$(kubectl get pod -l \"app.kubernetes.io/name=proxy\" -o name) -- sh -c \"/entrypoint.sh add $FQDN\""
-}
-
-echo "Access the manager at https://$FQDN"
+#!/usr/bin/env bash
+
+# Retained legacy entry point. See README.md for the separate CLI lifecycles.
+printf '%s\n' \
+ 'This legacy load-test script is no longer supported.' \
+ 'Use kubernetes/or-eks-cluster for the cluster and kubernetes/or-eks-stack for each stack.' \
+ "See the README.md in this script's directory for image, profile, DNS, and cleanup configuration." >&2
+exit 1
diff --git a/test/load2-eks/profiles/2xlarge/cluster.yaml b/test/load2-eks/profiles/2xlarge/cluster.yaml
index 4768a0c6c..c91a1630b 100644
--- a/test/load2-eks/profiles/2xlarge/cluster.yaml
+++ b/test/load2-eks/profiles/2xlarge/cluster.yaml
@@ -29,6 +29,12 @@ nodeGroups:
- ${AWS_REGION}a
addons:
+ - name: vpc-cni
+ version: latest
+ attachPolicyARNs:
+ - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
+ configurationValues: '{"enableNetworkPolicy":"true"}'
+ resolveConflicts: overwrite
- name: aws-ebs-csi-driver
version: latest
wellKnownPolicies:
diff --git a/test/load2-eks/profiles/2xlarge/keycloak.yaml b/test/load2-eks/profiles/2xlarge/keycloak.yaml
new file mode 100644
index 000000000..f4dc0a8c0
--- /dev/null
+++ b/test/load2-eks/profiles/2xlarge/keycloak.yaml
@@ -0,0 +1,11 @@
+# Single-node load profiles cannot assume spare capacity for surge Pods.
+strategy:
+ type: Recreate
+
+resources:
+ limits:
+ cpu: 1500m
+ memory: 5Gi
+ requests:
+ cpu: 1200m
+ memory: 5Gi
diff --git a/test/load2-eks/profiles/2xlarge/values-manager-eks-load.yaml b/test/load2-eks/profiles/2xlarge/manager.yaml
similarity index 84%
rename from test/load2-eks/profiles/2xlarge/values-manager-eks-load.yaml
rename to test/load2-eks/profiles/2xlarge/manager.yaml
index f754b9b11..e2a53de9a 100644
--- a/test/load2-eks/profiles/2xlarge/values-manager-eks-load.yaml
+++ b/test/load2-eks/profiles/2xlarge/manager.yaml
@@ -1,4 +1,10 @@
+# Single-node load profiles cannot assume spare capacity for surge Pods.
+strategy:
+ type: Recreate
+
+# CLI: render LOAD_MANAGER_REPOSITORY; legacy scripts override image.repository.
image:
+ repository: "${LOAD_MANAGER_REPOSITORY}"
pullPolicy: Always
tag: "load2"
@@ -31,7 +37,8 @@ resources:
cpu: 3000m
memory: 13Gi
-volumeSize: "16Gi"
+persistence:
+ size: "32Gi"
or:
env:
diff --git a/test/load1-eks/values-postgresql-eks-load.yaml b/test/load2-eks/profiles/2xlarge/postgresql.yaml
similarity index 76%
rename from test/load1-eks/values-postgresql-eks-load.yaml
rename to test/load2-eks/profiles/2xlarge/postgresql.yaml
index 7b8df849e..6a07578c3 100644
--- a/test/load1-eks/values-postgresql-eks-load.yaml
+++ b/test/load2-eks/profiles/2xlarge/postgresql.yaml
@@ -1,4 +1,5 @@
-volumeSize: "16Gi"
+persistence:
+ size: "256Gi"
resources:
limits:
diff --git a/test/load2-eks/profiles/2xlarge/values-proxy-eks-load.yaml b/test/load2-eks/profiles/2xlarge/proxy.yaml
similarity index 100%
rename from test/load2-eks/profiles/2xlarge/values-proxy-eks-load.yaml
rename to test/load2-eks/profiles/2xlarge/proxy.yaml
diff --git a/test/load2-eks/profiles/2xlarge/values-keycloak-eks-load.yaml b/test/load2-eks/profiles/2xlarge/values-keycloak-eks-load.yaml
deleted file mode 100644
index ed0ca53cc..000000000
--- a/test/load2-eks/profiles/2xlarge/values-keycloak-eks-load.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-resources:
- limits:
- cpu: 1500m
- memory: 5Gi
- requests:
- cpu: 1200m
- memory: 5Gi
diff --git a/test/load2-eks/profiles/4xlarge/cluster.yaml b/test/load2-eks/profiles/4xlarge/cluster.yaml
index 79302d4d1..3e2ec96f4 100644
--- a/test/load2-eks/profiles/4xlarge/cluster.yaml
+++ b/test/load2-eks/profiles/4xlarge/cluster.yaml
@@ -29,6 +29,12 @@ nodeGroups:
- ${AWS_REGION}a
addons:
+ - name: vpc-cni
+ version: latest
+ attachPolicyARNs:
+ - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
+ configurationValues: '{"enableNetworkPolicy":"true"}'
+ resolveConflicts: overwrite
- name: aws-ebs-csi-driver
version: latest
wellKnownPolicies:
diff --git a/test/load2-eks/profiles/4xlarge/keycloak.yaml b/test/load2-eks/profiles/4xlarge/keycloak.yaml
new file mode 100644
index 000000000..05743562c
--- /dev/null
+++ b/test/load2-eks/profiles/4xlarge/keycloak.yaml
@@ -0,0 +1,11 @@
+# Single-node load profiles cannot assume spare capacity for surge Pods.
+strategy:
+ type: Recreate
+
+resources:
+ limits:
+ cpu: 2000m
+ memory: 5Gi
+ requests:
+ cpu: 2000m
+ memory: 5Gi
diff --git a/test/load2-eks/profiles/4xlarge/values-manager-eks-load.yaml b/test/load2-eks/profiles/4xlarge/manager.yaml
similarity index 84%
rename from test/load2-eks/profiles/4xlarge/values-manager-eks-load.yaml
rename to test/load2-eks/profiles/4xlarge/manager.yaml
index 1a257ee75..e6683c5ca 100644
--- a/test/load2-eks/profiles/4xlarge/values-manager-eks-load.yaml
+++ b/test/load2-eks/profiles/4xlarge/manager.yaml
@@ -1,4 +1,10 @@
+# Single-node load profiles cannot assume spare capacity for surge Pods.
+strategy:
+ type: Recreate
+
+# CLI: render LOAD_MANAGER_REPOSITORY; legacy scripts override image.repository.
image:
+ repository: "${LOAD_MANAGER_REPOSITORY}"
pullPolicy: Always
tag: "load2"
@@ -31,7 +37,8 @@ resources:
cpu: 8000m
memory: 38Gi
-volumeSize: "16Gi"
+persistence:
+ size: "32Gi"
or:
env:
diff --git a/test/load2-eks/profiles/4xlarge/values-postgresql-eks-load.yaml b/test/load2-eks/profiles/4xlarge/postgresql.yaml
similarity index 76%
rename from test/load2-eks/profiles/4xlarge/values-postgresql-eks-load.yaml
rename to test/load2-eks/profiles/4xlarge/postgresql.yaml
index a96b0791a..0933ee004 100644
--- a/test/load2-eks/profiles/4xlarge/values-postgresql-eks-load.yaml
+++ b/test/load2-eks/profiles/4xlarge/postgresql.yaml
@@ -1,4 +1,5 @@
-volumeSize: "16Gi"
+persistence:
+ size: "256Gi"
resources:
limits:
diff --git a/test/load2-eks/profiles/4xlarge/values-proxy-eks-load.yaml b/test/load2-eks/profiles/4xlarge/proxy.yaml
similarity index 100%
rename from test/load2-eks/profiles/4xlarge/values-proxy-eks-load.yaml
rename to test/load2-eks/profiles/4xlarge/proxy.yaml
diff --git a/test/load2-eks/profiles/4xlarge/values-keycloak-eks-load.yaml b/test/load2-eks/profiles/4xlarge/values-keycloak-eks-load.yaml
deleted file mode 100644
index 3ddd56060..000000000
--- a/test/load2-eks/profiles/4xlarge/values-keycloak-eks-load.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-resources:
- limits:
- cpu: 2000m
- memory: 5Gi
- requests:
- cpu: 2000m
- memory: 5Gi
diff --git a/test/load2-eks/profiles/8xlarge/cluster.yaml b/test/load2-eks/profiles/8xlarge/cluster.yaml
index 54e4dbf3e..fb6e08477 100644
--- a/test/load2-eks/profiles/8xlarge/cluster.yaml
+++ b/test/load2-eks/profiles/8xlarge/cluster.yaml
@@ -29,6 +29,12 @@ nodeGroups:
- ${AWS_REGION}a
addons:
+ - name: vpc-cni
+ version: latest
+ attachPolicyARNs:
+ - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
+ configurationValues: '{"enableNetworkPolicy":"true"}'
+ resolveConflicts: overwrite
- name: aws-ebs-csi-driver
version: latest
wellKnownPolicies:
diff --git a/test/load2-eks/profiles/8xlarge/keycloak.yaml b/test/load2-eks/profiles/8xlarge/keycloak.yaml
new file mode 100644
index 000000000..8240ba223
--- /dev/null
+++ b/test/load2-eks/profiles/8xlarge/keycloak.yaml
@@ -0,0 +1,11 @@
+# Single-node load profiles cannot assume spare capacity for surge Pods.
+strategy:
+ type: Recreate
+
+resources:
+ limits:
+ cpu: 3000m
+ memory: 6Gi
+ requests:
+ cpu: 3000m
+ memory: 6Gi
diff --git a/test/load2-eks/profiles/8xlarge/values-manager-eks-load.yaml b/test/load2-eks/profiles/8xlarge/manager.yaml
similarity index 84%
rename from test/load2-eks/profiles/8xlarge/values-manager-eks-load.yaml
rename to test/load2-eks/profiles/8xlarge/manager.yaml
index 6abd04a45..2dae1067c 100644
--- a/test/load2-eks/profiles/8xlarge/values-manager-eks-load.yaml
+++ b/test/load2-eks/profiles/8xlarge/manager.yaml
@@ -1,4 +1,10 @@
+# Single-node load profiles cannot assume spare capacity for surge Pods.
+strategy:
+ type: Recreate
+
+# CLI: render LOAD_MANAGER_REPOSITORY; legacy scripts override image.repository.
image:
+ repository: "${LOAD_MANAGER_REPOSITORY}"
pullPolicy: Always
tag: "load2"
@@ -31,7 +37,8 @@ resources:
cpu: 18000m
memory: 68Gi
-volumeSize: "16Gi"
+persistence:
+ size: "32Gi"
or:
env:
diff --git a/test/load2-eks/profiles/8xlarge/values-postgresql-eks-load.yaml b/test/load2-eks/profiles/8xlarge/postgresql.yaml
similarity index 77%
rename from test/load2-eks/profiles/8xlarge/values-postgresql-eks-load.yaml
rename to test/load2-eks/profiles/8xlarge/postgresql.yaml
index 57dce1e33..913932500 100644
--- a/test/load2-eks/profiles/8xlarge/values-postgresql-eks-load.yaml
+++ b/test/load2-eks/profiles/8xlarge/postgresql.yaml
@@ -1,4 +1,5 @@
-volumeSize: "16Gi"
+persistence:
+ size: "256Gi"
resources:
limits:
diff --git a/test/load2-eks/profiles/8xlarge/values-proxy-eks-load.yaml b/test/load2-eks/profiles/8xlarge/proxy.yaml
similarity index 100%
rename from test/load2-eks/profiles/8xlarge/values-proxy-eks-load.yaml
rename to test/load2-eks/profiles/8xlarge/proxy.yaml
diff --git a/test/load2-eks/profiles/8xlarge/values-keycloak-eks-load.yaml b/test/load2-eks/profiles/8xlarge/values-keycloak-eks-load.yaml
deleted file mode 100644
index 22ca73363..000000000
--- a/test/load2-eks/profiles/8xlarge/values-keycloak-eks-load.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-resources:
- limits:
- cpu: 3000m
- memory: 6Gi
- requests:
- cpu: 3000m
- memory: 6Gi
diff --git a/test/load2-eks/profiles/large/cluster.yaml b/test/load2-eks/profiles/large/cluster.yaml
index 80c923311..20ae209bc 100644
--- a/test/load2-eks/profiles/large/cluster.yaml
+++ b/test/load2-eks/profiles/large/cluster.yaml
@@ -29,6 +29,12 @@ nodeGroups:
- ${AWS_REGION}a
addons:
+ - name: vpc-cni
+ version: latest
+ attachPolicyARNs:
+ - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
+ configurationValues: '{"enableNetworkPolicy":"true"}'
+ resolveConflicts: overwrite
- name: aws-ebs-csi-driver
version: latest
wellKnownPolicies:
diff --git a/test/load2-eks/profiles/large/keycloak.yaml b/test/load2-eks/profiles/large/keycloak.yaml
new file mode 100644
index 000000000..ac675c207
--- /dev/null
+++ b/test/load2-eks/profiles/large/keycloak.yaml
@@ -0,0 +1,11 @@
+# Single-node load profiles cannot assume spare capacity for surge Pods.
+strategy:
+ type: Recreate
+
+resources:
+ limits:
+ cpu: 280m
+ memory: 768Mi
+ requests:
+ cpu: 280m
+ memory: 768Mi
diff --git a/test/load2-eks/profiles/large/values-manager-eks-load.yaml b/test/load2-eks/profiles/large/manager.yaml
similarity index 84%
rename from test/load2-eks/profiles/large/values-manager-eks-load.yaml
rename to test/load2-eks/profiles/large/manager.yaml
index a5644c35d..89daa76e3 100644
--- a/test/load2-eks/profiles/large/values-manager-eks-load.yaml
+++ b/test/load2-eks/profiles/large/manager.yaml
@@ -1,4 +1,10 @@
+# Single-node load profiles cannot assume spare capacity for surge Pods.
+strategy:
+ type: Recreate
+
+# CLI: render LOAD_MANAGER_REPOSITORY; legacy scripts override image.repository.
image:
+ repository: "${LOAD_MANAGER_REPOSITORY}"
pullPolicy: Always
tag: "load2"
@@ -31,7 +37,8 @@ resources:
cpu: 400m
memory: 1792Mi
-volumeSize: "16Gi"
+persistence:
+ size: "32Gi"
or:
env:
diff --git a/test/load2-eks/profiles/large/values-postgresql-eks-load.yaml b/test/load2-eks/profiles/large/postgresql.yaml
similarity index 76%
rename from test/load2-eks/profiles/large/values-postgresql-eks-load.yaml
rename to test/load2-eks/profiles/large/postgresql.yaml
index ce6c97c90..d724a0011 100644
--- a/test/load2-eks/profiles/large/values-postgresql-eks-load.yaml
+++ b/test/load2-eks/profiles/large/postgresql.yaml
@@ -1,4 +1,5 @@
-volumeSize: "16Gi"
+persistence:
+ size: "256Gi"
resources:
limits:
diff --git a/test/load2-eks/profiles/large/values-proxy-eks-load.yaml b/test/load2-eks/profiles/large/proxy.yaml
similarity index 100%
rename from test/load2-eks/profiles/large/values-proxy-eks-load.yaml
rename to test/load2-eks/profiles/large/proxy.yaml
diff --git a/test/load2-eks/profiles/large/values-keycloak-eks-load.yaml b/test/load2-eks/profiles/large/values-keycloak-eks-load.yaml
deleted file mode 100644
index eeceaf6e6..000000000
--- a/test/load2-eks/profiles/large/values-keycloak-eks-load.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-resources:
- limits:
- cpu: 280m
- memory: 768Mi
- requests:
- cpu: 280m
- memory: 768Mi
diff --git a/test/load2-eks/profiles/xlarge-minimal/cluster.yaml b/test/load2-eks/profiles/xlarge-minimal/cluster.yaml
index af9c2800d..a2fdbabcd 100644
--- a/test/load2-eks/profiles/xlarge-minimal/cluster.yaml
+++ b/test/load2-eks/profiles/xlarge-minimal/cluster.yaml
@@ -29,6 +29,12 @@ nodeGroups:
- ${AWS_REGION}a
addons:
+ - name: vpc-cni
+ version: latest
+ attachPolicyARNs:
+ - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
+ configurationValues: '{"enableNetworkPolicy":"true"}'
+ resolveConflicts: overwrite
- name: aws-ebs-csi-driver
version: latest
wellKnownPolicies:
diff --git a/test/load2-eks/profiles/xlarge-minimal/keycloak.yaml b/test/load2-eks/profiles/xlarge-minimal/keycloak.yaml
new file mode 100644
index 000000000..ac675c207
--- /dev/null
+++ b/test/load2-eks/profiles/xlarge-minimal/keycloak.yaml
@@ -0,0 +1,11 @@
+# Single-node load profiles cannot assume spare capacity for surge Pods.
+strategy:
+ type: Recreate
+
+resources:
+ limits:
+ cpu: 280m
+ memory: 768Mi
+ requests:
+ cpu: 280m
+ memory: 768Mi
diff --git a/test/load2-eks/profiles/xlarge-minimal/values-manager-eks-load.yaml b/test/load2-eks/profiles/xlarge-minimal/manager.yaml
similarity index 84%
rename from test/load2-eks/profiles/xlarge-minimal/values-manager-eks-load.yaml
rename to test/load2-eks/profiles/xlarge-minimal/manager.yaml
index bf5a9b77f..4c870ecfb 100644
--- a/test/load2-eks/profiles/xlarge-minimal/values-manager-eks-load.yaml
+++ b/test/load2-eks/profiles/xlarge-minimal/manager.yaml
@@ -1,4 +1,10 @@
+# Single-node load profiles cannot assume spare capacity for surge Pods.
+strategy:
+ type: Recreate
+
+# CLI: render LOAD_MANAGER_REPOSITORY; legacy scripts override image.repository.
image:
+ repository: "${LOAD_MANAGER_REPOSITORY}"
pullPolicy: Always
tag: "load2"
@@ -31,7 +37,8 @@ resources:
cpu: 400m
memory: 3840Mi
-volumeSize: "16Gi"
+persistence:
+ size: "32Gi"
or:
env:
diff --git a/test/load2-eks/profiles/xlarge-minimal/values-postgresql-eks-load.yaml b/test/load2-eks/profiles/xlarge-minimal/postgresql.yaml
similarity index 76%
rename from test/load2-eks/profiles/xlarge-minimal/values-postgresql-eks-load.yaml
rename to test/load2-eks/profiles/xlarge-minimal/postgresql.yaml
index ce6c97c90..d724a0011 100644
--- a/test/load2-eks/profiles/xlarge-minimal/values-postgresql-eks-load.yaml
+++ b/test/load2-eks/profiles/xlarge-minimal/postgresql.yaml
@@ -1,4 +1,5 @@
-volumeSize: "16Gi"
+persistence:
+ size: "256Gi"
resources:
limits:
diff --git a/test/load2-eks/profiles/xlarge-minimal/values-proxy-eks-load.yaml b/test/load2-eks/profiles/xlarge-minimal/proxy.yaml
similarity index 100%
rename from test/load2-eks/profiles/xlarge-minimal/values-proxy-eks-load.yaml
rename to test/load2-eks/profiles/xlarge-minimal/proxy.yaml
diff --git a/test/load2-eks/profiles/xlarge-minimal/values-keycloak-eks-load.yaml b/test/load2-eks/profiles/xlarge-minimal/values-keycloak-eks-load.yaml
deleted file mode 100644
index eeceaf6e6..000000000
--- a/test/load2-eks/profiles/xlarge-minimal/values-keycloak-eks-load.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-resources:
- limits:
- cpu: 280m
- memory: 768Mi
- requests:
- cpu: 280m
- memory: 768Mi
diff --git a/test/load2-eks/profiles/xlarge/cluster.yaml b/test/load2-eks/profiles/xlarge/cluster.yaml
index af9c2800d..a2fdbabcd 100644
--- a/test/load2-eks/profiles/xlarge/cluster.yaml
+++ b/test/load2-eks/profiles/xlarge/cluster.yaml
@@ -29,6 +29,12 @@ nodeGroups:
- ${AWS_REGION}a
addons:
+ - name: vpc-cni
+ version: latest
+ attachPolicyARNs:
+ - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
+ configurationValues: '{"enableNetworkPolicy":"true"}'
+ resolveConflicts: overwrite
- name: aws-ebs-csi-driver
version: latest
wellKnownPolicies:
diff --git a/test/load2-eks/profiles/xlarge/keycloak.yaml b/test/load2-eks/profiles/xlarge/keycloak.yaml
new file mode 100644
index 000000000..ae573fe1f
--- /dev/null
+++ b/test/load2-eks/profiles/xlarge/keycloak.yaml
@@ -0,0 +1,11 @@
+# Single-node load profiles cannot assume spare capacity for surge Pods.
+strategy:
+ type: Recreate
+
+resources:
+ limits:
+ cpu: 500m
+ memory: 1Gi
+ requests:
+ cpu: 500m
+ memory: 1Gi
diff --git a/test/load2-eks/profiles/xlarge/values-manager-eks-load.yaml b/test/load2-eks/profiles/xlarge/manager.yaml
similarity index 84%
rename from test/load2-eks/profiles/xlarge/values-manager-eks-load.yaml
rename to test/load2-eks/profiles/xlarge/manager.yaml
index cde96d4e0..7c9d24ad9 100644
--- a/test/load2-eks/profiles/xlarge/values-manager-eks-load.yaml
+++ b/test/load2-eks/profiles/xlarge/manager.yaml
@@ -1,4 +1,10 @@
+# Single-node load profiles cannot assume spare capacity for surge Pods.
+strategy:
+ type: Recreate
+
+# CLI: render LOAD_MANAGER_REPOSITORY; legacy scripts override image.repository.
image:
+ repository: "${LOAD_MANAGER_REPOSITORY}"
pullPolicy: Always
tag: "load2"
@@ -31,7 +37,8 @@ resources:
cpu: 1500m
memory: 8Gi
-volumeSize: "16Gi"
+persistence:
+ size: "32Gi"
or:
env:
diff --git a/test/load2-eks/profiles/xlarge/values-postgresql-eks-load.yaml b/test/load2-eks/profiles/xlarge/postgresql.yaml
similarity index 76%
rename from test/load2-eks/profiles/xlarge/values-postgresql-eks-load.yaml
rename to test/load2-eks/profiles/xlarge/postgresql.yaml
index b46ba7c2a..fdecc3d64 100644
--- a/test/load2-eks/profiles/xlarge/values-postgresql-eks-load.yaml
+++ b/test/load2-eks/profiles/xlarge/postgresql.yaml
@@ -1,4 +1,5 @@
-volumeSize: "16Gi"
+persistence:
+ size: "256Gi"
resources:
limits:
diff --git a/test/load2-eks/profiles/xlarge/values-proxy-eks-load.yaml b/test/load2-eks/profiles/xlarge/proxy.yaml
similarity index 100%
rename from test/load2-eks/profiles/xlarge/values-proxy-eks-load.yaml
rename to test/load2-eks/profiles/xlarge/proxy.yaml
diff --git a/test/load2-eks/profiles/xlarge/values-keycloak-eks-load.yaml b/test/load2-eks/profiles/xlarge/values-keycloak-eks-load.yaml
deleted file mode 100644
index c9c050969..000000000
--- a/test/load2-eks/profiles/xlarge/values-keycloak-eks-load.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-resources:
- limits:
- cpu: 500m
- memory: 1Gi
- requests:
- cpu: 500m
- memory: 1Gi
diff --git a/test/load2-eks/values-or-setup-eks-load.yaml b/test/load2-eks/values-or-setup-eks-load.yaml
deleted file mode 100644
index 4cc16003e..000000000
--- a/test/load2-eks/values-or-setup-eks-load.yaml
+++ /dev/null
@@ -1,6 +0,0 @@
-managerVolumeSize: "32Gi"
-psqlVolumeSize: "256Gi"
-
-aws:
- # Set enabled to true if deployed under AWS
- enabled: true
diff --git a/test/load2-eks/values-proxy-acm-load.yaml b/test/load2-eks/values-proxy-acm-load.yaml
index 34b63abbf..e3b13c3b9 100644
--- a/test/load2-eks/values-proxy-acm-load.yaml
+++ b/test/load2-eks/values-proxy-acm-load.yaml
@@ -1,3 +1,4 @@
+# Legacy ACM/NLB experiment only; never pass this overlay to or-stack.
service:
http:
annotations:
@@ -7,10 +8,14 @@ service:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-west-1:<openremote account id>:certificate/..."
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443,8883"
+ mqtts:
+ enabled: true
+
+persistence:
+ enabled: false
+
or:
+ disableAcme: true
haproxyConfig: /etc/haproxy/haproxy-edge-terminated-tls.cfg
httpsPort: 8080
mqttPort: 1883
- env:
- - name: DISABLE_ACME
- value: "true"
diff --git a/test/or-eks-load b/test/or-eks-load
new file mode 100755
index 000000000..f1dc23a3f
--- /dev/null
+++ b/test/or-eks-load
@@ -0,0 +1,284 @@
+#!/usr/bin/env bash
+# Load-test orchestration. The EKS CLIs own cluster and stack lifecycle mechanics.
+set -Eeuo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+TEMP_DIR=""
+cleanup() { if [ -n "$TEMP_DIR" ]; then rm -rf "$TEMP_DIR"; fi; }
+trap cleanup EXIT
+trap 'exit 130' INT
+trap 'exit 143' TERM
+log() { printf '[or-eks-load] %s\n' "$*"; }
+die() { printf '[or-eks-load] ERROR: %s\n' "$*" >&2; exit 1; }
+require() { command -v "$1" >/dev/null || die "Required command '$1' was not found."; }
+usage() {
+ cat <<'HELP'
+Usage: test/or-eks-load <command> --config <deployment.env> [options]
+
+ up Reconcile DNS IAM, create/reconcile the cluster, deploy and verify.
+ cluster-up Reconcile DNS IAM and cluster only; verify ExternalDNS access.
+ deploy Deploy on an existing configured cluster; verify DNS access first.
+ status Show stack status.
+ credentials Retrieve stack credentials.
+ uninstall Stop the stack, retaining data and credentials.
+ destroy-stack Delete the stack and its data; requires --confirm <stack-name>.
+ destroy-cluster Delete an empty cluster; requires --confirm <cluster-name>.
+ prepare Write editable values; requires --values-dir <new-directory>.
+
+The config is a trusted Bash file. See test/load2-eks/deployment.env.example.
+AWS credentials come from the normal credential chain unless CLUSTER_AWS_PROFILE
+or DNS_AWS_PROFILE is configured. Never put access keys in the config.
+Relative LOAD_VALUES_DIR paths are relative to the config file. prepare's output
+path is relative to the current directory. No command builds or pushes images.
+HELP
+}
+
+COMMAND="${1:---help}"
+case "$COMMAND" in -h|--help) usage; exit 0;; esac
+shift
+CONFIG_FILE=""; CONFIRM=""; OUTPUT_DIR=""
+while [ "$#" -gt 0 ]; do
+ case "$1" in
+ --config|--confirm|--values-dir)
+ [ "$#" -ge 2 ] && [ -n "$2" ] || die "$1 requires a value."
+ case "$1" in --config) CONFIG_FILE="$2";; --confirm) CONFIRM="$2";; --values-dir) OUTPUT_DIR="$2";; esac
+ shift 2;;
+ -h|--help) usage; exit 0;;
+ *) die "Unknown option '$1'. Use --help; EKS CLI options belong in the config.";;
+ esac
+done
+case "$COMMAND" in up|cluster-up|deploy|status|credentials|uninstall|destroy-stack|destroy-cluster|prepare) ;; *) die "Unknown command '$COMMAND'.";; esac
+[ -f "$CONFIG_FILE" ] || die "Supply --config with a deployment configuration file."
+CONFIG_FILE="$(cd "$(dirname "$CONFIG_FILE")" && pwd)/$(basename "$CONFIG_FILE")"
+
+# Config owns these settings; old exports from a manual deployment must not leak in.
+LOAD_TEST=""; LOAD_PROFILE=""; CLUSTER_NAME=""; STACK_NAME=""
+AWS_REGION="eu-west-1"; LOAD_HOSTNAME=""; LOAD_MANAGER_REPOSITORY=""
+CLUSTER_AWS_PROFILE=""; DNS_AWS_PROFILE=""; DNS_ZONE_ID=""; DNS_DOMAIN=""
+LOAD_VALUES_DIR=""; LOAD_EXPOSURE="haproxy"; LOAD_MQTTS_HOSTNAME=""
+LOAD_TIMEOUT="90m"; CERTIFICATE_ARN=""; MQTTS_CERTIFICATE_ARN=""
+source "$CONFIG_FILE"
+# The low-level CLIs support their own environment overrides. This config is the
+# sole source for this helper; direct CLI invocation remains available separately.
+for variable in $(compgen -v OR_EKS_ || true) $(compgen -v OR_STACK_ || true); do unset "$variable"; done
+
+profile_dir() {
+ case "$LOAD_TEST" in
+ load1) [ -z "$LOAD_PROFILE" ] || die "load1 has one fixed profile; omit LOAD_PROFILE."; PROFILE_DIR="$SCRIPT_DIR/load1-eks";;
+ load2)
+ LOAD_PROFILE="${LOAD_PROFILE:-large}"
+ case "$LOAD_PROFILE" in large|xlarge-minimal|xlarge|2xlarge|4xlarge|8xlarge) ;; *) die "Unknown load2 profile '$LOAD_PROFILE'.";; esac
+ PROFILE_DIR="$SCRIPT_DIR/load2-eks/profiles/$LOAD_PROFILE";;
+ *) die "LOAD_TEST must be load1 or load2.";;
+ esac
+}
+make_temp() { if [ -z "$TEMP_DIR" ]; then TEMP_DIR="$(mktemp -d "${TMPDIR:-/tmp}/or-eks-load.XXXXXX")"; fi; }
+render_values() {
+ local directory="$1" component
+ profile_dir
+ [[ "$LOAD_MANAGER_REPOSITORY" =~ ^[A-Za-z0-9][A-Za-z0-9._:/-]*$ ]] &&
+ [[ "$LOAD_MANAGER_REPOSITORY" = */* && "$LOAD_MANAGER_REPOSITORY" != *//* && "$LOAD_MANAGER_REPOSITORY" != */ && "${LOAD_MANAGER_REPOSITORY##*/}" != *:* ]] || die "Set LOAD_MANAGER_REPOSITORY to the full image repository (including /openremote/manager), without a scheme or tag."
+ require envsubst
+ mkdir -p "$(dirname "$directory")"
+ mkdir "$directory" || die "Values directory already exists: $directory (not overwritten)."
+ for component in manager postgresql keycloak proxy; do
+ LOAD_MANAGER_REPOSITORY="$LOAD_MANAGER_REPOSITORY" envsubst '${LOAD_MANAGER_REPOSITORY}' < "$PROFILE_DIR/$component.yaml" > "$directory/$component.yaml"
+ done
+}
+if [ "$COMMAND" = prepare ]; then
+ [ -n "$OUTPUT_DIR" ] || die "prepare requires --values-dir <new-directory>."
+ render_values "$OUTPUT_DIR"
+ log "Prepared values in $OUTPUT_DIR. Set LOAD_VALUES_DIR in $CONFIG_FILE to reuse them."
+ exit 0
+fi
+[ -z "$OUTPUT_DIR" ] || die "--values-dir is only for prepare; use LOAD_VALUES_DIR in the config for deployment."
+[[ "$CLUSTER_NAME" =~ ^[A-Za-z][A-Za-z0-9-]*$ ]] || die "Set CLUSTER_NAME to a valid EKS cluster name."
+[[ "$AWS_REGION" =~ ^[a-z0-9-]+$ ]] || die "Invalid AWS_REGION."
+if [ "$COMMAND" != cluster-up ] && [ "$COMMAND" != destroy-cluster ]; then
+ [[ "$STACK_NAME" =~ ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ ]] && [ "${#STACK_NAME}" -le 63 ] || die "Set STACK_NAME to a valid Kubernetes namespace."
+ case "$STACK_NAME" in default|kube-system|kube-public|kube-node-lease) die "STACK_NAME names a reserved namespace.";; esac
+fi
+K_CONTEXT="$CLUSTER_NAME@$AWS_REGION"
+IRSA_ROLE_NAME="openremote-${CLUSTER_NAME}-${AWS_REGION}-external-dns"
+[ "${#IRSA_ROLE_NAME}" -le 64 ] || die "CLUSTER_NAME is too long for its IAM role name."
+
+aws_cluster() {
+ if [ -n "$CLUSTER_AWS_PROFILE" ]; then aws --profile "$CLUSTER_AWS_PROFILE" --region "$AWS_REGION" "$@";
+ else aws --region "$AWS_REGION" "$@"; fi
+}
+aws_dns() {
+ if [ -n "$DNS_AWS_PROFILE" ]; then aws --profile "$DNS_AWS_PROFILE" --region "$AWS_REGION" "$@";
+ else aws_cluster "$@"; fi
+}
+cluster_cli() {
+ "$ROOT/kubernetes/or-eks-cluster" "$@" --name "$CLUSTER_NAME" --region "$AWS_REGION" --profile "$CLUSTER_AWS_PROFILE"
+}
+stack_cli() {
+ "$ROOT/kubernetes/or-eks-stack" "$@" --name "$STACK_NAME" --cluster "$CLUSTER_NAME" --region "$AWS_REGION" --profile "$CLUSTER_AWS_PROFILE"
+}
+case "$COMMAND" in
+ status|credentials|uninstall) stack_cli "$COMMAND"; exit;;
+ destroy-stack)
+ [ "$CONFIRM" = "$STACK_NAME" ] || die "Use --confirm $STACK_NAME to delete this stack and its data."
+ stack_cli destroy --confirm "$CONFIRM"; exit;;
+ destroy-cluster)
+ [ "$CONFIRM" = "$CLUSTER_NAME" ] || die "Use --confirm $CLUSTER_NAME to delete this empty cluster."
+ cluster_cli destroy --confirm "$CONFIRM"
+ log "The DNS-account IAM stack is retained for reuse; no other stacks were deleted."
+ exit;;
+esac
+
+for tool in aws jq kubectl helm; do require "$tool"; done
+if [ "$COMMAND" != deploy ]; then require eksctl; require envsubst; fi
+[[ "$LOAD_HOSTNAME" =~ ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)+$ ]] || die "Set LOAD_HOSTNAME to a lowercase DNS hostname, without scheme or path."
+case "$LOAD_EXPOSURE" in haproxy|ingress) ;; *) die "LOAD_EXPOSURE must be haproxy or ingress.";; esac
+if [ "$LOAD_EXPOSURE" = ingress ]; then
+ [[ "$LOAD_MQTTS_HOSTNAME" =~ ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)+$ ]] &&
+ [ "$LOAD_MQTTS_HOSTNAME" != "$LOAD_HOSTNAME" ] || die "Ingress requires a separate valid LOAD_MQTTS_HOSTNAME."
+elif [ -n "$CERTIFICATE_ARN" ] || [ -n "$MQTTS_CERTIFICATE_ARN" ]; then
+ die "ACM certificate settings require LOAD_EXPOSURE=ingress."
+fi
+profile_dir
+make_temp
+umask 077
+if [ "$COMMAND" != cluster-up ]; then
+ require curl
+ require dig
+ require python3 # The underlying CLI uses Python for its MQTTS TLS probe.
+ if [ -n "$LOAD_VALUES_DIR" ]; then
+ case "$LOAD_VALUES_DIR" in /*) ;; *) LOAD_VALUES_DIR="$(dirname "$CONFIG_FILE")/$LOAD_VALUES_DIR";; esac
+ else
+ LOAD_VALUES_DIR="$TEMP_DIR/values"
+ render_values "$LOAD_VALUES_DIR"
+ fi
+ for component in manager postgresql keycloak proxy; do
+ [ -f "$LOAD_VALUES_DIR/$component.yaml" ] || die "Missing prepared values: $component.yaml"
+ done
+ if grep -F '${LOAD_MANAGER_REPOSITORY}' "$LOAD_VALUES_DIR/manager.yaml" >/dev/null; then die "Unrendered repository in prepared values; use prepare first."; fi
+ # Catch URL composition regressions before provisioning or waiting on AWS.
+ rendered="$(helm template keycloak "$ROOT/kubernetes/keycloak" --values "$LOAD_VALUES_DIR/keycloak.yaml" --set-string "or.hostname=https://$LOAD_HOSTNAME/auth")"
+ actual="$(awk '/name: KC_HOSTNAME$/ {getline; gsub(/^[[:space:]]*value: "|"$/, ""); print}' <<< "$rendered")"
+ [ "$actual" = "https://$LOAD_HOSTNAME/auth" ] || die "Keycloak chart renders an invalid public URL: $actual"
+fi
+
+CLUSTER_ACCOUNT_ID="$(aws_cluster sts get-caller-identity --query Account --output text)"
+[[ "$CLUSTER_ACCOUNT_ID" =~ ^[0-9]{12}$ ]] || die "Could not identify the cluster AWS account."
+log "Cluster account: $CLUSTER_ACCOUNT_ID; cluster: $CLUSTER_NAME; region: $AWS_REGION."
+CLUSTER_EXISTS=false
+if cluster_status="$(aws_cluster eks describe-cluster --name "$CLUSTER_NAME" --query cluster.status --output text 2>&1)"; then
+ CLUSTER_EXISTS=true
+ case "$cluster_status" in ACTIVE) ;; CREATING|UPDATING) aws_cluster eks wait cluster-active --name "$CLUSTER_NAME";; *) die "Cluster status is $cluster_status; inspect it before retrying.";; esac
+else
+ case "$cluster_status" in *ResourceNotFoundException*) ;; *) die "Could not inspect cluster: $cluster_status";; esac
+fi
+
+# Read existing shared DNS configuration before reconciling it. In particular,
+# never change TXT ownership or the managed zone underneath existing stacks.
+EXISTING_DNS=""
+if [ "$CLUSTER_EXISTS" = true ]; then
+ cluster_cli kubeconfig
+ releases="$(helm list --all --kube-context "$K_CONTEXT" --namespace kube-system --filter '^openremote-external-dns$' --output json)"
+ if [ "$(jq 'length' <<< "$releases")" -gt 0 ]; then
+ EXISTING_DNS="$(helm get values openremote-external-dns --kube-context "$K_CONTEXT" --namespace kube-system --output json)"
+ fi
+fi
+check_domain() {
+ [[ "$LOAD_HOSTNAME" = *."$DNS_DOMAIN" ]] || die "LOAD_HOSTNAME must be below DNS domain $DNS_DOMAIN."
+ if [ "$LOAD_EXPOSURE" = ingress ]; then
+ [[ "$LOAD_MQTTS_HOSTNAME" = *."$DNS_DOMAIN" ]] || die "LOAD_MQTTS_HOSTNAME must be below DNS domain $DNS_DOMAIN."
+ fi
+}
+if [ "$COMMAND" = deploy ]; then
+ [ "$CLUSTER_EXISTS" = true ] && [ -n "$EXISTING_DNS" ] || die "deploy requires an existing cluster with managed ExternalDNS; run up first."
+ actual_zone="$(jq -er '.extraArgs["zone-id-filter"]' <<< "$EXISTING_DNS")"
+ actual_domain="$(jq -er '.domainFilters[0]' <<< "$EXISTING_DNS")"
+ [ -z "$DNS_ZONE_ID" ] || [ "${DNS_ZONE_ID#/hostedzone/}" = "$actual_zone" ] || die "Configured DNS_ZONE_ID differs from the cluster."
+ [ -z "$DNS_DOMAIN" ] || [ "${DNS_DOMAIN%.}" = "$actual_domain" ] || die "Configured DNS_DOMAIN differs from the cluster."
+ DNS_ZONE_ID="$actual_zone"; DNS_DOMAIN="$actual_domain"
+ DNS_ROLE_ARN="$(jq -er '.extraArgs["aws-assume-role"]' <<< "$EXISTING_DNS")"
+ DNS_EXTERNAL_ID="$(jq -r '.extraArgs["aws-assume-role-external-id"] // ""' <<< "$EXISTING_DNS")"
+ [ "$(jq -r '.extraArgs["dry-run"] // false' <<< "$EXISTING_DNS")" = false ] || die "ExternalDNS is in dry-run mode; reconcile it before deploying."
+ check_domain
+else
+ DNS_ACCOUNT_ID="$(aws_dns sts get-caller-identity --query Account --output text)"
+ [[ "$DNS_ACCOUNT_ID" =~ ^[0-9]{12}$ ]] || die "Could not identify the DNS AWS account."
+ zones="$(aws_dns route53 list-hosted-zones --output json)"
+ candidates="$(jq --arg host "$LOAD_HOSTNAME" --arg id "${DNS_ZONE_ID#/hostedzone/}" --arg domain "${DNS_DOMAIN%.}" '
+ [.HostedZones[] | select(.Config.PrivateZone == false) | .Name |= rtrimstr(".") | .Id |= sub("^/hostedzone/"; "") |
+ select(.Name as $name | $host | endswith("." + $name)) |
+ select($id == "" or .Id == $id) | select($domain == "" or .Name == $domain)] |
+ sort_by(.Name | length) | reverse | if length > 0 then .[0].Name as $name | map(select(.Name == $name)) else [] end' <<< "$zones")"
+ [ "$(jq 'length' <<< "$candidates")" -gt 0 ] || die "No matching public DNS zone in account $DNS_ACCOUNT_ID. Check DNS_AWS_PROFILE, DNS_ZONE_ID and LOAD_HOSTNAME."
+ if [ "$(jq 'length' <<< "$candidates")" -ne 1 ]; then
+ jq -r '.[] | "Candidate: \(.Id) \(.Name)"' <<< "$candidates" >&2
+ die "Multiple matching zones; set DNS_ZONE_ID in the config."
+ fi
+ DNS_ZONE_ID="$(jq -r '.[0].Id' <<< "$candidates")"; DNS_DOMAIN="$(jq -r '.[0].Name' <<< "$candidates")"
+ check_domain
+ DNS_OWNER_ID="${CLUSTER_ACCOUNT_ID}-${CLUSTER_NAME}-${AWS_REGION}"
+ if [ -n "$EXISTING_DNS" ]; then
+ [ "$(jq -r '.extraArgs["zone-id-filter"]' <<< "$EXISTING_DNS")" = "$DNS_ZONE_ID" ] &&
+ [ "$(jq -r '.domainFilters[0]' <<< "$EXISTING_DNS")" = "$DNS_DOMAIN" ] || die "Existing cluster manages a different DNS zone/domain; use its configuration or a separate cluster."
+ DNS_OWNER_ID="$(jq -er '.txtOwnerId | select(length > 0)' <<< "$EXISTING_DNS")"
+ fi
+ # External IDs identify a relationship, not a secret. This stable, account-
+ # qualified value and the exact source-role condition survive retries/machines.
+ DNS_EXTERNAL_ID="openremote-${CLUSTER_ACCOUNT_ID}-${CLUSTER_NAME}-${AWS_REGION}"
+ DNS_IAM_STACK="or-load-${CLUSTER_ACCOUNT_ID}-${CLUSTER_NAME}-${AWS_REGION}-dns"
+ log "Reconciling DNS role in account $DNS_ACCOUNT_ID for $DNS_DOMAIN ($DNS_ZONE_ID)."
+ aws_dns cloudformation deploy --stack-name "$DNS_IAM_STACK" --template-file "$ROOT/kubernetes/cluster/eks/external-dns-route53-role.yaml" --capabilities CAPABILITY_NAMED_IAM --no-fail-on-empty-changeset --parameter-overrides "HostedZoneId=$DNS_ZONE_ID" "DomainName=$DNS_DOMAIN" "ClusterAccountId=$CLUSTER_ACCOUNT_ID" "ClusterExternalDNSRoleName=$IRSA_ROLE_NAME" "ExternalId=$DNS_EXTERNAL_ID"
+ DNS_ROLE_ARN="$(aws_dns cloudformation describe-stacks --stack-name "$DNS_IAM_STACK" --query 'Stacks[0].Outputs[?OutputKey==`RoleArn`].OutputValue | [0]' --output text)"
+ [[ "$DNS_ROLE_ARN" = arn:aws:iam::"$DNS_ACCOUNT_ID":role/* ]] || die "DNS IAM stack did not return a valid role ARN."
+ cluster_action=apply
+ if [ "$CLUSTER_EXISTS" = false ]; then cluster_action=create; fi
+ cluster_cli "$cluster_action" --config "$PROFILE_DIR/cluster.yaml" --external-dns-zone-id "$DNS_ZONE_ID" --external-dns-domain "$DNS_DOMAIN" --external-dns-role-arn "$DNS_ROLE_ARN" --external-dns-external-id "$DNS_EXTERNAL_ID" --external-dns-owner-id "$DNS_OWNER_ID" --external-dns-irsa-role-name "$IRSA_ROLE_NAME"
+fi
+
+# Test the controller's actual credential chain, not the operator's admin access.
+# The short-lived service-account JWT and STS results stay in private temp files
+# or subprocess-local variables and are never printed.
+set_credentials() {
+ AWS_ACCESS_KEY_ID="$(jq -er '.Credentials.AccessKeyId' <<< "$1")" || return
+ AWS_SECRET_ACCESS_KEY="$(jq -er '.Credentials.SecretAccessKey' <<< "$1")" || return
+ AWS_SESSION_TOKEN="$(jq -er '.Credentials.SessionToken' <<< "$1")" || return
+ export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
+}
+probe_dns() (
+ set -e
+ unset AWS_PROFILE AWS_DEFAULT_PROFILE AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN AWS_SECURITY_TOKEN AWS_WEB_IDENTITY_TOKEN_FILE AWS_ROLE_ARN AWS_ROLE_SESSION_NAME
+ credentials="$(aws --region "$AWS_REGION" sts assume-role-with-web-identity --no-sign-request --role-arn "$SOURCE_ROLE_ARN" --role-session-name or-load-preflight --web-identity-token "file://$TEMP_DIR/token" --duration-seconds 900 --output json)" || exit
+ set_credentials "$credentials" || exit
+ assume_args=(sts assume-role --role-arn "$DNS_ROLE_ARN" --role-session-name or-load-dns-preflight --duration-seconds 900 --output json)
+ if [ -n "$DNS_EXTERNAL_ID" ]; then assume_args+=(--external-id "$DNS_EXTERNAL_ID"); fi
+ credentials="$(aws --region "$AWS_REGION" "${assume_args[@]}")" || exit
+ set_credentials "$credentials" || exit
+ aws --region "$AWS_REGION" route53 list-hosted-zones --max-items 1 --output json >/dev/null || exit
+ aws --region "$AWS_REGION" route53 list-resource-record-sets --hosted-zone-id "$DNS_ZONE_ID" --max-items 1 --output json >/dev/null
+)
+log "Checking DNS access as the ExternalDNS service account."
+SOURCE_ROLE_ARN="$(kubectl --context "$K_CONTEXT" --namespace kube-system get serviceaccount external-dns -o 'jsonpath={.metadata.annotations.eks\.amazonaws\.com/role-arn}')"
+[ -n "$SOURCE_ROLE_ARN" ] || die "ExternalDNS service account has no IAM role annotation."
+kubectl --context "$K_CONTEXT" --namespace kube-system create token external-dns --audience sts.amazonaws.com --duration=10m > "$TEMP_DIR/token"
+probe_ok=false
+for attempt in 1 2 3 4 5 6; do
+ if probe_dns 2> "$TEMP_DIR/probe-error"; then probe_ok=true; break; fi
+ if [ "$attempt" -lt 6 ]; then sleep 5; fi
+done
+if [ "$probe_ok" != true ]; then
+ cat "$TEMP_DIR/probe-error" >&2
+ die "ExternalDNS cannot assume its roles or read Route 53. No stack deployment was attempted. Check IAM trust, external ID and account policies."
+fi
+log "ExternalDNS credential chain and Route 53 reads verified."
+if [ "$COMMAND" = cluster-up ]; then exit 0; fi
+
+stack_args=(apply --hostname "$LOAD_HOSTNAME" --exposure "$LOAD_EXPOSURE" --mqtts --values-dir "$LOAD_VALUES_DIR" --timeout "$LOAD_TIMEOUT")
+if [ "$LOAD_EXPOSURE" = ingress ]; then stack_args+=(--mqtts-hostname "$LOAD_MQTTS_HOSTNAME"); fi
+if [ -n "$CERTIFICATE_ARN" ]; then stack_args+=(--certificate-mode existing --certificate-arn "$CERTIFICATE_ARN"); fi
+if [ -n "$MQTTS_CERTIFICATE_ARN" ]; then stack_args+=(--mqtts-certificate-mode existing --mqtts-certificate-arn "$MQTTS_CERTIFICATE_ARN"); fi
+stack_cli "${stack_args[@]}"
+issuer="https://$LOAD_HOSTNAME/auth/realms/master"
+discovery="$(curl --fail --silent --show-error --connect-timeout 10 --max-time 30 "$issuer/.well-known/openid-configuration")"
+jq -e --arg issuer "$issuer" '.issuer == $issuer and (.authorization_endpoint | startswith($issuer + "/"))' <<< "$discovery" >/dev/null || die "Keycloak advertises an incorrect issuer or login URL; inspect KC_HOSTNAME before running a load test."
+log "Ready: https://$LOAD_HOSTNAME/manager/"
+printf 'Retrieve credentials: %q credentials --config %q\n' "$SCRIPT_DIR/or-eks-load" "$CONFIG_FILE"