Skip to content

armgit / core

Public Imported from GitHub
0

100% open-source IoT Platform - Integrate your devices, add intelligence, build custom apps

Configuration, script and documentation to deploy under AWS EKS (#1968)

Eric Bariaux201f5134defb
diff --git a/kubernetes/README-AWS.md b/kubernetes/README-AWS.md
new file mode 100644
index 000000000..3e948d1fd
--- /dev/null
+++ b/kubernetes/README-AWS.md
@@ -0,0 +1,187 @@
+# Kubernetes deployment under AWS (EKS)
+
+## TL;DR
+
+Requirements: you need to have kubectl, helm, [aws cli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and [eksctl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html#eksctl-install-update) installed beforehand.
+
+The scripts have been tested within the openremote account.
+You must get proper credentials and set the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN variable before running them.
+
+The `eks-setup-haproxy.sh` script creates a new EKS cluster from scratch and deploys an OR stack in it.
+The `eks-cleanup-haproxy.sh` script deletes all the components that have been created by the above script.
+
+The hostname is defined as an environment variable in the script, by default, it is testmanager.openremote.app
+
+See comments within the script for more information.
+
+## Setup
+
+We have two different options for deployment: using a HAProxy container or not.
+There are pros and cons to both approaches, see [Networking](#Networking) section for more details.
+
+We're creating an EKS cluster with a managed node group.
+By default, the group uses 2 VMs of type t2.large. The script places them in AZ eu-west-1a and eu-west-1b
+
+### Persistence
+
+For persistence, the PVs are backed by [EBS volumes](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html).
+Those must reside in the same AZ as the node that will be using them.
+This means that pods must be deployed to specific nodes using node affinity.
+
+#### PosgreSQL data directory
+
+PostgreSQL wants the data folder to be empty (on first startup), but an empty ext4 EBS volume contains a lost+found folder.
+This causes the following error to be reported
+```
+initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
+initdb: detail: It contains a lost+found directory, perhaps due to it being a mount point.
+initdb: hint: Using a mount point directly as the data directory is not recommended.
+Create a subdirectory under the mount point.
+```
+
+For this reason, the volume mount section of the PostgreSQL statefulset has been updated to use subPath
+```yaml
+ volumeMounts:
+ - mountPath: /var/lib/postgresql/data
+ subPath: psql-data
+ name: postgresql-data
+```
+Make sure the `useSubPath` value is set to true.
+Pods within EKS running linux, `requiresPermissionsFix` must also be set to true to enable the init container (running as root) used to reset ownership and file permissions.
+```yaml
+ initContainers:
+ - name: psql-data-ownership
+ image: alpine:latest
+ command: ['sh', '-c', 'chown -R 70:70 /var/lib/postgresql/data && chmod -R 0750 /var/lib/postgresql/data']
+ securityContext:
+ runAsUser: 0
+ volumeMounts:
+ - mountPath: /var/lib/postgresql/data
+ subPath: psql-data
+ name: postgresql-data
+```
+### Networking
+
+#### Using HAProxy
+
+In this configuration, we deploy a HAProxy pod within the cluster.
+All traffic gets routed to HAProxy at layer 4 and HAProxy is responsible for TLS termination and management of all further routing to the other pods.
+This is similar to what is done when running under docker compose on a VM.
+
+All OR pods only expose an ClusterIP service and never an Ingress.
+HAProxy exposes a LoadBalancer service for communication from the outside world.
+We're using [AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.7/) to automatically create a Network Load Balancer (NLB) based on that service.
+One NLB is automatically created when a (LoadBalancer) Service object exists and destroyed when the Service is deleted.
+
+In this configuration, only MQTTS connectivity is available, not MQTT.
+
+#### Using Ingresses and individual LoadBalancer services
+
+TL;DR: Use scripts `eks-setup.sh` and `eks-cleanup.sh` for this configuration.
+
+In this configuration, standard kubernetes Ingress and Load Balancer services are used
+by each pod to expose their APIs to the outside world.
+
+We're using [AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.7/) to automatically create an Application Load Balancer based on the Ingress manifests.
+An Application Load Balancer (ALB) is automatically created when an Ingress object exists in the cluster and destroyed when there are none.
+
+By default, each ingress creates its own ALB, using the [IngressGroup](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/ingress/annotations/#ingressgroup) feature
+and the corresponding `alb.ingress.kubernetes.io/group.name` annotation on the Ingress allows to use a single ALB for multiple ingresses.
+
+The ALB will automatically assign a public "Amazon" DNS name for access.
+We want to use our own DNS names (for ease of use, consistency but also to associate a TLS certificate with it).
+For that, we use [Route 53](https://aws.amazon.com/route53/) and an alias record to directly route traffic to the load balancer, see [Routing traffic to an ELB load balancer - Amazon Route 53](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-elb-load-balancer.html).
+
+To create the certificate, we use [AWS Certificate Manager](https://docs.aws.amazon.com/acm/latest/userguide/acm-overview.html).
+Domain ownership validation is performed via DNS record.
+
+For MQTT(S), the manager uses two Load Balancer services (one for MQTT, one for MQTTS).
+This creates one Network Load Balancer per service.
+We need to use different domain names to route traffic to the appropriate NLB.
+We use subdomains for that, with prefix mqtt and mqtts respectively.
+
+For MQTTS, we use the same certificate as for the ingress.
+During certificate creation, we add a SAN for the mqtts subdomain.
+
+#### Annotations
+
+AWS LB Controller uses specific annotations on the Ingress and Service to configure the underlying infrastructure (ALB and NLB).
+See the values-eks.yaml files annotations fields for more information.
+
+## Configuration
+
+Some aspects of the setup are configurable.
+
+### eks-setup.sh script variables
+
+The script contains several variables that can be changed.
+
+### cluster.yaml
+
+The cluster creation call uses a configuration file.
+This configuration file defines how the cluster is created.
+See [Creating and managing clusters - eksctl](https://eksctl.io/usage/creating-and-managing-clusters/),
+[Config File Schema - eksctl](https://eksctl.io/usage/schema/) and
+[eksctl/examples at main · eksctl-io/eksctl](https://github.com/eksctl-io/eksctl/tree/main/examples)
+for more information.
+
+### values files
+
+In addition to the default values, values-eks.yaml files are used for the chart deployments.
+These contain EKS specific configuration that can be adapted or complemented if required.
+
+## Running demo under EKS
+
+### Building the OR demo image
+
+Use ```./gradlew -PSETUP_JAR=demo clean installDist``` to build the controller with the demo setup code.
+
+You can then push the image to the private ECR repository in our developer account with (credentials are for developer account):
+```bash
+export AWS_ACCESS_KEY_ID="…"
+export AWS_SECRET_ACCESS_KEY="…"
+export AWS_SESSION_TOKEN="…"
+
+aws ecr get-login-password --region eu-west-1| docker login --username AWS --password-stdin 134517981306.dkr.ecr.eu-west-1.amazonaws.com
+
+docker buildx build --no-cache --push --platform linux/amd64,linux/arm64 -t 134517981306.dkr.ecr.eu-west-1.amazonaws.com/openremote/manager:demo manager/build/install/manager/
+```
+
+### Starting a cluster with the OR demo image
+
+Modify the eks-setup-haproxy.sh script to start the manager with the appropriate configuration by replacing lines
+```bash
+helm install manager manager -f manager/values-haproxy-eks.yaml \
+ --set-string or.hostname=$FQDN
+```
+with
+```bash
+helm install manager manager -f manager/values-haproxy-eks.yaml -f manager/values-demo.yaml \
+ --set-string or.hostname=$FQDN
+```
+
+This values file does three things:
+- get the demo image from the private ECR
+- enable the demo setup
+- add an initial delay for the probes, given the setup code time to run (otherwise k8s kills the pod before setup code finishes)
+
+
+## Current limitations / explorations to do
+
+Although the eksctl command reports a successful delete of the cluster, the delete operation is still in progress.
+Most of the time, it fails, not being able to delete the VPC.
+It is best to always check the CloudFormation stacks using the AWS Console and manually delete any leftovers.
+
+We are not using EKS Auto Mode for now. When we want to move forward with EKS, we should investigate this option.
+
+As indicated above, we're using EBS volumes and need to make sure the pod using them is on a node in the same AZ.
+This limits the possibility of distributing pods on the different worker nodes and negates
+some of the automatic orchestration offered by k8s.
+At this stage, it's not critical as anyway the pods using volumes are the DB (which is a stateful set with a single pod)
+and the manager (which is not architected yet to support multiple replicas).
+We could explore using storage class and not explicit binding and/or using other volume types.
+We should anyway review the manager usage of persistent storage and limit the use of the file system for this purpose.
+
+There is some duplication in the configuration, with the same values appearing in the shell scripts and the cluster.yaml file.
+When moving to production usage, this should be reviewed.
+But also the shell script might be replaced with CloudFormation templates or using Terraform or similar tools.
diff --git a/kubernetes/README.md b/kubernetes/README.md
index 94d027b8e..c691392cb 100644
--- a/kubernetes/README.md
+++ b/kubernetes/README.md
@@ -3,41 +3,49 @@
This folder contains Helm charts for each individual OR component to deploy within kubernetes.
It supposes you have already installed the required tools on your machine:
- a Kubernetes installation, tests were performed with Docker Desktop on macOS.
-- an Ingress controller installed in the cluster, tests were performed with [Nginx Controller](https://kubernetes.github.io/ingress-nginx/deploy/#quick-start)
- kubectl
- helm
+There are two options on how the OR components can be accessed:
+- using a HAProxy pod to manage all connections
+- using the standard kubernetes network objects: Ingress and Service
+
+In the later case, an Ingress controller must be installed in the cluster ([Nginx Controller](https://kubernetes.github.io/ingress-nginx/deploy/#quick-start) was used during testing).
+In the former case, an Ingress controller should NOT be installed as this causes conflicts.
+
+This README file covers deployment on a local machine, for information on deploying into an EKS cluster on AWS, see README-AWS.md
+
## TL;DR
-### Create the Persistent Volumes
+The following steps use the first option, managing connections through HAProxy.
-Edit the *-pv.yaml files and update the spec.hostPath.path entry to point to a folder on your local machine.
+### Create the Persistent Volumes and the required secrets
+
+This is performed via the `or-setup` helm chart.
+Review the `values.yaml` file under the `or-setup` folder and create one with values appropriate for your environment.
+
+You should certainly edit the `basePath` value to point to a folder on your local machine.
Under macOS, it needs to be located under your home folder (/Users/xxx/...).
-Create the PVs by applying the files
-```bash
-kubectl apply -f openremote-manager-pv.yaml
-kubectl apply -f openremote-psql-pv.yaml
-```
-### Create the required secrets
-The openremote-secrets.yaml creates a secret to hold sensitive configuration from OpenRemote
+The `openremote-secrets.yaml` file under `or-setup/templates` creates a secret to hold sensitive configuration from OpenRemote
(database username and password, Keycloak admin password).
The values in this file are the default ones, update as required.
```bash
-kubectl apply -f openremote-secret.yaml
+helm install or-setup or-setup -f your_values.yaml
```
### Install the charts
Install the different charts in order
```bash
+helm install proxy proxy
helm install postgresql postgresql
helm install keycloak keycloak
helm install manager manager
```
-If running under linux, you must enable to requiresPermissionsFix flag when instaling postgresql
+If running under linux, you must enable to requiresPermissionsFix flag when installing postgresql
```bash
helm install postgresql postgresql --set requiresPermissionsFix=true
```
@@ -47,7 +55,7 @@ helm install postgresql postgresql --set requiresPermissionsFix=true
### Persistent Volumes lifecycle
Uninstalling the charts will delete the Persistent Volume Claims but not the PVs.
-Those will go back the 'Released' state, at the point they can't be bound again.
+Those will go to the 'Released' state, at which point they can't be bound again.
You need to manually transition them to 'Available' before you re-install the charts
```bash
kubectl patch pv manager-data-pv -p '{"spec":{"claimRef": null}}'
@@ -56,7 +64,7 @@ kubectl patch pv postgresql-data-pv -p '{"spec":{"claimRef": null}}'
### Release names
-Several objects name are constructed based on the release name.
+Several object names are constructed based on the release name.
The provided values files are based on the release name as shown above.
If you use a different release name, this will impact the name of certain objects and require you to adapt the values accordingly.
For instance, using psql instead of postgresql as the release name will modify the service name from postgresql to psql-postgresql.
@@ -64,17 +72,17 @@ This service name is used in both the keycloak and manager values files to defin
### Resources (requests and limits)
-The values file do not define any values for memory and CPU requests or limits.
+The values files do not define any values for memory and CPU requests or limits.
Although this works for local development work, it is strongly recommended to fix values for both when deploying to production.
Look at the commented `resources` section in the values files and provide actual values matching your deployment scenario in your custom values files.
### Duplicated configuration
-Some configuration information (e.g. the database name) are duplicated between the values files of the different charts.
+Some configuration information (e.g. the database name) is duplicated between the values files of the different charts.
This is a conscious decision at this time to keep the charts independent.
We'll be looking into improving on this in the future ([Have a mechanism to deploy the complete OR stack in a single operation · Issue #1651 · openremote/openremote](https://github.com/openremote/openremote/issues/1651))
-That being said, we have gone of a single Opaque secret to contain all secure information applicable to all charts instead of multiple kubernetes.io/basic-auth secrets for individual credentials.
+That being said, we are using a single Opaque secret to contain all secure information applicable to all charts instead of multiple kubernetes.io/basic-auth secrets for individual credentials.
## Additional information
@@ -84,8 +92,8 @@ The most important configuration information has been exposed in the helm values
It is sometimes necessary to provide more configuration to the containers.
This is done through environment variables, as it was done before with docker compose.
In the values files, there is `or.env` property available to define any environment variable that will get passed to the container.
-For instance, this snippet of a values file defines a value for the KC_HOSTNAME_STRICT_HTTPS environment variable passed to the keycloak container.
-Important: the value always needs to be encloded in double quotes, even for boolean or numeric values.
+For instance, this snippet of a values files defines a value for the KC_HOSTNAME_STRICT_HTTPS environment variable passed to the keycloak container.
+Important: the value always needs to be enclosed in double quotes, even for boolean or numeric values.
```yaml
or:
env:
@@ -93,19 +101,52 @@ or:
value: "false"
```
-### Accessing MQTT
+### When using HAProxy
+
+#### Accessing MQTT
+
+MQTTS is accessible on port 8883 through HAProxy.
+Non-TLS access to MQTT requires using a manual port forward to the pod e.g.
+`kubectl port-forward manager-… 1883:1883`
+
+#### Using with IDE for development
+
+It is not possible to run the manager inside the IDE and have HAProxy deployed as a pod in the cluster.
+Use an ingress instead for that scenario.
+
+#### Certificate management
+
+HAProxy is responsible for certificate management (either using the default self-signed or using ACME).
+Please refer to the proxy documentation for more information.
+
+### When using Ingress (no HAProxy pod)
+
+For this scenario, make sure an Ingress controller is installed in the cluster.
+Review the values files for your deployment scenario, in particular make sure to configure and enable the ingress on keycloak and manager pods.
+Install the different charts in order
+```bash
+helm install postgresql postgresql
+helm install keycloak keycloak
+helm install manager manager
+```
+
+#### Accessing MQTT
+
+If, in the manager values files, you enable the MQTT/MQTTS services, you can directly access them (by default on port 1883 and 8883).
+**In this configuration, the MQTTS service does not have a certificate and although named MQTTS will only accept MQTT connections !!!**
-TODO: port forward or service for MQTT port(s)
+Alternatively, if you do not enable a service, you can use manual port forwarding to the pod e.g.
+`kubectl port-forward manager-… 1883:1883`
-### Using with IDE for development
+#### Using with IDE for development
-Install the postgresql and keycloak charts. For keycloak, use the specific values files to configure support plain HTTP calls and set the HTTP port.
+Install the postgresql and keycloak charts. For keycloak, use the specific values files to configure support for plain HTTP calls and set the HTTP port.
```bash
helm install postgresql postgresql
helm install keycloak keycloak -f keycloak/values-dev.yaml
```
-Forward ports for direct access to database and keycloak.
+Forward ports for direct access to postgresql and keycloak.
```bash
kubectl port-forward svc/postgresql 5432:5432
kubectl port-forward svc/keycloak 8081:8080
diff --git a/kubernetes/cluster.yaml b/kubernetes/cluster.yaml
new file mode 100644
index 000000000..4f8a4b3ec
--- /dev/null
+++ b/kubernetes/cluster.yaml
@@ -0,0 +1,18 @@
+apiVersion: eksctl.io/v1alpha5
+kind: ClusterConfig
+
+metadata:
+ name: testcluster
+ region: eu-west-1
+ # Kubernetes version to use for cluster, good idea to keep up to date
+ version: "1.33"
+
+availabilityZones:
+ - eu-west-1a
+ - eu-west-1b
+
+nodeGroups:
+ - name: ng-1
+ instanceType: t2.large
+ volumeSize: 32
+ desiredCapacity: 2
diff --git a/kubernetes/eks-cleanup-haproxy.sh b/kubernetes/eks-cleanup-haproxy.sh
new file mode 100755
index 000000000..3f8e15714
--- /dev/null
+++ b/kubernetes/eks-cleanup-haproxy.sh
@@ -0,0 +1,47 @@
+#!/bin/bash
+
+. ./eks-common.sh
+
+DNS_NAME=$(aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[0].DNSName")
+HOSTED_ZONE_ID=$(aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[0].CanonicalHostedZoneId")
+
+echo "Delete DNS record $FQDN"
+
+aws route53 change-resource-record-sets \
+ --hosted-zone-id /hostedzone/Z08751721JH0NB6LLCB4V \
+ --change-batch \
+ '{"Changes": [ { "Action": "DELETE", "ResourceRecordSet": { "Name": "'$FQDN'", "Type": "A", "AliasTarget":{ "HostedZoneId": '$HOSTED_ZONE_ID',"DNSName": '$DNS_NAME',"EvaluateTargetHealth": false} } } ]}' \
+ --profile dnschg
+
+helm uninstall manager
+helm uninstall keycloak
+helm uninstall postgresql
+helm uninstall proxy
+
+# Give the AWS LB Controller time to delete the NLB after the Service is deleted
+while aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?Type=='network']" 2>/dev/null | grep '"Code": "active"'; do
+ echo "Waiting for load balancer to be deleted..."
+ sleep 10
+done
+helm uninstall aws-load-balancer-controller -n kube-system
+
+MANAGER_VOLUMEID=$(kubectl get pv manager-data-pv -o=jsonpath='{.spec.awsElasticBlockStore.volumeID}')
+PSQL_VOLUMEID=$(kubectl get pv postgresql-data-pv -o=jsonpath='{.spec.awsElasticBlockStore.volumeID}')
+kubectl delete pv manager-data-pv
+kubectl delete pv postgresql-data-pv
+aws ec2 delete-volume --volume-id $MANAGER_VOLUMEID
+aws ec2 delete-volume --volume-id $PSQL_VOLUMEID
+helm uninstall or-setup
+
+# Manually deleting all addons, this should not be required but otherwise the delete cluster fails
+# with "2 pods are unevictable from node ..." error messages
+eksctl delete addon --cluster $CLUSTER_NAME --name aws-ebs-csi-driver
+eksctl delete addon --cluster $CLUSTER_NAME --name vpc-cni
+eksctl delete addon --cluster $CLUSTER_NAME --name metrics-server
+eksctl delete addon --cluster $CLUSTER_NAME --name coredns
+eksctl delete addon --cluster $CLUSTER_NAME --name kube-proxy
+
+eksctl delete cluster --wait --name $CLUSTER_NAME || {
+ echo "Clean-up has failed, please check the status in AWS CloudFormation console at https://eu-west-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks?filteringText=&filteringStatus=active&viewNested=true"
+ echo "Failed clean-ups are often caused by a dangling VPC. Manually delete the VPC and then retry the cluster stack delete operation."
+}
diff --git a/kubernetes/eks-cleanup.sh b/kubernetes/eks-cleanup.sh
new file mode 100755
index 000000000..b4eae1951
--- /dev/null
+++ b/kubernetes/eks-cleanup.sh
@@ -0,0 +1,103 @@
+#!/bin/bash
+
+. ./eks-common.sh
+
+CERTIFICATE_ARN=$(kubectl get ingress manager -o jsonpath='{.metadata.annotations.\alb\.ingress\.kubernetes\.io\/certificate-arn}')
+
+DNS_RECORD_NAME=$(aws acm describe-certificate --certificate-arn $CERTIFICATE_ARN --profile or --query "Certificate.DomainValidationOptions[0].ResourceRecord.Name")
+DNS_RECORD_VALUE=$(aws acm describe-certificate --certificate-arn $CERTIFICATE_ARN --profile or --query "Certificate.DomainValidationOptions[0].ResourceRecord.Value")
+
+echo "Delete DNS record $DNS_RECORD_NAME"
+aws route53 change-resource-record-sets \
+ --hosted-zone-id /hostedzone/Z08751721JH0NB6LLCB4V \
+ --change-batch \
+ '{"Changes": [ { "Action": "DELETE", "ResourceRecordSet": { "Name": '$DNS_RECORD_NAME', "Type": "CNAME", "TTL": 300, "ResourceRecords" : [ { "Value": '$DNS_RECORD_VALUE' } ] } } ]}' \
+ --profile dnschg
+
+DNS_RECORD_NAME=$(aws acm describe-certificate --certificate-arn $CERTIFICATE_ARN --profile or --query "Certificate.DomainValidationOptions[1].ResourceRecord.Name")
+DNS_RECORD_VALUE=$(aws acm describe-certificate --certificate-arn $CERTIFICATE_ARN --profile or --query "Certificate.DomainValidationOptions[1].ResourceRecord.Value")
+
+echo "Delete DNS record $DNS_RECORD_NAME"
+aws route53 change-resource-record-sets \
+ --hosted-zone-id /hostedzone/Z08751721JH0NB6LLCB4V \
+ --change-batch \
+ '{"Changes": [ { "Action": "DELETE", "ResourceRecordSet": { "Name": '$DNS_RECORD_NAME', "Type": "CNAME", "TTL": 300, "ResourceRecords" : [ { "Value": '$DNS_RECORD_VALUE' } ] } } ]}' \
+ --profile dnschg
+
+DNS_NAME=$(aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?Type=='application'].DNSName | [0]")
+HOSTED_ZONE_ID=$(aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?Type=='application'].CanonicalHostedZoneId | [0]")
+
+echo "Delete DNS record $FQDN"
+
+aws route53 change-resource-record-sets \
+ --hosted-zone-id /hostedzone/Z08751721JH0NB6LLCB4V \
+ --change-batch \
+ '{"Changes": [ { "Action": "DELETE", "ResourceRecordSet": { "Name": "'$FQDN'", "Type": "A", "AliasTarget":{ "HostedZoneId": '$HOSTED_ZONE_ID',"DNSName": '$DNS_NAME',"EvaluateTargetHealth": false} } } ]}' \
+ --profile dnschg
+
+DNS_NAME=$(kubectl get svc manager-mqtt -o jsonpath="{.status.loadBalancer.ingress[0].hostname}")
+HOSTED_ZONE_ID=$(aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?DNSName=='$DNS_NAME'].CanonicalHostedZoneId | [0]")
+
+echo "Delete DNS record $MQTT_FQDN"
+
+aws route53 change-resource-record-sets \
+ --hosted-zone-id /hostedzone/Z08751721JH0NB6LLCB4V \
+ --change-batch \
+ '{"Changes": [ { "Action": "DELETE", "ResourceRecordSet": { "Name": "'$MQTT_FQDN'", "Type": "A", "AliasTarget":{ "HostedZoneId": '$HOSTED_ZONE_ID',"DNSName": '\"$DNS_NAME\"',"EvaluateTargetHealth": false} } } ]}' \
+ --profile dnschg
+
+DNS_NAME=$(kubectl get svc manager-mqtts -o jsonpath="{.status.loadBalancer.ingress[0].hostname}")
+HOSTED_ZONE_ID=$(aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?DNSName=='$DNS_NAME'].CanonicalHostedZoneId | [0]")
+
+echo "Delete DNS record $MQTTS_FQDN"
+
+aws route53 change-resource-record-sets \
+ --hosted-zone-id /hostedzone/Z08751721JH0NB6LLCB4V \
+ --change-batch \
+ '{"Changes": [ { "Action": "DELETE", "ResourceRecordSet": { "Name": "'$MQTTS_FQDN'", "Type": "A", "AliasTarget":{ "HostedZoneId": '$HOSTED_ZONE_ID',"DNSName": '\"$DNS_NAME\"',"EvaluateTargetHealth": false} } } ]}' \
+ --profile dnschg
+
+helm uninstall manager
+helm uninstall keycloak
+helm uninstall postgresql
+
+# Give the AWS LB Controller time to delete the ALB and NLB after the Service is deleted
+while aws elbv2 describe-load-balancers --profile or --query "LoadBalancers" --output text 2>/dev/null | grep -q .; do
+ echo "Waiting for load balancer to be deleted..."
+ sleep 10
+done
+helm uninstall aws-load-balancer-controller -n kube-system
+
+echo "Delete certificate $CERTIFICATE_ARN"
+MAX_RETRIES=30
+for ((i=1; i<=MAX_RETRIES; i++)); do
+ if aws acm delete-certificate --certificate-arn "$CERTIFICATE_ARN" --profile or; then
+ break
+ else
+ sleep 10
+ fi
+done
+if (( i > MAX_RETRIES )); then
+ echo "⚠️ Timed out trying to delete certificate $CERTIFICATE_ARN after $MAX_RETRIES attempts."
+fi
+
+MANAGER_VOLUMEID=$(kubectl get pv manager-data-pv -o=jsonpath='{.spec.awsElasticBlockStore.volumeID}')
+PSQL_VOLUMEID=$(kubectl get pv postgresql-data-pv -o=jsonpath='{.spec.awsElasticBlockStore.volumeID}')
+kubectl delete pv manager-data-pv
+kubectl delete pv postgresql-data-pv
+aws ec2 delete-volume --volume-id $MANAGER_VOLUMEID
+aws ec2 delete-volume --volume-id $PSQL_VOLUMEID
+helm uninstall or-setup
+
+# Manually deleting all addons, this should not be required but otherwise the delete cluster fails
+# with "2 pods are unevictable from node ..." error messages
+eksctl delete addon --cluster $CLUSTER_NAME --name aws-ebs-csi-driver
+eksctl delete addon --cluster $CLUSTER_NAME --name vpc-cni
+eksctl delete addon --cluster $CLUSTER_NAME --name metrics-server
+eksctl delete addon --cluster $CLUSTER_NAME --name coredns
+eksctl delete addon --cluster $CLUSTER_NAME --name kube-proxy
+
+eksctl delete cluster --wait --name $CLUSTER_NAME || {
+ echo "Clean-up has failed, please check the status in AWS CloudFormation console at https://eu-west-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks?filteringText=&filteringStatus=active&viewNested=true"
+ echo "Failed clean-ups are often caused by a dangling VPC. Manually delete the VPC and then retry the cluster stack delete operation."
+}
diff --git a/kubernetes/eks-common.sh b/kubernetes/eks-common.sh
new file mode 100755
index 000000000..73a2c7829
--- /dev/null
+++ b/kubernetes/eks-common.sh
@@ -0,0 +1,39 @@
+#!/bin/bash
+
+set -eo pipefail
+
+# Name of cluster, not exposed but must be unique within account
+# !!! make sure to also update in cluster.yaml
+CLUSTER_NAME=testcluster
+
+# Hostname to use for public access to this instance, always under the openremote.app domain
+HOSTNAME=testmanager
+FQDN=$HOSTNAME.openremote.app
+MQTT_FQDN=mqtt.$FQDN
+MQTTS_FQDN=mqtts.$FQDN
+
+# !!! make sure region in cluster.yaml matches
+AWS_REGION="eu-west-1"
+AWS_ACCOUNT_ID="463235666115" # openremote
+
+DNSCHG_ROLE_ARN="arn:aws:iam::134517981306:role/route53-full-access"
+
+aws configure set region $AWS_REGION
+aws configure --profile or set region $AWS_REGION
+aws configure --profile or set output json
+aws configure --profile dnschg set source_profile or
+aws configure --profile dnschg set role_arn $DNSCHG_ROLE_ARN
+
+if [ -n "$AWS_ACCESS_KEY_ID" ]; then
+ aws configure --profile or set aws_access_key_id $AWS_ACCESS_KEY_ID
+fi
+
+if [ -n "$AWS_SECRET_ACCESS_KEY" ]; then
+ aws configure --profile or set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
+fi
+
+if [ -n "$AWS_SESSION_TOKEN" ]; then
+ aws configure --profile or set aws_session_token $AWS_SESSION_TOKEN
+fi
+
+aws sts assume-role --role-arn $DNSCHG_ROLE_ARN --role-session-name dnschg --profile or --query "Credentials.[AccessKeyId, SecretAccessKey, SessionToken]" --output text | awk -F'\t' '{print "aws_access_key_id "$1"\naws_secret_access_key "$2"\naws_session_token "$3 }' | xargs -L 1 aws configure --profile dnschg set
diff --git a/kubernetes/eks-setup-haproxy.sh b/kubernetes/eks-setup-haproxy.sh
new file mode 100755
index 000000000..97d086ad1
--- /dev/null
+++ b/kubernetes/eks-setup-haproxy.sh
@@ -0,0 +1,112 @@
+#!/bin/bash
+
+. ./eks-common.sh
+
+# TODO: --name $CLUSTER_NAME -> in cluster.yaml
+# TODO: region is duplicated in cluster.yaml
+eksctl create cluster -f cluster.yaml --profile or
+
+# TODO: maybe extract cluster name after cluster creation so source of truth is in cluster.yaml, not other way around
+# Not straightforward
+
+# TODO: it might be possible to replace some of the operations below by configuration options in cluster.yaml
+oidc_id=$(aws eks describe-cluster --profile or --name $CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)
+
+eksctl utils associate-iam-oidc-provider --profile or --cluster $CLUSTER_NAME --approve
+
+eksctl create iamserviceaccount --profile or \
+ --name ebs-csi-controller-sa \
+ --namespace kube-system \
+ --cluster $CLUSTER_NAME \
+ --role-name AmazonEKS_EBS_CSI_DriverRole \
+ --role-only \
+ --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
+ --approve
+
+eksctl create addon --profile or --cluster $CLUSTER_NAME --name aws-ebs-csi-driver --version latest \
+ --service-account-role-arn arn:aws:iam::$AWS_ACCOUNT_ID:role/AmazonEKS_EBS_CSI_DriverRole --force
+
+# [Installation Guide - AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/deploy/installation/)
+
+curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.11.0/docs/install/iam_policy.json
+
+POLICY_NAME="AWSLoadBalancerControllerIAMPolicy"
+POLICY_ARN="arn:aws:iam::$AWS_ACCOUNT_ID:policy/$POLICY_NAME"
+
+if ! aws iam get-policy --policy-arn "$POLICY_ARN" >/dev/null 2>&1; then
+ echo "Policy does not exist. Creating..."
+ aws iam create-policy \
+ --policy-name "$POLICY_NAME" \
+ --policy-document file://iam_policy.json
+fi
+
+rm -f iam_policy.json
+
+eksctl create iamserviceaccount --profile or \
+ --cluster=$CLUSTER_NAME \
+ --namespace=kube-system \
+ --name=aws-load-balancer-controller \
+ --role-name AmazonEKSLoadBalancerControllerRole \
+ --attach-policy-arn=arn:aws:iam::$AWS_ACCOUNT_ID:policy/AWSLoadBalancerControllerIAMPolicy \
+ --approve
+
+# [Install AWS Load Balancer Controller with Helm - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/lbc-helm.html)
+
+helm repo add eks https://aws.github.io/eks-charts
+
+helm repo update eks
+
+helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
+ -n kube-system \
+ --set clusterName=$CLUSTER_NAME \
+ --set serviceAccount.create=false \
+ --set serviceAccount.name=aws-load-balancer-controller
+
+PSQL_VOLUMEID=$(aws ec2 create-volume --size 1 --availability-zone eu-west-1a --tag-specifications "ResourceType=volume,Tags=[{Key=Name,Value=psql-data}]" --query VolumeId)
+MANAGER_VOLUMEID=$(aws ec2 create-volume --size 1 --availability-zone eu-west-1a --tag-specifications "ResourceType=volume,Tags=[{Key=Name,Value=manager-data}]" --query VolumeId)
+
+# Wait for AWS LB ctrl to be ready
+kubectl rollout status deployment aws-load-balancer-controller -n kube-system --timeout=300s
+
+CLUSTER_DNS=$(kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}:{.spec.ports[?(@.name=="dns")].port}')
+
+helm install or-setup or-setup --set aws.enabled=true --set aws.managerVolumeId=$MANAGER_VOLUMEID --set aws.psqlVolumeId=$PSQL_VOLUMEID
+
+helm install proxy proxy -f proxy/values-eks.yaml --set or.nameserver=$CLUSTER_DNS --set or.hostname=$FQDN
+
+helm install postgresql postgresql -f postgresql/values-eks.yaml
+
+# Waiting for the LB to be created
+# AWS LB Controller only creates an Network LB if there's a service
+# Following logic is assuming there is the only (network) LB in the account
+while ! aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?Type=='network']" 2>/dev/null | grep '"Code": "active"'; do
+ echo "Waiting for load balancer to be created..."
+ sleep 10
+done
+
+DNS_NAME=$(aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?Type=='network'].DNSName | [0]")
+HOSTED_ZONE_ID=$(aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?Type=='network'].CanonicalHostedZoneId | [0]")
+
+aws route53 change-resource-record-sets \
+ --hosted-zone-id /hostedzone/Z08751721JH0NB6LLCB4V \
+ --change-batch \
+ '{"Changes": [ { "Action": "UPSERT", "ResourceRecordSet": { "Name": "'$FQDN'", "Type": "A", "AliasTarget":{ "HostedZoneId": '$HOSTED_ZONE_ID',"DNSName": '$DNS_NAME',"EvaluateTargetHealth": false} } } ]}' \
+ --profile dnschg
+
+helm install keycloak keycloak -f keycloak/values-haproxy.yaml \
+ --set-string or.hostname=$FQDN
+helm install manager manager -f manager/values-haproxy-eks.yaml \
+ --set-string or.hostname=$FQDN
+
+while ! dig +short $FQDN | grep -qE '^[0-9]'; do
+ echo "Waiting for DNS resolution..."
+ sleep 5
+done
+
+# Now that DNS is in place, HAProxy can properly create the certificate
+kubectl exec $(kubectl get pod -l "app.kubernetes.io/name=proxy" -o name) -- sh -c "/entrypoint.sh add $FQDN" || {
+ echo "Certificate generation failed, please wait a moment and manually execute the following command:"
+ echo 'kubectl exec $(kubectl get pod -l "app.kubernetes.io/name=proxy" -o name) -- sh -c "/entrypoint.sh add <fully qualified hostname>"'
+}
+
+echo "Access the manager at https://$FQDN"
diff --git a/kubernetes/eks-setup.sh b/kubernetes/eks-setup.sh
new file mode 100755
index 000000000..c1dd8ada0
--- /dev/null
+++ b/kubernetes/eks-setup.sh
@@ -0,0 +1,158 @@
+#!/bin/bash
+
+. ./eks-common.sh
+
+# TODO: --name $CLUSTER_NAME -> in cluster.yaml
+# TODO: region is duplicated in cluster.yaml
+eksctl create cluster -f cluster.yaml --profile or
+
+# TODO: maybe extract cluster name after cluster creation so source of truth is in cluster.yaml, not other way around
+# Not straightforward
+
+# TODO: it might be possible to replace some of the operations below by configuration options in cluster.yaml
+oidc_id=$(aws eks describe-cluster --profile or --name $CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)
+
+eksctl utils associate-iam-oidc-provider --profile or --cluster $CLUSTER_NAME --approve
+
+eksctl create iamserviceaccount --profile or \
+ --name ebs-csi-controller-sa \
+ --namespace kube-system \
+ --cluster $CLUSTER_NAME \
+ --role-name AmazonEKS_EBS_CSI_DriverRole \
+ --role-only \
+ --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
+ --approve
+
+eksctl create addon --profile or --cluster $CLUSTER_NAME --name aws-ebs-csi-driver --version latest \
+ --service-account-role-arn arn:aws:iam::$AWS_ACCOUNT_ID:role/AmazonEKS_EBS_CSI_DriverRole --force
+
+# [Installation Guide - AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/deploy/installation/)
+
+curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.11.0/docs/install/iam_policy.json
+
+POLICY_NAME="AWSLoadBalancerControllerIAMPolicy"
+POLICY_ARN="arn:aws:iam::$AWS_ACCOUNT_ID:policy/$POLICY_NAME"
+
+if ! aws iam get-policy --policy-arn "$POLICY_ARN" >/dev/null 2>&1; then
+ echo "Policy does not exist. Creating..."
+ aws iam create-policy \
+ --policy-name "$POLICY_NAME" \
+ --policy-document file://iam_policy.json
+fi
+
+rm -f iam_policy.json
+
+eksctl create iamserviceaccount --profile or \
+ --cluster=$CLUSTER_NAME \
+ --namespace=kube-system \
+ --name=aws-load-balancer-controller \
+ --role-name AmazonEKSLoadBalancerControllerRole \
+ --attach-policy-arn=arn:aws:iam::$AWS_ACCOUNT_ID:policy/AWSLoadBalancerControllerIAMPolicy \
+ --approve
+
+# [Install AWS Load Balancer Controller with Helm - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/lbc-helm.html)
+
+helm repo add eks https://aws.github.io/eks-charts
+
+helm repo update eks
+
+helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
+ -n kube-system \
+ --set clusterName=$CLUSTER_NAME \
+ --set serviceAccount.create=false \
+ --set serviceAccount.name=aws-load-balancer-controller
+
+PSQL_VOLUMEID=$(aws ec2 create-volume --size 1 --availability-zone eu-west-1a --tag-specifications "ResourceType=volume,Tags=[{Key=Name,Value=psql-data}]" --query VolumeId)
+MANAGER_VOLUMEID=$(aws ec2 create-volume --size 1 --availability-zone eu-west-1a --tag-specifications "ResourceType=volume,Tags=[{Key=Name,Value=manager-data}]" --query VolumeId)
+
+# Wait for AWS LB ctrl to be ready
+kubectl rollout status deployment aws-load-balancer-controller -n kube-system --timeout=300s
+
+helm install or-setup or-setup --set aws.enabled=true --set aws.managerVolumeId=$MANAGER_VOLUMEID --set aws.psqlVolumeId=$PSQL_VOLUMEID
+
+CERTIFICATE_ARN=$(aws acm request-certificate --domain-name $FQDN --subject-alternative-names $MQTTS_FQDN --validation-method DNS --profile or --query "CertificateArn" --output text)
+aws acm describe-certificate --certificate-arn $CERTIFICATE_ARN --profile or
+
+helm install postgresql postgresql -f postgresql/values-eks.yaml
+helm install keycloak keycloak -f keycloak/values-eks.yaml \
+ --set-string or.hostname=$FQDN \
+ --set-string 'ingress.annotations.alb\.ingress\.kubernetes\.io\/certificate-arn'=$CERTIFICATE_ARN
+helm install manager manager -f manager/values-eks.yaml \
+ --set-string or.hostname=$FQDN \
+ --set-string 'ingress.annotations.alb\.ingress\.kubernetes\.io\/certificate-arn'=$CERTIFICATE_ARN \
+ --set-string 'service.mqtts.annotations.service\.beta\.kubernetes\.io\/aws-load-balancer-ssl-cert'=$CERTIFICATE_ARN
+
+# For FQDN
+
+DNS_RECORD_NAME=$(aws acm describe-certificate --certificate-arn $CERTIFICATE_ARN --profile or --query "Certificate.DomainValidationOptions[0].ResourceRecord.Name")
+DNS_RECORD_VALUE=$(aws acm describe-certificate --certificate-arn $CERTIFICATE_ARN --profile or --query "Certificate.DomainValidationOptions[0].ResourceRecord.Value")
+
+aws route53 change-resource-record-sets \
+ --hosted-zone-id /hostedzone/Z08751721JH0NB6LLCB4V \
+ --change-batch \
+ '{"Changes": [ { "Action": "UPSERT", "ResourceRecordSet": { "Name": '$DNS_RECORD_NAME', "Type": "CNAME", "TTL": 300, "ResourceRecords" : [ { "Value": '$DNS_RECORD_VALUE' } ] } } ]}' \
+ --profile dnschg
+
+# For MQTTS_FQDN
+
+DNS_RECORD_NAME=$(aws acm describe-certificate --certificate-arn $CERTIFICATE_ARN --profile or --query "Certificate.DomainValidationOptions[1].ResourceRecord.Name")
+DNS_RECORD_VALUE=$(aws acm describe-certificate --certificate-arn $CERTIFICATE_ARN --profile or --query "Certificate.DomainValidationOptions[1].ResourceRecord.Value")
+
+aws route53 change-resource-record-sets \
+ --hosted-zone-id /hostedzone/Z08751721JH0NB6LLCB4V \
+ --change-batch \
+ '{"Changes": [ { "Action": "UPSERT", "ResourceRecordSet": { "Name": '$DNS_RECORD_NAME', "Type": "CNAME", "TTL": 300, "ResourceRecords" : [ { "Value": '$DNS_RECORD_VALUE' } ] } } ]}' \
+ --profile dnschg
+
+# AWS LB Controller only creates an Application LB if there's an ingress
+# Following logic is assuming there is the only (application) LB in the account
+while ! aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?Type=='application']" 2>/dev/null | grep '"Code": "active"'; do
+ echo "Waiting for application load balancer to be created..."
+ sleep 10
+done
+
+# We're re-directing the FQDN to the application LB (web interface)
+until DNS_NAME=$(aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?Type=='application'].DNSName | [0]") && [ -n "$DNS_NAME" ]; do
+ echo "Waiting for LoadBalancer DNS..."; sleep 5
+done
+HOSTED_ZONE_ID=$(aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?Type=='application'].CanonicalHostedZoneId | [0]")
+
+aws route53 change-resource-record-sets \
+ --hosted-zone-id /hostedzone/Z08751721JH0NB6LLCB4V \
+ --change-batch \
+ '{"Changes": [ { "Action": "UPSERT", "ResourceRecordSet": { "Name": "'$FQDN'", "Type": "A", "AliasTarget":{ "HostedZoneId": '$HOSTED_ZONE_ID',"DNSName": '$DNS_NAME',"EvaluateTargetHealth": false} } } ]}' \
+ --profile dnschg
+
+# We're re-directing the MQTT_FQDN to its network load balancer
+until DNS_NAME=$(kubectl get svc manager-mqtt -o jsonpath="{.status.loadBalancer.ingress[0].hostname}") && [ -n "$DNS_NAME" ]; do
+ echo "Waiting for LoadBalancer DNS..."; sleep 5
+done
+while ! aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?DNSName=='$DNS_NAME']" 2>/dev/null | grep '"Code": "active"'; do
+ echo "Waiting for network load balancer to be created for $DNS_NAME..."
+ sleep 10
+done
+HOSTED_ZONE_ID=$(aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?DNSName=='$DNS_NAME'].CanonicalHostedZoneId | [0]")
+
+aws route53 change-resource-record-sets \
+ --hosted-zone-id /hostedzone/Z08751721JH0NB6LLCB4V \
+ --change-batch \
+ '{"Changes": [ { "Action": "UPSERT", "ResourceRecordSet": { "Name": "'$MQTT_FQDN'", "Type": "A", "AliasTarget":{ "HostedZoneId": '$HOSTED_ZONE_ID',"DNSName": '\"$DNS_NAME\"',"EvaluateTargetHealth": false} } } ]}' \
+ --profile dnschg
+
+# We're re-directing the MQTTS_FQDN to its network load balancer
+until DNS_NAME=$(kubectl get svc manager-mqtts -o jsonpath="{.status.loadBalancer.ingress[0].hostname}") && [ -n "$DNS_NAME" ]; do
+ echo "Waiting for LoadBalancer DNS..."; sleep 5
+done
+while ! aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?DNSName=='$DNS_NAME']" 2>/dev/null | grep '"Code": "active"'; do
+ echo "Waiting for network load balancer to be created for $DNS_NAME..."
+ sleep 10
+done
+HOSTED_ZONE_ID=$(aws elbv2 describe-load-balancers --profile or --query "LoadBalancers[?DNSName=='$DNS_NAME'].CanonicalHostedZoneId | [0]")
+
+aws route53 change-resource-record-sets \
+ --hosted-zone-id /hostedzone/Z08751721JH0NB6LLCB4V \
+ --change-batch \
+ '{"Changes": [ { "Action": "UPSERT", "ResourceRecordSet": { "Name": "'$MQTTS_FQDN'", "Type": "A", "AliasTarget":{ "HostedZoneId": '$HOSTED_ZONE_ID',"DNSName": '\"$DNS_NAME\"',"EvaluateTargetHealth": false} } } ]}' \
+ --profile dnschg
+
+echo "Access the manager at https://$FQDN"
diff --git a/kubernetes/keycloak/Chart.yaml b/kubernetes/keycloak/Chart.yaml
index e5e5550f8..72e195f8d 100644
--- a/kubernetes/keycloak/Chart.yaml
+++ b/kubernetes/keycloak/Chart.yaml
@@ -6,5 +6,4 @@ type: application
version: 0.1.0
-appVersion: "23.0.7.2"
-# Using latest for now but would be better to have specific SemVer version
+appVersion: "23.0.7.3"
diff --git a/kubernetes/keycloak/values-eks.yaml b/kubernetes/keycloak/values-eks.yaml
new file mode 100644
index 000000000..2b8d3e833
--- /dev/null
+++ b/kubernetes/keycloak/values-eks.yaml
@@ -0,0 +1,16 @@
+ingress:
+ enabled: true
+ hosts:
+ - paths:
+ - path: /auth
+ pathType: Prefix
+ className: "alb"
+ annotations:
+ alb.ingress.kubernetes.io/scheme: internet-facing
+ alb.ingress.kubernetes.io/target-type: ip
+ alb.ingress.kubernetes.io/group.name: eric-test-ig
+ alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-west-1:463235666115:certificate/...
+ tls: {}
+
+or:
+ hostname: k8stest.openremote.app
diff --git a/kubernetes/keycloak/values-haproxy.yaml b/kubernetes/keycloak/values-haproxy.yaml
new file mode 100644
index 000000000..d1c7c937d
--- /dev/null
+++ b/kubernetes/keycloak/values-haproxy.yaml
@@ -0,0 +1,5 @@
+ingress:
+ enabled: false
+
+or:
+ hostname: "localhost"
diff --git a/kubernetes/keycloak/values.yaml b/kubernetes/keycloak/values.yaml
index b270443ba..ab56e8c64 100644
--- a/kubernetes/keycloak/values.yaml
+++ b/kubernetes/keycloak/values.yaml
@@ -58,7 +58,7 @@ service:
# This block is for setting up the ingress for more information can be found here: https://kubernetes.io/docs/concepts/services-networking/ingress/
ingress:
- enabled: true
+ enabled: false
className: "nginx"
annotations: {}
# kubernetes.io/ingress.class: nginx
diff --git a/kubernetes/manager/Chart.yaml b/kubernetes/manager/Chart.yaml
index cf55fb6df..b50600adc 100644
--- a/kubernetes/manager/Chart.yaml
+++ b/kubernetes/manager/Chart.yaml
@@ -6,5 +6,4 @@ type: application
version: 0.1.0
-appVersion: "1.3.5"
-
+appVersion: "1.7.0"
diff --git a/kubernetes/manager/templates/_helpers.tpl b/kubernetes/manager/templates/_helpers.tpl
index dfdf6f972..67fcbe0d4 100644
--- a/kubernetes/manager/templates/_helpers.tpl
+++ b/kubernetes/manager/templates/_helpers.tpl
@@ -60,3 +60,32 @@ Create the name of the service account to use
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}
+
+{{/*
+Create the name of the mqtt service to use
+*/}}
+{{- define "manager.mqttServiceName" -}}
+{{- printf "%s-%s" (include "manager.fullname" .) "mqtt" | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{- define "manager.mqttService.labels" -}}
+helm.sh/chart: {{ include "manager.chart" . }}
+app.kubernetes.io/name: {{ include "manager.mqttServiceName" . }}
+{{- if .Chart.AppVersion }}
+app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
+{{- end }}
+app.kubernetes.io/managed-by: {{ .Release.Service }}
+{{- end }}
+
+{{- define "manager.mqttsServiceName" -}}
+{{- printf "%s-%s" (include "manager.fullname" .) "mqtts" | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{- define "manager.mqttsService.labels" -}}
+helm.sh/chart: {{ include "manager.chart" . }}
+app.kubernetes.io/name: {{ include "manager.mqttsServiceName" . }}
+{{- if .Chart.AppVersion }}
+app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
+{{- end }}
+app.kubernetes.io/managed-by: {{ .Release.Service }}
+{{- end }}
diff --git a/kubernetes/manager/templates/deployment.yaml b/kubernetes/manager/templates/deployment.yaml
index 4a0bbac87..ea03b7da8 100644
--- a/kubernetes/manager/templates/deployment.yaml
+++ b/kubernetes/manager/templates/deployment.yaml
@@ -34,10 +34,6 @@ spec:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
-# ports:
-# - name: http
-# containerPort: {{ .Values.service.port }}
-# protocol: TCP
livenessProbe:
{{- toYaml .Values.livenessProbe | nindent 12 }}
readinessProbe:
diff --git a/kubernetes/manager/templates/ingress.yaml b/kubernetes/manager/templates/ingress.yaml
index 79ec1fa07..10c46822f 100644
--- a/kubernetes/manager/templates/ingress.yaml
+++ b/kubernetes/manager/templates/ingress.yaml
@@ -39,7 +39,7 @@ spec:
service:
name: {{ include "manager.fullname" $ }}
port:
- number: {{ $.Values.service.port }}
+ number: {{ $.Values.service.http.port }}
{{- end }}
{{- end }}
{{- end }}
diff --git a/kubernetes/manager/templates/pvc.yaml b/kubernetes/manager/templates/pvc.yaml
index 14acf60d0..c93755c7f 100644
--- a/kubernetes/manager/templates/pvc.yaml
+++ b/kubernetes/manager/templates/pvc.yaml
@@ -7,6 +7,6 @@ spec:
- ReadWriteOnce
resources:
requests:
- storage: 1Gi
+ storage: {{ .Values.volumeSize }}
volumeName: manager-data-pv
storageClassName: manual
diff --git a/kubernetes/manager/templates/service.yaml b/kubernetes/manager/templates/service.yaml
index 25e29c0d5..f9a6f9256 100644
--- a/kubernetes/manager/templates/service.yaml
+++ b/kubernetes/manager/templates/service.yaml
@@ -5,11 +5,69 @@ metadata:
labels:
{{- include "manager.labels" . | nindent 4 }}
spec:
- type: {{ .Values.service.type }}
+ type: {{ .Values.service.http.type }}
ports:
- - port: {{ .Values.service.port }}
+ - port: {{ .Values.service.http.port }}
targetPort: 8080
protocol: TCP
name: http
+ {{- if and (not (.Values.service.mqtt.enabled)) (not (.Values.service.mqtts.enabled)) }}
+ - name: mqtt
+ port: 1883
+ targetPort: 1883
+ protocol: TCP
+ {{- end }}
+ {{- if .Values.or.metricsEnabled }}
+ - name: metrics
+ port: {{ .Values.or.metricsPort }}
+ targetPort: {{ .Values.or.metricsPort }}
+ protocol: TCP
+ {{- end }}
+ selector:
+ {{- include "manager.selectorLabels" . | nindent 4 }}
+---
+{{- if .Values.service.mqtt.enabled -}}
+apiVersion: v1
+kind: Service
+metadata:
+ name: {{ include "manager.mqttServiceName" . }}
+ labels:
+ {{- include "manager.mqttService.labels" . | nindent 4 }}
+ {{- with .Values.service.mqtt.annotations }}
+ annotations:
+ {{- toYaml . | nindent 4 }}
+ {{- end }}
+
+spec:
+ selector:
+ {{- include "manager.selectorLabels" . | nindent 4 }}
+ type: LoadBalancer
+ ports:
+ - name: mqtt
+ port: 1883
+ targetPort: 1883
+ protocol: TCP
+---
+{{- end }}
+{{- if .Values.service.mqtts.enabled -}}
+apiVersion: v1
+kind: Service
+metadata:
+ name: {{ include "manager.mqttsServiceName" . }}
+ labels:
+ {{- include "manager.mqttsService.labels" . | nindent 4 }}
+ {{- with .Values.service.mqtts.annotations }}
+ annotations:
+ {{- toYaml . | nindent 4 }}
+ {{- end }}
+
+spec:
selector:
{{- include "manager.selectorLabels" . | nindent 4 }}
+ type: LoadBalancer
+ ports:
+ - name: mqtts
+ port: 8883
+ targetPort: 1883
+ protocol: TCP
+{{- end }}
diff --git a/kubernetes/manager/templates/tests/test-connection.yaml b/kubernetes/manager/templates/tests/test-connection.yaml
index c8096dea2..8475f473c 100644
--- a/kubernetes/manager/templates/tests/test-connection.yaml
+++ b/kubernetes/manager/templates/tests/test-connection.yaml
@@ -11,5 +11,5 @@ spec:
- name: wget
image: busybox
command: ['wget']
- args: ['{{ include "manager.fullname" . }}:{{ .Values.service.port }}']
+ args: ['{{ include "manager.fullname" . }}:{{ .Values.service.http.port }}']
restartPolicy: Never
diff --git a/kubernetes/manager/values-demo.yaml b/kubernetes/manager/values-demo.yaml
new file mode 100644
index 000000000..8d84481d6
--- /dev/null
+++ b/kubernetes/manager/values-demo.yaml
@@ -0,0 +1,21 @@
+image:
+ repository: 134517981306.dkr.ecr.eu-west-1.amazonaws.com/openremote/manager
+ tag: demo
+
+livenessProbe:
+ httpGet:
+ path: /
+ port: 8080
+ initialDelaySeconds: 120
+ periodSeconds: 5
+readinessProbe:
+ httpGet:
+ path: /
+ port: 8080
+ initialDelaySeconds: 120
+ periodSeconds: 5
+
+or:
+ env:
+ - name: OR_SETUP_TYPE
+ value: "demo"
diff --git a/kubernetes/manager/values-eks.yaml b/kubernetes/manager/values-eks.yaml
new file mode 100644
index 000000000..1de0d411d
--- /dev/null
+++ b/kubernetes/manager/values-eks.yaml
@@ -0,0 +1,51 @@
+livenessProbe:
+ httpGet:
+ path: /
+ port: 8080
+ initialDelaySeconds: 5
+ periodSeconds: 20
+readinessProbe:
+ httpGet:
+ path: /
+ port: 8080
+ initialDelaySeconds: 5
+ periodSeconds: 20
+ingress:
+ enabled: true
+ hosts:
+ - paths:
+ - path: /
+ pathType: Prefix
+ className: "alb"
+ annotations:
+ alb.ingress.kubernetes.io/scheme: internet-facing
+ alb.ingress.kubernetes.io/target-type: ip
+ alb.ingress.kubernetes.io/group.name: eric-test-ig
+ alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-west-1:463235666115:certificate/...
+ tls: {}
+service:
+ mqtt:
+ enabled: true
+ annotations:
+ service.beta.kubernetes.io/aws-load-balancer-type: "external"
+ service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
+ service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
+ mqtts:
+ enabled: true
+ annotations:
+ service.beta.kubernetes.io/aws-load-balancer-type: "external"
+ service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
+ service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
+ service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu-west-1:463235666115:certificate/...
+affinity:
+ nodeAffinity:
+ requiredDuringSchedulingIgnoredDuringExecution:
+ nodeSelectorTerms:
+ - matchExpressions:
+ - key: topology.kubernetes.io/zone
+ operator: In
+ values:
+ - eu-west-1a
+
+or:
+ hostname: k8stest.openremote.app
diff --git a/kubernetes/manager/values-haproxy-eks.yaml b/kubernetes/manager/values-haproxy-eks.yaml
new file mode 100644
index 000000000..f36adcba2
--- /dev/null
+++ b/kubernetes/manager/values-haproxy-eks.yaml
@@ -0,0 +1,15 @@
+ingress:
+ enabled: false
+
+affinity:
+ nodeAffinity:
+ requiredDuringSchedulingIgnoredDuringExecution:
+ nodeSelectorTerms:
+ - matchExpressions:
+ - key: topology.kubernetes.io/zone
+ operator: In
+ values:
+ - eu-west-1a
+
+or:
+ hostname: k8stest.openremote.app
diff --git a/kubernetes/manager/values.yaml b/kubernetes/manager/values.yaml
index 3cc485be5..92371530c 100644
--- a/kubernetes/manager/values.yaml
+++ b/kubernetes/manager/values.yaml
@@ -49,16 +49,23 @@ securityContext: {}
# runAsNonRoot: true
# runAsUser: 1000
-# This is for setting up a service more information can be found here: https://kubernetes.io/docs/concepts/services-networking/service/
+# HTTP service is always active but its port can be adapted
+# MQTT(S) services values are for directly exposing the MQTT(S) ports of the manager,
+# set to false when using HA Proxy
service:
- # This sets the service type more information can be found here: https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types
- type: ClusterIP
- # This sets the ports more information can be found here: https://kubernetes.io/docs/concepts/services-networking/service/#field-spec-ports
- port: 8080
-
-# This block is for setting up the ingress for more information can be found here: https://kubernetes.io/docs/concepts/services-networking/ingress/
+ http:
+ type: ClusterIP
+ port: 8080
+ mqtt:
+ enabled: false
+ annotations: {}
+ mqtts:
+ enabled: false
+ annotations: {}
+
+# Expose an Ingress for direct HTTP(S) access to the manager, disable when using HA Proxy
ingress:
- enabled: true
+ enabled: false
className: "nginx"
annotations: {}
# kubernetes.io/ingress.class: nginx
@@ -99,6 +106,9 @@ readinessProbe:
initialDelaySeconds: 5
periodSeconds: 5
+# Size of volume requested in PVC, must be aligned with available PV
+volumeSize: "1Gi"
+
# Additional volumes on the output Deployment definition.
volumes: []
# - name: foo
diff --git a/kubernetes/or-setup/Chart.yaml b/kubernetes/or-setup/Chart.yaml
new file mode 100644
index 000000000..441c11c80
--- /dev/null
+++ b/kubernetes/or-setup/Chart.yaml
@@ -0,0 +1,8 @@
+apiVersion: v2
+name: or-setup
+description: A Helm chart for setting up additional elements required for an OR stack deployment in k8s
+
+type: application
+
+version: 0.1.0
+
diff --git a/kubernetes/or-setup/templates/openremote-manager-pv.yaml b/kubernetes/or-setup/templates/openremote-manager-pv.yaml
new file mode 100644
index 000000000..1e1c0f6b9
--- /dev/null
+++ b/kubernetes/or-setup/templates/openremote-manager-pv.yaml
@@ -0,0 +1,23 @@
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+ name: manager-data-pv
+{{- if .Values.aws.enabled }}
+ labels:
+ type: amazonEBS
+{{- end }}
+spec:
+ capacity:
+ storage: {{ .Values.managerVolumeSize }}
+ accessModes:
+ - ReadWriteOnce
+ {{- if .Values.aws.enabled }}
+ awsElasticBlockStore:
+ volumeID: {{ .Values.aws.managerVolumeId }}
+ fsType: ext4
+ {{- else }}
+ hostPath:
+ path: "{{ .Values.basePath}}/manager-data"
+ {{- end }}
+ persistentVolumeReclaimPolicy: Retain
+ storageClassName: manual
diff --git a/kubernetes/or-setup/templates/openremote-psql-pv.yaml b/kubernetes/or-setup/templates/openremote-psql-pv.yaml
new file mode 100644
index 000000000..63e70c6b8
--- /dev/null
+++ b/kubernetes/or-setup/templates/openremote-psql-pv.yaml
@@ -0,0 +1,24 @@
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+ name: postgresql-data-pv
+ {{- if .Values.aws.enabled }}
+ labels:
+ type: amazonEBS
+ {{- end }}
+
+spec:
+ capacity:
+ storage: {{ .Values.psqlVolumeSize }}
+ accessModes:
+ - ReadWriteOnce
+ {{- if .Values.aws.enabled }}
+ awsElasticBlockStore:
+ volumeID: {{ .Values.aws.psqlVolumeId }}
+ fsType: ext4
+ {{- else }}
+ hostPath:
+ path: "{{ .Values.basePath}}/postgresql-data"
+ {{- end }}
+ persistentVolumeReclaimPolicy: Retain
+ storageClassName: manual
diff --git a/kubernetes/or-setup/templates/openremote-secret.yaml b/kubernetes/or-setup/templates/openremote-secret.yaml
new file mode 100644
index 000000000..8cb29c8b0
--- /dev/null
+++ b/kubernetes/or-setup/templates/openremote-secret.yaml
@@ -0,0 +1,9 @@
+apiVersion: v1
+kind: Secret
+metadata:
+ name: openremote-secret
+type: Opaque
+data:
+ admin-password: c2VjcmV0
+ postgresql-username: cG9zdGdyZXM=
+ postgresql-password: cG9zdGdyZXM=
diff --git a/kubernetes/or-setup/values.yaml b/kubernetes/or-setup/values.yaml
new file mode 100644
index 000000000..1aa37572e
--- /dev/null
+++ b/kubernetes/or-setup/values.yaml
@@ -0,0 +1,14 @@
+managerVolumeSize: "1Gi"
+psqlVolumeSize: "1Gi"
+
+# Configuration when deploying under EKS in AWS
+aws:
+ # Set enabled to true if deployed under AWS
+ enabled: false
+ # Id of the EBS volume used for manager persistent storage
+ managerVolumeId: vol-xxx
+ # Id of the EBS volume used for postgreSQL persistent storage
+ psqlVolumeId: vol-xxx
+
+# Root path for hostpath storage when deployed locally. On macOS, this needs to be under user's home folder
+basePath: "/Users/username/openremote"
diff --git a/kubernetes/postgresql/Chart.yaml b/kubernetes/postgresql/Chart.yaml
index f51f6e557..01bfd4e13 100644
--- a/kubernetes/postgresql/Chart.yaml
+++ b/kubernetes/postgresql/Chart.yaml
@@ -7,4 +7,3 @@ type: application
version: 0.1.0
appVersion: "15.6.0.4"
-# Using latest for now but would be better to have specific SemVer version
diff --git a/kubernetes/postgresql/templates/pvc.yaml b/kubernetes/postgresql/templates/pvc.yaml
index 45efda72c..e1f79c6f2 100644
--- a/kubernetes/postgresql/templates/pvc.yaml
+++ b/kubernetes/postgresql/templates/pvc.yaml
@@ -7,6 +7,6 @@ spec:
- ReadWriteOnce
resources:
requests:
- storage: 1Gi
+ storage: {{ .Values.volumeSize }}
volumeName: postgresql-data-pv
storageClassName: manual
diff --git a/kubernetes/postgresql/templates/statefulset.yaml b/kubernetes/postgresql/templates/statefulset.yaml
index 2ef3ec5b3..decec1900 100644
--- a/kubernetes/postgresql/templates/statefulset.yaml
+++ b/kubernetes/postgresql/templates/statefulset.yaml
@@ -68,6 +68,9 @@ spec:
{{- end }}
volumeMounts:
- mountPath: /var/lib/postgresql/data
+ {{- if .Values.useSubPath }}
+ subPath: psql-data
+ {{- end }}
name: postgresql-data
{{- with .Values.volumeMounts }}
{{- toYaml . | nindent 12 }}
@@ -81,6 +84,9 @@ spec:
runAsUser: 0
volumeMounts:
- mountPath: /var/lib/postgresql/data
+ {{- if .Values.useSubPath }}
+ subPath: psql-data
+ {{- end }}
name: postgresql-data
{{- end }}
volumes:
diff --git a/kubernetes/postgresql/values-eks.yaml b/kubernetes/postgresql/values-eks.yaml
new file mode 100644
index 000000000..effbd7369
--- /dev/null
+++ b/kubernetes/postgresql/values-eks.yaml
@@ -0,0 +1,12 @@
+requiresPermissionsFix: true
+useSubPath: true
+
+affinity:
+ nodeAffinity:
+ requiredDuringSchedulingIgnoredDuringExecution:
+ nodeSelectorTerms:
+ - matchExpressions:
+ - key: topology.kubernetes.io/zone
+ operator: In
+ values:
+ - eu-west-1a
diff --git a/kubernetes/postgresql/values.yaml b/kubernetes/postgresql/values.yaml
index 7a54a1974..75a998789 100644
--- a/kubernetes/postgresql/values.yaml
+++ b/kubernetes/postgresql/values.yaml
@@ -38,8 +38,11 @@ podAnnotations: {}
# For more information checkout: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
podLabels: {}
-podSecurityContext: {}
- # fsGroup: 2000
+# Align with uid/gid used by Openremote PostgreSQL image
+podSecurityContext:
+ runAsUser: 70
+ runAsGroup: 70
+ fsGroup: 70
securityContext: {}
# capabilities:
@@ -78,6 +81,9 @@ readinessProbe:
command:
- pg_isready
+# Size of volume requested in PVC, must be aligned with available PV
+volumeSize: "1Gi"
+
# Additional volumes on the output Deployment definition.
volumes: []
# - name: foo
@@ -91,10 +97,13 @@ volumeMounts: []
# mountPath: "/etc/foo"
# readOnly: true
-# Should an init container run to fix the permissions on the mounted data volume ?
+# Should an init container run to fix the permissions on the mounted data volume?
# This is not required on macOS but is on linux
requiresPermissionsFix: false
+# Indicates if subpath must be used when mounting the psql data volume, required for AWS EBS volumes
+useSubPath: false
+
nodeSelector: {}
tolerations: []
diff --git a/kubernetes/proxy/.helmignore b/kubernetes/proxy/.helmignore
new file mode 100644
index 000000000..0e8a0eb36
--- /dev/null
+++ b/kubernetes/proxy/.helmignore
@@ -0,0 +1,23 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*.orig
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/
diff --git a/kubernetes/proxy/Chart.yaml b/kubernetes/proxy/Chart.yaml
new file mode 100644
index 000000000..ea13fc3ff
--- /dev/null
+++ b/kubernetes/proxy/Chart.yaml
@@ -0,0 +1,9 @@
+apiVersion: v2
+name: proxy
+description: A Helm chart for deploying OR HAProxy in Kubernetes
+
+type: application
+
+version: 0.1.0
+
+appVersion: "2.9.15.0"
diff --git a/kubernetes/proxy/templates/NOTES.txt b/kubernetes/proxy/templates/NOTES.txt
new file mode 100644
index 000000000..9442cddfc
--- /dev/null
+++ b/kubernetes/proxy/templates/NOTES.txt
@@ -0,0 +1 @@
+OpenRemote HAProxy {{ .Chart.AppVersion }}
diff --git a/kubernetes/proxy/templates/_helpers.tpl b/kubernetes/proxy/templates/_helpers.tpl
new file mode 100644
index 000000000..dab065ea8
--- /dev/null
+++ b/kubernetes/proxy/templates/_helpers.tpl
@@ -0,0 +1,62 @@
+{{/*
+Expand the name of the chart.
+*/}}
+{{- define "proxy.name" -}}
+{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{/*
+Create a default fully qualified app name.
+We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
+If release name contains chart name it will be used as a full name.
+*/}}
+{{- define "proxy.fullname" -}}
+{{- if .Values.fullnameOverride }}
+{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- $name := default .Chart.Name .Values.nameOverride }}
+{{- if contains $name .Release.Name }}
+{{- .Release.Name | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
+{{- end }}
+{{- end }}
+{{- end }}
+
+{{/*
+Create chart name and version as used by the chart label.
+*/}}
+{{- define "proxy.chart" -}}
+{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{/*
+Common labels
+*/}}
+{{- define "proxy.labels" -}}
+helm.sh/chart: {{ include "proxy.chart" . }}
+{{ include "proxy.selectorLabels" . }}
+{{- if .Chart.AppVersion }}
+app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
+{{- end }}
+app.kubernetes.io/managed-by: {{ .Release.Service }}
+{{- end }}
+
+{{/*
+Selector labels
+*/}}
+{{- define "proxy.selectorLabels" -}}
+app.kubernetes.io/name: {{ include "proxy.name" . }}
+app.kubernetes.io/instance: {{ .Release.Name }}
+{{- end }}
+
+{{/*
+Create the name of the service account to use
+*/}}
+{{- define "proxy.serviceAccountName" -}}
+{{- if .Values.serviceAccount.create }}
+{{- default (include "proxy.fullname" .) .Values.serviceAccount.name }}
+{{- else }}
+{{- default "default" .Values.serviceAccount.name }}
+{{- end }}
+{{- end }}
diff --git a/kubernetes/proxy/templates/deployment.yaml b/kubernetes/proxy/templates/deployment.yaml
new file mode 100644
index 000000000..3d171b49a
--- /dev/null
+++ b/kubernetes/proxy/templates/deployment.yaml
@@ -0,0 +1,114 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: {{ include "proxy.fullname" . }}
+ labels:
+ {{- include "proxy.labels" . | nindent 4 }}
+spec:
+ replicas: {{ .Values.replicaCount }}
+ selector:
+ matchLabels:
+ {{- include "proxy.selectorLabels" . | nindent 6 }}
+ template:
+ metadata:
+ {{- with .Values.podAnnotations }}
+ annotations:
+ {{- toYaml . | nindent 8 }}
+ {{- end }}
+ labels:
+ {{- include "proxy.labels" . | nindent 8 }}
+ {{- with .Values.podLabels }}
+ {{- toYaml . | nindent 8 }}
+ {{- end }}
+ spec:
+ {{- with .Values.imagePullSecrets }}
+ imagePullSecrets:
+ {{- toYaml . | nindent 8 }}
+ {{- end }}
+ serviceAccountName: {{ include "proxy.serviceAccountName" . }}
+ securityContext:
+ {{- toYaml .Values.podSecurityContext | nindent 8 }}
+ containers:
+ - name: {{ .Chart.Name }}
+ securityContext:
+ {{- toYaml .Values.securityContext | nindent 12 }}
+ image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
+ imagePullPolicy: {{ .Values.image.pullPolicy }}
+# ports:
+# - name: http
+# containerPort: {{ .Values.service.port }}
+# protocol: TCP
+ livenessProbe:
+ {{- toYaml .Values.livenessProbe | nindent 12 }}
+ readinessProbe:
+ {{- toYaml .Values.readinessProbe | nindent 12 }}
+ resources:
+ {{- toYaml .Values.resources | nindent 12 }}
+ env:
+ - name: TZ
+ value: "{{ .Values.or.timeZone }}"
+ - name: LOGFILE
+ value: "{{ .Values.or.proxyLogfile }}"
+ - name: PROXY_LOGLEVEL
+ value: "{{ .Values.or.proxyLoglevel }}"
+ - name: DOMAINNAME
+ value: "{{ .Values.or.hostname }}"
+ - name: DOMAINNAMES
+ value: "{{ .Values.or.additionalHostnames }}"
+ - name: HAPROXY_CONFIG
+ value: "{{ .Values.or.haproxyConfig }}"
+ - name: HTTP_PORT
+ value: "{{ .Values.or.httpPort }}"
+ - name: HTTPS_PORT
+ value: "{{ .Values.or.httpsPort }}"
+ - name: HTTPS_FORWARDED_PORT
+ value: "{{ .Values.or.httpsForwardedPort }}"
+ - name: NAMESERVER
+ value: "{{ .Values.or.nameserver }}"
+ - name: LE_EMAIL
+ value: "{{ .Values.or.leEmail }}"
+ - name: LE_EXTRA_ARGS
+ value: "{{ .Values.or.leExtraArgs }}"
+ - name: AWS_ROUTE53_ROLE
+ value: "{{ .Values.or.awsRoute53Role }}"
+ - name: MANAGER_HOST
+ value: "{{ .Values.or.managerHost }}"
+ - name: MANAGER_WEB_PORT
+ value: "{{ .Values.or.managerWebPort }}"
+ - name: MANAGER_MQTT_PORT
+ value: "{{ .Values.or.managerMqttPort }}"
+ {{- if hasKey .Values.or "managerPathPrefix" }}
+ - name: MANAGER_PATH_PREFIX
+ value: "{{ .Values.or.managerPathPrefix }}"
+ {{- end }}
+ - name: KEYCLOAK_HOST
+ value: "{{ .Values.or.keycloakHost }}"
+ - name: KEYCLOAK_PORT
+ value: "{{ .Values.or.keycloakPort }}"
+ {{- if hasKey .Values.or "keycloakPathPrefix" }}
+ - name: KEYCLOAK_PATH_PREFIX
+ value: "{{ .Values.or.keycloakPathPrefix }}"
+ {{- end }}
+ {{- with .Values.or.env }}
+ {{- toYaml . | nindent 12 }}
+ {{- end }}
+ {{- with .Values.volumeMounts }}
+ volumeMounts:
+ {{- toYaml . | nindent 12 }}
+ {{- end }}
+ {{- with .Values.volumes }}
+ volumes:
+ {{- toYaml . | nindent 8 }}
+ {{- end }}
+ {{- with .Values.nodeSelector }}
+ nodeSelector:
+ {{- toYaml . | nindent 8 }}
+ {{- end }}
+ {{- with .Values.affinity }}
+ affinity:
+ {{- toYaml . | nindent 8 }}
+ {{- end }}
+ {{- with .Values.tolerations }}
+ tolerations:
+ {{- toYaml . | nindent 8 }}
+ {{- end }}
diff --git a/kubernetes/proxy/templates/service.yaml b/kubernetes/proxy/templates/service.yaml
new file mode 100644
index 000000000..239ce8a1a
--- /dev/null
+++ b/kubernetes/proxy/templates/service.yaml
@@ -0,0 +1,31 @@
+apiVersion: v1
+kind: Service
+metadata:
+ name: {{ include "proxy.fullname" . }}
+ labels:
+ {{- include "proxy.labels" . | nindent 4 }}
+ {{- with .Values.service.annotations }}
+ annotations:
+ {{- toYaml . | nindent 4 }}
+ {{- end }}
+spec:
+ type: {{ .Values.service.type }}
+ ports:
+ - port: {{ .Values.service.port }}
+ targetPort: {{ .Values.or.httpPort }}
+ protocol: TCP
+ name: http
+ - port: 443
+ targetPort: {{ .Values.or.httpsPort }}
+ protocol: TCP
+ name: https
+ - port: 8883
+ targetPort: 8883
+ protocol: TCP
+ name: mqtts
+ - port: 8404
+ targetPort: 8404
+ protocol: TCP
+ name: metrics
+ selector:
+ {{- include "proxy.selectorLabels" . | nindent 4 }}
diff --git a/kubernetes/proxy/templates/serviceaccount.yaml b/kubernetes/proxy/templates/serviceaccount.yaml
new file mode 100644
index 000000000..a66559b0c
--- /dev/null
+++ b/kubernetes/proxy/templates/serviceaccount.yaml
@@ -0,0 +1,13 @@
+{{- if .Values.serviceAccount.create -}}
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+ name: {{ include "proxy.serviceAccountName" . }}
+ labels:
+ {{- include "proxy.labels" . | nindent 4 }}
+ {{- with .Values.serviceAccount.annotations }}
+ annotations:
+ {{- toYaml . | nindent 4 }}
+ {{- end }}
+automountServiceAccountToken: {{ .Values.serviceAccount.automount }}
+{{- end }}
diff --git a/kubernetes/proxy/templates/tests/test-connection.yaml b/kubernetes/proxy/templates/tests/test-connection.yaml
new file mode 100644
index 000000000..79f35f2da
--- /dev/null
+++ b/kubernetes/proxy/templates/tests/test-connection.yaml
@@ -0,0 +1,15 @@
+apiVersion: v1
+kind: Pod
+metadata:
+ name: "{{ include "proxy.fullname" . }}-test-connection"
+ labels:
+ {{- include "proxy.labels" . | nindent 4 }}
+ annotations:
+ "helm.sh/hook": test
+spec:
+ containers:
+ - name: wget
+ image: busybox
+ command: ['wget']
+ args: ['{{ include "proxy.fullname" . }}:{{ .Values.service.port }}']
+ restartPolicy: Never
diff --git a/kubernetes/proxy/values-eks.yaml b/kubernetes/proxy/values-eks.yaml
new file mode 100644
index 000000000..dad8438fd
--- /dev/null
+++ b/kubernetes/proxy/values-eks.yaml
@@ -0,0 +1,8 @@
+service:
+ annotations:
+ service.beta.kubernetes.io/aws-load-balancer-type: "external"
+ service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
+ service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
+
+or:
+ hostname: "testmanager.openremote.app"
diff --git a/kubernetes/proxy/values.yaml b/kubernetes/proxy/values.yaml
new file mode 100644
index 000000000..b63c10cd5
--- /dev/null
+++ b/kubernetes/proxy/values.yaml
@@ -0,0 +1,165 @@
+# Default values for proxy.
+# This is a YAML-formatted file.
+# Declare variables to be passed into your templates.
+
+# This will set the replicaset count more information can be found here: https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/
+replicaCount: 1
+
+# This sets the container image more information can be found here: https://kubernetes.io/docs/concepts/containers/images/
+image:
+ repository: openremote/proxy
+ # This sets the pull policy for images.
+ pullPolicy: IfNotPresent
+ # Overrides the image tag whose default is the chart appVersion.
+ tag: ""
+
+# This is for the secretes for pulling an image from a private repository more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
+imagePullSecrets: []
+# This is to override the chart name.
+nameOverride: ""
+fullnameOverride: ""
+
+# This section builds out the service account more information can be found here: https://kubernetes.io/docs/concepts/security/service-accounts/
+serviceAccount:
+ # Specifies whether a service account should be created
+ create: true
+ # Automatically mount a ServiceAccount's API credentials?
+ automount: true
+ # Annotations to add to the service account
+ annotations: {}
+ # The name of the service account to use.
+ # If not set and create is true, a name is generated using the fullname template
+ name: ""
+
+# This is for setting Kubernetes Annotations to a Pod.
+# For more information checkout: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
+podAnnotations: {}
+# This is for setting Kubernetes Labels to a Pod.
+# For more information checkout: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
+podLabels: {}
+
+podSecurityContext: {}
+ # fsGroup: 2000
+
+securityContext: {}
+ # capabilities:
+ # drop:
+ # - ALL
+ # readOnlyRootFilesystem: true
+ # runAsNonRoot: true
+ # runAsUser: 1000
+
+# This is for setting up a service more information can be found here: https://kubernetes.io/docs/concepts/services-networking/service/
+service:
+ # This sets the service type more information can be found here: https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types
+ type: LoadBalancer
+ # This sets the ports more information can be found here: https://kubernetes.io/docs/concepts/services-networking/service/#field-spec-ports
+ port: 80
+ annotations: {}
+
+resources: {}
+ # We usually recommend not to specify default resources and to leave this as a conscious
+ # choice for the user. This also increases chances charts run on environments with little
+ # resources, such as Minikube. If you do want to specify resources, uncomment the following
+ # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
+ # limits:
+ # cpu: 100m
+ # memory: 128Mi
+ # requests:
+ # cpu: 100m
+ # memory: 128Mi
+
+# This is to setup the liveness and readiness probes more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
+livenessProbe:
+ httpGet:
+ path: /docker-health
+ port: 8080
+readinessProbe:
+ httpGet:
+ path: /docker-health
+ port: 8080
+
+# Additional volumes on the output Deployment definition.
+volumes: []
+# - name: foo
+# secret:
+# secretName: mysecret
+# optional: false
+
+# Additional volumeMounts on the output Deployment definition.
+volumeMounts: []
+# - name: foo
+# mountPath: "/etc/foo"
+# readOnly: true
+
+nodeSelector: {}
+
+tolerations: []
+
+affinity: {}
+
+or:
+ # Additional environment variables passed to the container
+ env: []
+ # - name: foo
+ # value: bar
+
+ # It is important that all services have the same timezone and are time synchronized.
+ # Bearer tokens are only valid for minutes, and authentication fails if Keycloak drifts
+ timeZone: "Europe/Amsterdam"
+
+ # File to which entrypoint logs are written in addition to std out
+ proxyLogfile: "/var/log/proxy.log"
+
+ # Customize proxy log, defaults to 'notice', requests are logged at 'info'.
+ # Available levels are: emerg, alert, crit, err, warning, notice, info, debug
+ proxyLoglevel: "notice"
+
+ # IANA TLD subdomain for which an SSL certificate should be requested (leave blank for private/externally
+ # generated certificates).
+ hostname: "localhost"
+
+ # IANA TLD comma separated subdomains for which an SSL certificates should be requested (leave blank for
+ # private/externally generated certificates); this is an alternative to DOMAINNAME for multiple domains.
+ additionalHostnames: ""
+
+ # USE A CUSTOM PROXY CONFIG - COPY FROM https://raw.githubusercontent.com/openremote/proxy/main/haproxy.cfg
+ haproxyConfig: "/etc/haproxy/haproxy.cfg"
+
+ # Email address for SSL certificate issuance and expiration notifications.
+ leEmail: ""
+
+ # Enable --staging will manage fake certificates, so avoid hitting the rate limits of Let's Encrypt
+ # when testing (this is important, you are limited to 5 duplicate certificates per week!)
+ # e.g. '--staging'
+ leExtraArgs: ""
+
+ # Provide this value if requesting wildcard certs with Route53 DNS zone; this should be the role to assume to
+ # perform route53 operations if calling across AWS accounts
+ awsRoute53Role: ""
+
+ # FQDN of OpenRemote Manager within the cluster
+ managerHost: "manager.default.svc.cluster.local"
+ # Web server port of OpenRemote Manager (default `8080`)
+ managerWebPort: 8080
+ # MQTT broker port of OpenRemote Manager (default `1883`)
+ managerMqttPort: 1883
+ # The path prefix used for OpenRemote Manager HTTP requests (default not set, example: `/openremote`)
+# managerPathPrefix:
+
+ # FQDN of Keycloak server
+ keycloakHost: "keycloak.default.svc.cluster.local"
+ # Web server port of Keycloak server (default `8080`)
+ keycloakPort: 8080
+ # The path prefix used for Keycloak HTTP requests (default not set, example: `/keycloak`)
+# keycloakPathPrefix:
+
+ # The container binds to this port for handling HTTP requests, must be a non-privileged port (>= 1024) when running inside the cluster
+ httpPort: 8080
+ # The container binds to this port for handling HTTPS requests, must be a non-privileged port (>= 1024) when running inside the cluster
+ httpsPort: 8443
+ # The port set in the `X-Forwarded-Port` header of requests sent to the Manager/Keycloak
+ httpsForwardedPort: 443
+ # The nameserver hostname and port used for resolving the Manager/Keycloak hosts
+ # Use `kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}:{.spec.ports[?(@.name=="dns")].port}'` to get the IP of the cluster DNS
+ nameserver: 10.96.0.10:53