100% open-source IoT Platform - Integrate your devices, add intelligence, build custom apps
Java asset-managementbuildingdashboarddataplatformdevice-managementedge-computingenergyenergy-managementenergy-monitorfleet-managementfleet-telematicsiotiot-applicationiot-platformmodbusmqttmqtt-brokerota-updaterules-enginesmartcity
CI/CD cleanup
Rich Turnerda0a6436f250
diff --git a/.ci_cd/README.md b/.ci_cd/README.mdindex c0daa85e9..6148540e4 100644--- a/.ci_cd/README.md+++ b/.ci_cd/README.md@@ -58,14 +58,12 @@ docker image tag updates. The file layout is:# `deploy.sh` Bash scriptBash script that handles the actual deployment to a specific host; environment variables should be already loaded into-the shell or be available in `temp/env` file which will be automatically loaded. If `ssh.env` is found this env file is-also automatically loaded. See [Variables section below](#variables) for available and required values. The `deploy.sh`-in this repository is tailored for deployments on AWS and as such it performs the following functions:+the shell or be available in `temp.env` file (for environment variables not to be sent to the host) or `temp/env` file (for environment variables that should also be sent to the host) which will be automatically loaded. See [Variables section below](#variables) for available and required values. The `deploy.sh`+in this repository is tailored for deployments on AWS and as such it makes use of the AWS scripts included (to use this functionality then the required environment variables must also be defined), AWS functionality used is as follows:-1. Login to AWS - If `AWS_KEY` and `AWS_SECRET` variables are defined, also sets the region to `AWS_REGION`-1. Whitelist deployment runner (machine actioning the deployment) - If `IPV4` variable is defined then this IP address will be added to the `ssh-access` security group in the default VPC-1. Executes deployment docker logic (refer to `deploy.sh` for details)-1. Remove deployment runner from whitelist - Once deployment is completed or failed the deployment runner will be removed from the whitelist+1. Login to AWS+1. Whitelist deployment runner (machine actioning the deployment) - If `IPV4` or `IPV6` variable is defined+1. Revoke deployment runner from whitelist - Once deployment is completed or failed the deployment runner will be removed from the whitelist## Docker compose file resolutionThe docker compose file used for the deployment is resolved as follows; if resolved file does not exist then deployment@@ -107,6 +105,8 @@ The following variables are supported by deployments; any additional variables cavailable to the shell where the docker compose stack is brought up; so any variables required in the docker composeprofile can be specified in any of the places variables are loaded from (github secrets, inputs, and/or env files).+Standard variables:+* `ENVIRONMENT` - Used to control which env file(s), docker compose file and github secrets to use for deployment.* `MANAGER_TAG` - The docker tag to pull for the manager image (if not specified)* `CLEAN_INSTALL` - Indicates if the or_postgresql-data volume should be removed before starting the stack@@ -116,7 +116,7 @@ profile can be specified in any of the places variables are loaded from (github* `SSH_KEY` - SSH private key (this is written to `ssh.key` file and loaded by `deploy.sh` for SSH/SCP commands)* `SSH_PASSWORD` - SSH password (alternative to private key authentication)* `SSH_PORT` - SSH port (default: 22)-* `AWS_KEY` - AWS access key ID+* `AWS_KEY` - AWS access key ID (should be prefixed with `_TEMP_`* `AWS_SECRET` - AWS access key secret* `AWS_REGION` - AWS region to use* `OR_ADMIN_PASSWORD` - Admin password to be set on deployment; sets the OR_ADMIN_PASSWORD env variable@@ -128,7 +128,7 @@ profile can be specified in any of the places variables are loaded from (github* `AWS_EFS_MAP` - Name of EFS map volume to mount at `/deployment.local/map`## Github secrets-Variables are also loaded from Github secrets using the '_{ENVIRONMENT}_' prefix naming convention (note it doesn't use-github environment secrets as these only seem to be supported for public repositories); any secrets that begin with `_`-will only be loaded if they begin with `_{ENVIRONMENT}_` and this prefix will be removed, all other secrets will be-loaded independent of the `ENVIRONMENT` variable.+Variables stored as github secrets can use the following prefixes (the prefixes will be automatically stripped); any github secrets that begin with `_` but don't match the following prefixes as well as secrets that contain `.` will be excluded:++* `_TEMP_` - Indicates that the variable should only be used by the deploy script and should not be made available to the host (useful for AWS credentials etc.)+* `_$ENVIRONMENT_` - Variables prefixed with requested `ENVIRONMENT` name (useful for environment specific variable values)diff --git a/.ci_cd/deploy.sh b/.ci_cd/deploy.shindex eff6b6f23..8f14c7ef8 100644--- a/.ci_cd/deploy.sh+++ b/.ci_cd/deploy.sh@@ -34,13 +34,6 @@ if [ -f "temp/env" ]; thencat temp/envfi-# Check host is defined-if [ -z "$OR_HOSTNAME" ]; then- echo "Host is not set"- exit 1-fi-HOST="$OR_HOSTNAME"-# Load temp environment variables into this sessionif [ -f "temp.env" ]; thenecho "Loading temp environment variables: 'temp.env'"@@ -49,6 +42,13 @@ if [ -f "temp.env" ]; thenset +afi+# Check host is defined+if [ -z "$OR_HOSTNAME" ]; then+ echo "Host is not set"+ exit 1+fi+HOST="$OR_HOSTNAME"+# Copy CI/CD files into temp direcho "Copying CI/CD files into temp dir"if [ "$IS_CUSTOM_PROJECT" == 'true' ]; thendiff --git a/.github/workflows/ci_cd.yml b/.github/workflows/ci_cd.ymlindex b8925e895..761db8348 100644--- a/.github/workflows/ci_cd.yml+++ b/.github/workflows/ci_cd.yml@@ -20,8 +20,7 @@## - '.ci_cd/env/.env'# - '.ci_cd/env/{ENVIRONMENT}.env'-# - secrets without '_' prefix-# - secrets with '_{ENVIRONMENT}_' prefix (secrets with any other '_' prefix will be excluded)+# - secrets# - inputs (from manual trigger using workflow_dispatch)## The above variables are output to 'temp/env' file for each requested deployment. If a secret called 'SSH_PASSWORD' is@@ -84,12 +83,6 @@ on:description: 'Repo branch or commit SHA to checkout'OR_HOSTNAME:description: 'Host to deploy to (e.g. demo.openremote.app)'- SSH_PORT:- description: 'SSH port'- SSH_USER:- description: 'SSH Username'- SSH_PASSWORD:- description: 'SSH password'OR_ADMIN_PASSWORD:description: 'Admin password override'@@ -429,6 +422,7 @@ jobs:if [ -n "$MANAGER_TAGS" ]; thencommand="docker build --push --build-arg GIT_COMMIT=$commitSha --platform linux/amd64,linux/aarch64 $MANAGER_TAGS $buildPath"+ echo "::set-output name=pushRequired::true"elsecommand="docker build --build-arg GIT_COMMIT=$commitSha --platform linux/amd64,linux/aarch64 -t openremote/manager:$commitShaShort $buildPath"fi@@ -496,11 +490,11 @@ jobs:IS_CUSTOM_PROJECT: ${{ steps.check_custom_project.outputs.files_exists }}- name: Login to DockerHub- if: ${{ steps.manager-docker-command.outputs.value != '' || steps.deployment-docker-command.outputs.value != '' }}+ if: ${{ steps.manager-docker-command.outputs.pushRequired == 'true' }}uses: docker/login-action@v1with:- username: ${{ steps.inputs-and-secrets.outputs.DOCKERHUB_USER }}- password: ${{ steps.inputs-and-secrets.outputs.DOCKERHUB_PASSWORD }}+ username: ${{ steps.inputs-and-secrets.outputs._TEMP_DOCKERHUB_USER || steps.inputs-and-secrets.outputs.DOCKERHUB_USER }}+ password: ${{ steps.inputs-and-secrets.outputs._TEMP_DOCKERHUB_PASSWORD || steps.inputs-and-secrets.outputs.DOCKERHUB_PASSWORD }}- name: set up QEMUif: ${{ steps.manager-docker-command.outputs.value != '' || steps.deployment-docker-command.outputs.value != '' }}@@ -618,7 +612,41 @@ jobs:- name: Run install distif: steps.install-command.outputs.value != ''- run: ${{ steps.install-command.outputs.value }}+ shell: python+ run: |+ import json+ import os+ import sys+ import subprocess++ inputsAndSecrets = json.loads(os.getenv("INPUTS_AND_SECRETS"))++ # Output inputs and secrets as environment variables for build+ for key, value in inputsAndSecrets.items():+ if "." in key:+ continue++ # Look for temp and env prefixed keys+ if key.startswith("_"):+ if key.startswith("_TEMP_"):+ key = key.replace("_TEMP_", "")+ envFile = "temp.env"+ else:+ continue++ os.system(f"echo 'Setting environment variable {key}...'")+ os.putenv(key, value)++ buildCmd = os.getenv("CMD")+ result = subprocess.run(f"bash {buildCmd}", shell=True)++ if result.returncode != 0:+ os.system("echo 'installDist failed'")+ sys.exit(result.returncode)++ env:+ CMD: ${{ steps.install-command.outputs.value }}+ INPUTS_AND_SECRETS: ${{ toJSON(steps.inputs-and-secrets.outputs) }}timeout-minutes: 10- name: Run manager docker command@@ -699,31 +727,32 @@ jobs:for key, value in inputsAndSecrets.items():if "." in key:continue-- # Look for env prefixed keys++ envFile = "temp/env"++ # Look for temp and env prefixed keysif key.startswith("_"):- if key.startswith(prefix):+ if key.startswith("_TEMP_"):+ key = key.replace("_TEMP_", "")+ envFile = "temp.env"+ elif key.startswith(prefix):key = key.replace(prefix, "")else:continue-- if key == "DOCKERHUB_PASSWORD":- continue+if key == "github_token":continueelse:os.system(f"echo 'Secret found {key}...'")if key == "SSH_KEY":os.system(f"echo \"{value}\" > ssh.key")- elif key == "SSH_PASSWORD" or key == "AWS_ACCESS_KEY_ID" or key == "AWS_SECRET_ACCESS_KEY":- os.system(f"echo '{key}=\"{value}\"' >> temp.env")else:lines = len(value.split("\n"))if lines > 1:- os.system(f"echo '{key}=\"' >> temp/env")- os.system(f"echo '{value}\"' >> temp/env")+ os.system(f"echo '{key}=\"' >> {envFile}")+ os.system(f"echo '{value}\"' >> {envFile}")else:- os.system(f"echo '{key}=\"{value}\"' >> temp/env")+ os.system(f"echo '{key}=\"{value}\"' >> {envFile}")# Set CIDR environment variableif ipv4 is not None and ipv4 != '':diff --git a/.github/workflows/provision-account.yml b/.github/workflows/provision-account.ymlindex cf7f644bc..afdd55341 100644--- a/.github/workflows/provision-account.yml+++ b/.github/workflows/provision-account.yml@@ -46,7 +46,7 @@ jobs:username: ${{ github.actor }}organization: openremoteteam: CI/CD- GITHUB_TOKEN: ${{ secrets.ACCESS_TOKEN_FOR_READING_ORG_TEAMS }} # Personal access token used to query github (Requires scope: `read:org`)+ GITHUB_TOKEN: ${{ secrets._TEMP_ACCESS_TOKEN_FOR_READING_ORG_TEAMS }} # Personal access token used to query github (Requires scope: `read:org`)- name: Checkoutuses: actions/checkout@v2diff --git a/.github/workflows/provision-host.yml b/.github/workflows/provision-host.ymlindex 79726648d..3ec30396d 100644--- a/.github/workflows/provision-host.yml+++ b/.github/workflows/provision-host.yml@@ -47,7 +47,7 @@ jobs:username: ${{ github.actor }}organization: openremoteteam: CI/CD- GITHUB_TOKEN: ${{ secrets.ACCESS_TOKEN_FOR_READING_ORG_TEAMS }} # Personal access token used to query github (Requires scope: `read:org`)+ GITHUB_TOKEN: ${{ secrets._TEMP_ACCESS_TOKEN_FOR_READING_ORG_TEAMS }} # Personal access token used to query github (Requires scope: `read:org`)- name: Checkoutuses: actions/checkout@v2