Skip to content

armgit / core

Public Imported from GitHub
0

100% open-source IoT Platform - Integrate your devices, add intelligence, build custom apps

CI/CD cleanup

Rich Turnerda0a6436f250
diff --git a/.ci_cd/README.md b/.ci_cd/README.md
index c0daa85e9..6148540e4 100644
--- a/.ci_cd/README.md
+++ b/.ci_cd/README.md
@@ -58,14 +58,12 @@ docker image tag updates. The file layout is:
# `deploy.sh` Bash script
Bash script that handles the actual deployment to a specific host; environment variables should be already loaded into
-the shell or be available in `temp/env` file which will be automatically loaded. If `ssh.env` is found this env file is
-also automatically loaded. See [Variables section below](#variables) for available and required values. The `deploy.sh`
-in this repository is tailored for deployments on AWS and as such it performs the following functions:
+the shell or be available in `temp.env` file (for environment variables not to be sent to the host) or `temp/env` file (for environment variables that should also be sent to the host) which will be automatically loaded. See [Variables section below](#variables) for available and required values. The `deploy.sh`
+in this repository is tailored for deployments on AWS and as such it makes use of the AWS scripts included (to use this functionality then the required environment variables must also be defined), AWS functionality used is as follows:
-1. Login to AWS - If `AWS_KEY` and `AWS_SECRET` variables are defined, also sets the region to `AWS_REGION`
-1. Whitelist deployment runner (machine actioning the deployment) - If `IPV4` variable is defined then this IP address will be added to the `ssh-access` security group in the default VPC
-1. Executes deployment docker logic (refer to `deploy.sh` for details)
-1. Remove deployment runner from whitelist - Once deployment is completed or failed the deployment runner will be removed from the whitelist
+1. Login to AWS
+1. Whitelist deployment runner (machine actioning the deployment) - If `IPV4` or `IPV6` variable is defined
+1. Revoke deployment runner from whitelist - Once deployment is completed or failed the deployment runner will be removed from the whitelist
## Docker compose file resolution
The docker compose file used for the deployment is resolved as follows; if resolved file does not exist then deployment
@@ -107,6 +105,8 @@ The following variables are supported by deployments; any additional variables c
available to the shell where the docker compose stack is brought up; so any variables required in the docker compose
profile can be specified in any of the places variables are loaded from (github secrets, inputs, and/or env files).
+Standard variables:
+
* `ENVIRONMENT` - Used to control which env file(s), docker compose file and github secrets to use for deployment.
* `MANAGER_TAG` - The docker tag to pull for the manager image (if not specified)
* `CLEAN_INSTALL` - Indicates if the or_postgresql-data volume should be removed before starting the stack
@@ -116,7 +116,7 @@ profile can be specified in any of the places variables are loaded from (github
* `SSH_KEY` - SSH private key (this is written to `ssh.key` file and loaded by `deploy.sh` for SSH/SCP commands)
* `SSH_PASSWORD` - SSH password (alternative to private key authentication)
* `SSH_PORT` - SSH port (default: 22)
-* `AWS_KEY` - AWS access key ID
+* `AWS_KEY` - AWS access key ID (should be prefixed with `_TEMP_`
* `AWS_SECRET` - AWS access key secret
* `AWS_REGION` - AWS region to use
* `OR_ADMIN_PASSWORD` - Admin password to be set on deployment; sets the OR_ADMIN_PASSWORD env variable
@@ -128,7 +128,7 @@ profile can be specified in any of the places variables are loaded from (github
* `AWS_EFS_MAP` - Name of EFS map volume to mount at `/deployment.local/map`
## Github secrets
-Variables are also loaded from Github secrets using the '_{ENVIRONMENT}_' prefix naming convention (note it doesn't use
-github environment secrets as these only seem to be supported for public repositories); any secrets that begin with `_`
-will only be loaded if they begin with `_{ENVIRONMENT}_` and this prefix will be removed, all other secrets will be
-loaded independent of the `ENVIRONMENT` variable.
+Variables stored as github secrets can use the following prefixes (the prefixes will be automatically stripped); any github secrets that begin with `_` but don't match the following prefixes as well as secrets that contain `.` will be excluded:
+
+* `_TEMP_` - Indicates that the variable should only be used by the deploy script and should not be made available to the host (useful for AWS credentials etc.)
+* `_$ENVIRONMENT_` - Variables prefixed with requested `ENVIRONMENT` name (useful for environment specific variable values)
diff --git a/.ci_cd/deploy.sh b/.ci_cd/deploy.sh
index eff6b6f23..8f14c7ef8 100644
--- a/.ci_cd/deploy.sh
+++ b/.ci_cd/deploy.sh
@@ -34,13 +34,6 @@ if [ -f "temp/env" ]; then
cat temp/env
fi
-# Check host is defined
-if [ -z "$OR_HOSTNAME" ]; then
- echo "Host is not set"
- exit 1
-fi
-HOST="$OR_HOSTNAME"
-
# Load temp environment variables into this session
if [ -f "temp.env" ]; then
echo "Loading temp environment variables: 'temp.env'"
@@ -49,6 +42,13 @@ if [ -f "temp.env" ]; then
set +a
fi
+# Check host is defined
+if [ -z "$OR_HOSTNAME" ]; then
+ echo "Host is not set"
+ exit 1
+fi
+HOST="$OR_HOSTNAME"
+
# Copy CI/CD files into temp dir
echo "Copying CI/CD files into temp dir"
if [ "$IS_CUSTOM_PROJECT" == 'true' ]; then
diff --git a/.github/workflows/ci_cd.yml b/.github/workflows/ci_cd.yml
index b8925e895..761db8348 100644
--- a/.github/workflows/ci_cd.yml
+++ b/.github/workflows/ci_cd.yml
@@ -20,8 +20,7 @@
#
# - '.ci_cd/env/.env'
# - '.ci_cd/env/{ENVIRONMENT}.env'
-# - secrets without '_' prefix
-# - secrets with '_{ENVIRONMENT}_' prefix (secrets with any other '_' prefix will be excluded)
+# - secrets
# - inputs (from manual trigger using workflow_dispatch)
#
# The above variables are output to 'temp/env' file for each requested deployment. If a secret called 'SSH_PASSWORD' is
@@ -84,12 +83,6 @@ on:
description: 'Repo branch or commit SHA to checkout'
OR_HOSTNAME:
description: 'Host to deploy to (e.g. demo.openremote.app)'
- SSH_PORT:
- description: 'SSH port'
- SSH_USER:
- description: 'SSH Username'
- SSH_PASSWORD:
- description: 'SSH password'
OR_ADMIN_PASSWORD:
description: 'Admin password override'
@@ -429,6 +422,7 @@ jobs:
if [ -n "$MANAGER_TAGS" ]; then
command="docker build --push --build-arg GIT_COMMIT=$commitSha --platform linux/amd64,linux/aarch64 $MANAGER_TAGS $buildPath"
+ echo "::set-output name=pushRequired::true"
else
command="docker build --build-arg GIT_COMMIT=$commitSha --platform linux/amd64,linux/aarch64 -t openremote/manager:$commitShaShort $buildPath"
fi
@@ -496,11 +490,11 @@ jobs:
IS_CUSTOM_PROJECT: ${{ steps.check_custom_project.outputs.files_exists }}
- name: Login to DockerHub
- if: ${{ steps.manager-docker-command.outputs.value != '' || steps.deployment-docker-command.outputs.value != '' }}
+ if: ${{ steps.manager-docker-command.outputs.pushRequired == 'true' }}
uses: docker/login-action@v1
with:
- username: ${{ steps.inputs-and-secrets.outputs.DOCKERHUB_USER }}
- password: ${{ steps.inputs-and-secrets.outputs.DOCKERHUB_PASSWORD }}
+ username: ${{ steps.inputs-and-secrets.outputs._TEMP_DOCKERHUB_USER || steps.inputs-and-secrets.outputs.DOCKERHUB_USER }}
+ password: ${{ steps.inputs-and-secrets.outputs._TEMP_DOCKERHUB_PASSWORD || steps.inputs-and-secrets.outputs.DOCKERHUB_PASSWORD }}
- name: set up QEMU
if: ${{ steps.manager-docker-command.outputs.value != '' || steps.deployment-docker-command.outputs.value != '' }}
@@ -618,7 +612,41 @@ jobs:
- name: Run install dist
if: steps.install-command.outputs.value != ''
- run: ${{ steps.install-command.outputs.value }}
+ shell: python
+ run: |
+ import json
+ import os
+ import sys
+ import subprocess
+
+ inputsAndSecrets = json.loads(os.getenv("INPUTS_AND_SECRETS"))
+
+ # Output inputs and secrets as environment variables for build
+ for key, value in inputsAndSecrets.items():
+ if "." in key:
+ continue
+
+ # Look for temp and env prefixed keys
+ if key.startswith("_"):
+ if key.startswith("_TEMP_"):
+ key = key.replace("_TEMP_", "")
+ envFile = "temp.env"
+ else:
+ continue
+
+ os.system(f"echo 'Setting environment variable {key}...'")
+ os.putenv(key, value)
+
+ buildCmd = os.getenv("CMD")
+ result = subprocess.run(f"bash {buildCmd}", shell=True)
+
+ if result.returncode != 0:
+ os.system("echo 'installDist failed'")
+ sys.exit(result.returncode)
+
+ env:
+ CMD: ${{ steps.install-command.outputs.value }}
+ INPUTS_AND_SECRETS: ${{ toJSON(steps.inputs-and-secrets.outputs) }}
timeout-minutes: 10
- name: Run manager docker command
@@ -699,31 +727,32 @@ jobs:
for key, value in inputsAndSecrets.items():
if "." in key:
continue
-
- # Look for env prefixed keys
+
+ envFile = "temp/env"
+
+ # Look for temp and env prefixed keys
if key.startswith("_"):
- if key.startswith(prefix):
+ if key.startswith("_TEMP_"):
+ key = key.replace("_TEMP_", "")
+ envFile = "temp.env"
+ elif key.startswith(prefix):
key = key.replace(prefix, "")
else:
continue
-
- if key == "DOCKERHUB_PASSWORD":
- continue
+
if key == "github_token":
continue
else:
os.system(f"echo 'Secret found {key}...'")
if key == "SSH_KEY":
os.system(f"echo \"{value}\" > ssh.key")
- elif key == "SSH_PASSWORD" or key == "AWS_ACCESS_KEY_ID" or key == "AWS_SECRET_ACCESS_KEY":
- os.system(f"echo '{key}=\"{value}\"' >> temp.env")
else:
lines = len(value.split("\n"))
if lines > 1:
- os.system(f"echo '{key}=\"' >> temp/env")
- os.system(f"echo '{value}\"' >> temp/env")
+ os.system(f"echo '{key}=\"' >> {envFile}")
+ os.system(f"echo '{value}\"' >> {envFile}")
else:
- os.system(f"echo '{key}=\"{value}\"' >> temp/env")
+ os.system(f"echo '{key}=\"{value}\"' >> {envFile}")
# Set CIDR environment variable
if ipv4 is not None and ipv4 != '':
diff --git a/.github/workflows/provision-account.yml b/.github/workflows/provision-account.yml
index cf7f644bc..afdd55341 100644
--- a/.github/workflows/provision-account.yml
+++ b/.github/workflows/provision-account.yml
@@ -46,7 +46,7 @@ jobs:
username: ${{ github.actor }}
organization: openremote
team: CI/CD
- GITHUB_TOKEN: ${{ secrets.ACCESS_TOKEN_FOR_READING_ORG_TEAMS }} # Personal access token used to query github (Requires scope: `read:org`)
+ GITHUB_TOKEN: ${{ secrets._TEMP_ACCESS_TOKEN_FOR_READING_ORG_TEAMS }} # Personal access token used to query github (Requires scope: `read:org`)
- name: Checkout
uses: actions/checkout@v2
diff --git a/.github/workflows/provision-host.yml b/.github/workflows/provision-host.yml
index 79726648d..3ec30396d 100644
--- a/.github/workflows/provision-host.yml
+++ b/.github/workflows/provision-host.yml
@@ -47,7 +47,7 @@ jobs:
username: ${{ github.actor }}
organization: openremote
team: CI/CD
- GITHUB_TOKEN: ${{ secrets.ACCESS_TOKEN_FOR_READING_ORG_TEAMS }} # Personal access token used to query github (Requires scope: `read:org`)
+ GITHUB_TOKEN: ${{ secrets._TEMP_ACCESS_TOKEN_FOR_READING_ORG_TEAMS }} # Personal access token used to query github (Requires scope: `read:org`)
- name: Checkout
uses: actions/checkout@v2