Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

ci: pin plugin preview donor to PR head (#1631)

Ian Oberste025978044aa
diff --git a/.github/workflows/docker-publish-on-pr.yml b/.github/workflows/docker-publish-on-pr.yml
index de010703..7d43487b 100644
--- a/.github/workflows/docker-publish-on-pr.yml
+++ b/.github/workflows/docker-publish-on-pr.yml
@@ -30,15 +30,56 @@ jobs:
id: guard
env:
GH_TOKEN: ${{ github.token }}
+ EVENT_HEAD: ${{ github.event.pull_request.head.sha }}
+ PR_NUMBER: ${{ github.event.pull_request.number }}
+ REPOSITORY: ${{ github.repository }}
run: |
- files=$(gh pr view "${{ github.event.pull_request.number }}" \
- --repo "${{ github.repository }}" --json files --jq '.files[].path')
- echo "Changed files:"; echo "$files"
- if echo "$files" | grep -qE '^(Dockerfile([.].*)?|\.dockerignore|\.github/)'; then
+ skip_preview() {
echo "skip=true" >> "$GITHUB_OUTPUT"
- echo "::warning::PR modifies the build/CI surface; auto-preview skipped pending manual maintainer build."
+ echo "plugin_changed=false" >> "$GITHUB_OUTPUT"
+ echo "skip_reason=$1" >> "$GITHUB_OUTPUT"
+ echo "::warning::$1"
+ exit 0
+ }
+
+ pr=$(gh pr view "$PR_NUMBER" --repo "$REPOSITORY" \
+ --json changedFiles,headRefOid)
+ head_before=$(printf '%s' "$pr" | jq -r '.headRefOid')
+ expected_files=$(printf '%s' "$pr" | jq -r '.changedFiles')
+ if [ "$head_before" != "$EVENT_HEAD" ]; then
+ skip_preview "PR head changed while this run was queued; the newer synchronize event will publish the preview."
+ fi
+
+ files=$(gh api --paginate --slurp \
+ "repos/$REPOSITORY/pulls/$PR_NUMBER/files?per_page=100")
+ head_after=$(gh pr view "$PR_NUMBER" --repo "$REPOSITORY" \
+ --json headRefOid --jq '.headRefOid')
+ if [ "$head_after" != "$EVENT_HEAD" ]; then
+ skip_preview "PR head changed while files were being inspected; the newer synchronize event will publish the preview."
+ fi
+
+ fetched_files=$(printf '%s' "$files" | jq '[.[][]] | length')
+ if [ "$fetched_files" -ne "$expected_files" ]; then
+ skip_preview "GitHub returned an incomplete changed-file list; refusing to publish a preview."
+ fi
+
+ echo "Changed files:"
+ printf '%s' "$files" | jq -r '.[][] | .filename'
+
+ if printf '%s' "$files" | jq -e \
+ 'any(.[][]; .filename | test("^(Dockerfile([.].*)?|[.]dockerignore|[.]github/)"))' \
+ >/dev/null; then
+ skip_preview "PR modifies the build/CI surface; auto-preview skipped pending manual maintainer build."
+ fi
+
+ echo "skip=false" >> "$GITHUB_OUTPUT"
+ echo "skip_reason=" >> "$GITHUB_OUTPUT"
+ if printf '%s' "$files" | jq -e \
+ 'any(.[][]; .filename | test("^contrib/mysql-iam-plugin/"))' \
+ >/dev/null; then
+ echo "plugin_changed=true" >> "$GITHUB_OUTPUT"
else
- echo "skip=false" >> "$GITHUB_OUTPUT"
+ echo "plugin_changed=false" >> "$GITHUB_OUTPUT"
fi
- name: Comment that auto-preview was skipped
@@ -48,17 +89,15 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
issue-number: ${{ github.event.pull_request.number }}
body: |
- Auto-preview build skipped: this PR modifies the build/CI surface
- (`Dockerfile`, `.dockerignore`, or files under `.github/`). A maintainer
- will review and build the preview image manually before it is published.
+ Auto-preview build skipped: ${{ steps.guard.outputs.skip_reason }}
# checkout@v6 now refuses fork-PR checkouts from pull_request_target
# unless explicitly allowed, which has failed this job for every fork PR
# since 2026-07-20. Checking out the PR head is intentional here: the
# guard above keeps PRs that touch the build/CI surface from reaching
- # this step, and the build passes no secrets (only MINISTACK_VERSION as
- # a build-arg). persist-credentials is disabled so the untrusted tree
- # never holds the repo token — no later step needs git credentials.
+ # this step, and the builds receive only non-secret version and image
+ # reference arguments. persist-credentials is disabled so the untrusted
+ # tree never holds the repo token — no later step needs git credentials.
- uses: actions/checkout@v6
if: steps.guard.outputs.skip != 'true'
with:
@@ -93,6 +132,56 @@ jobs:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
+ # ── Full (Debian/glibc) image — DuckDB-backed Athena, psycopg2, pymysql.
+ # Built automatically for plugin-source changes so the regular preview can
+ # consume artifacts from this exact PR head. The `full-preview` label remains
+ # the opt-in path for other DuckDB/Postgres/MySQL end-to-end changes.
+ - name: Extract metadata (full)
+ id: meta_full
+ if: >-
+ steps.guard.outputs.skip != 'true' &&
+ (steps.guard.outputs.plugin_changed == 'true' ||
+ contains(github.event.pull_request.labels.*.name, 'full-preview'))
+ uses: docker/metadata-action@v6
+ with:
+ images: |
+ ${{ env.DOCKERHUB_IMAGE_ORG }}
+ flavor: |
+ latest=false
+ tags: |
+ type=ref,event=pr,suffix=-${{ steps.short_sha.outputs.short_sha }}-full
+
+ - name: Build and push (full)
+ id: build_full
+ if: >-
+ steps.guard.outputs.skip != 'true' &&
+ (steps.guard.outputs.plugin_changed == 'true' ||
+ contains(github.event.pull_request.labels.*.name, 'full-preview'))
+ uses: docker/build-push-action@v7
+ with:
+ context: .
+ file: Dockerfile.full
+ platforms: linux/amd64,linux/arm64
+ push: true
+ tags: ${{ steps.meta_full.outputs.tags }}
+ labels: ${{ steps.meta_full.outputs.labels }}
+ build-args: |
+ MINISTACK_VERSION=pr-${{ github.event.pull_request.number }}-${{ steps.short_sha.outputs.short_sha }}
+ # Read-only cache on pull_request_target (see note on the regular build above).
+ cache-from: type=gha,scope=full-pr
+
+ - name: Resolve plugin donor
+ id: plugin_donor
+ if: steps.guard.outputs.skip != 'true'
+ env:
+ FULL_DIGEST: ${{ steps.build_full.outputs.digest }}
+ run: |
+ if [ -n "$FULL_DIGEST" ]; then
+ echo "image=${DOCKERHUB_IMAGE_ORG}@${FULL_DIGEST}" >> "$GITHUB_OUTPUT"
+ else
+ echo "image=ghcr.io/ministackorg/ministack:full" >> "$GITHUB_OUTPUT"
+ fi
+
# ── Regular (Alpine) image ──
- name: Extract metadata (regular)
id: meta
@@ -105,6 +194,7 @@ jobs:
type=ref,event=pr,suffix=-${{ steps.short_sha.outputs.short_sha }}
- name: Build and push (regular)
+ id: build_regular
if: steps.guard.outputs.skip != 'true'
uses: docker/build-push-action@v7
with:
@@ -116,42 +206,54 @@ jobs:
labels: ${{ steps.meta.outputs.labels }}
build-args: |
MINISTACK_VERSION=pr-${{ github.event.pull_request.number }}-${{ steps.short_sha.outputs.short_sha }}
+ PLUGIN_DONOR_IMAGE=${{ steps.plugin_donor.outputs.image }}
# pull_request_target receives a read-only Actions cache as of GitHub's
# 2026-06-26 change (read-only cache for untrusted triggers), so a cache-to
# write fails the build with "failed to reserve cache". Restore only here;
# the cache is populated by a trusted (push) workflow.
cache-from: type=gha,scope=regular-pr
- # ── Full (Debian/glibc) image — DuckDB-backed Athena, psycopg2, pymysql.
- # Only built when the PR carries the `full-preview` label (opt-in to keep
- # default PR cost down). Add the label to any PR that touches the full
- # variant or needs DuckDB/Postgres/MySQL paths verified end-to-end.
- - name: Extract metadata (full)
- id: meta_full
- if: steps.guard.outputs.skip != 'true' && contains(github.event.pull_request.labels.*.name, 'full-preview')
- uses: docker/metadata-action@v6
- with:
- images: |
- ${{ env.DOCKERHUB_IMAGE_ORG }}
- flavor: |
- latest=false
- tags: |
- type=ref,event=pr,suffix=-${{ steps.short_sha.outputs.short_sha }}-full
+ - name: Verify full and regular plugin artifacts match
+ if: >-
+ steps.guard.outputs.skip != 'true' &&
+ steps.build_full.outputs.digest != '' &&
+ steps.build_regular.outputs.digest != ''
+ env:
+ FULL_IMAGE: ${{ env.DOCKERHUB_IMAGE_ORG }}@${{ steps.build_full.outputs.digest }}
+ REGULAR_IMAGE: ${{ env.DOCKERHUB_IMAGE_ORG }}@${{ steps.build_regular.outputs.digest }}
+ run: |
+ set -eu
+ scratch=$(mktemp -d)
+ full_container=
+ regular_container=
+ cleanup() {
+ [ -z "$full_container" ] || docker rm "$full_container" >/dev/null 2>&1 || true
+ [ -z "$regular_container" ] || docker rm "$regular_container" >/dev/null 2>&1 || true
+ rm -rf "$scratch"
+ }
+ trap cleanup EXIT
- - name: Build and push (full)
- if: steps.guard.outputs.skip != 'true' && contains(github.event.pull_request.labels.*.name, 'full-preview')
- uses: docker/build-push-action@v7
- with:
- context: .
- file: Dockerfile.full
- platforms: linux/amd64,linux/arm64
- push: true
- tags: ${{ steps.meta_full.outputs.tags }}
- labels: ${{ steps.meta_full.outputs.labels }}
- build-args: |
- MINISTACK_VERSION=pr-${{ github.event.pull_request.number }}-${{ steps.short_sha.outputs.short_sha }}
- # Read-only cache on pull_request_target (see note on the regular build above).
- cache-from: type=gha,scope=full-pr
+ for platform in linux/amd64 linux/arm64; do
+ arch=${platform#linux/}
+ rm -rf "$scratch/full" "$scratch/regular"
+ mkdir -p "$scratch/full" "$scratch/regular"
+
+ docker pull --platform "$platform" "$FULL_IMAGE"
+ full_container=$(docker create --platform "$platform" "$FULL_IMAGE")
+ docker cp "$full_container:/opt/ministack/mysql-plugins/." "$scratch/full"
+
+ docker pull --platform "$platform" "$REGULAR_IMAGE"
+ regular_container=$(docker create --platform "$platform" "$REGULAR_IMAGE")
+ docker cp "$regular_container:/opt/ministack/mysql-plugins/." "$scratch/regular"
+
+ test -f "$scratch/full/8.0/$arch/aws_auth_plugin.so"
+ test -f "$scratch/full/8.4/$arch/aws_auth_plugin.so"
+ diff -qr "$scratch/full" "$scratch/regular"
+
+ docker rm "$full_container" "$regular_container" >/dev/null
+ full_container=
+ regular_container=
+ done
- name: Comment on PR with Docker image info
if: steps.guard.outputs.skip != 'true'
@@ -162,3 +264,4 @@ jobs:
body: |
Docker image for this PR has been published: `${{ steps.meta.outputs.tags }}`
${{ steps.meta_full.outputs.tags && format('Full variant: `{0}`', steps.meta_full.outputs.tags) || '' }}
+ Plugin donor used by the regular variant: `${{ steps.plugin_donor.outputs.image }}`
diff --git a/Dockerfile b/Dockerfile
index 889ec026..6838888e 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,6 +1,7 @@
# Non-release builds intentionally use the maintainer-requested fallback of the latest
# published full image. Release builds override this in docker-publish.yml with the
-# same-release full-image digest; plugin-source PRs use the full-preview label path.
+# same-release full-image digest; plugin-source PRs build and consume a head-matched
+# full-preview digest in docker-publish-on-pr.yml.
ARG PLUGIN_DONOR_IMAGE=ghcr.io/ministackorg/ministack:full
FROM python:3.13-alpine AS builder
diff --git a/README.md b/README.md
index 1e416656..aec77991 100644
--- a/README.md
+++ b/README.md
@@ -567,7 +567,7 @@ MiniStack expands AWS SAM transforms (`Transform: AWS::Serverless-2016-10-31`) i
| Service | Operations | Notes |
|---------|-----------|-------|
| **ECS** | CreateCluster, DeleteCluster, DescribeClusters, ListClusters, UpdateCluster, UpdateClusterSettings, PutClusterCapacityProviders, RegisterTaskDefinition, DeregisterTaskDefinition, DescribeTaskDefinition, ListTaskDefinitions, ListTaskDefinitionFamilies, DeleteTaskDefinitions, CreateService, DeleteService, DescribeServices, UpdateService, ListServices, ListServicesByNamespace, RunTask, StopTask, DescribeTasks, ListTasks, ExecuteCommand, UpdateTaskProtection, GetTaskProtection, CreateCapacityProvider, UpdateCapacityProvider, DeleteCapacityProvider, DescribeCapacityProviders, TagResource, UntagResource, ListTagsForResource, ListAccountSettings, PutAccountSetting, PutAccountSettingDefault, DeleteAccountSetting, PutAttributes, DeleteAttributes, ListAttributes, DescribeServiceDeployments, ListServiceDeployments, DescribeServiceRevisions, SubmitTaskStateChange, SubmitContainerStateChange, SubmitAttachmentStateChanges, DiscoverPollEndpoint | 47 actions; `RunTask` starts real Docker containers via Docker socket; full Terraform ECS coverage |
-| **RDS** | CreateDBInstance, DeleteDBInstance, DescribeDBInstances, ModifyDBInstance, StartDBInstance, StopDBInstance, RebootDBInstance, CreateDBInstanceReadReplica, RestoreDBInstanceFromDBSnapshot, CreateDBCluster, DeleteDBCluster, DescribeDBClusters, ModifyDBCluster, StartDBCluster, StopDBCluster, FailoverDBCluster, CreateDBSnapshot, DeleteDBSnapshot, DescribeDBSnapshots, CreateDBClusterSnapshot, DescribeDBClusterSnapshots, DeleteDBClusterSnapshot, CreateDBSubnetGroup, DeleteDBSubnetGroup, DescribeDBSubnetGroups, ModifyDBSubnetGroup, CreateDBParameterGroup, DeleteDBParameterGroup, DescribeDBParameterGroups, DescribeDBParameters, ModifyDBParameterGroup, CreateDBClusterParameterGroup, DescribeDBClusterParameterGroups, DeleteDBClusterParameterGroup, DescribeDBClusterParameters, ModifyDBClusterParameterGroup, CreateOptionGroup, DeleteOptionGroup, DescribeOptionGroups, DescribeOptionGroupOptions, ListTagsForResource, AddTagsToResource, RemoveTagsFromResource, DescribeDBEngineVersions, DescribeOrderableDBInstanceOptions, CreateGlobalCluster, DescribeGlobalClusters, DeleteGlobalCluster, RemoveFromGlobalCluster, ModifyGlobalCluster, CreateDBProxy, DescribeDBProxies, ModifyDBProxy, DeleteDBProxy, CreateDBProxyEndpoint, DescribeDBProxyEndpoints, ModifyDBProxyEndpoint, DeleteDBProxyEndpoint, DescribeDBProxyTargetGroups, ModifyDBProxyTargetGroup, RegisterDBProxyTargets, DeregisterDBProxyTargets, DescribeDBProxyTargets | `CreateDBInstance` spins up real Postgres/MySQL Docker container, returns actual `host:port` endpoint; a cluster's endpoint is its stable AWS-shaped name, registered as a Docker network alias on the backing container (on user-defined networks; the default bridge keeps the address form), so a stored endpoint keeps resolving after the container is replaced; Aurora MySQL 8.0/8.4 compatibility objects (`AWSAuthenticationPlugin`, RDS/Aurora stored procedures, predefined S3 roles) install automatically for supported MySQL images so Aurora provisioning workflows run — the plugin rejects authentication (non-enforcing); disable with `MINISTACK_MYSQL_IAM_AUTH=off`. RDS Proxy is control plane only — proxies, proxy endpoints, the default target group and registered targets are stored and returned with the AWS shapes, but nothing listens on the proxy endpoint hostname; clients keep connecting to the instance or cluster endpoint |
+| **RDS** | CreateDBInstance, DeleteDBInstance, DescribeDBInstances, ModifyDBInstance, StartDBInstance, StopDBInstance, RebootDBInstance, CreateDBInstanceReadReplica, RestoreDBInstanceFromDBSnapshot, CreateDBCluster, DeleteDBCluster, DescribeDBClusters, ModifyDBCluster, StartDBCluster, StopDBCluster, FailoverDBCluster, CreateDBSnapshot, DeleteDBSnapshot, DescribeDBSnapshots, CreateDBClusterSnapshot, DescribeDBClusterSnapshots, DeleteDBClusterSnapshot, CreateDBSubnetGroup, DeleteDBSubnetGroup, DescribeDBSubnetGroups, ModifyDBSubnetGroup, CreateDBParameterGroup, DeleteDBParameterGroup, DescribeDBParameterGroups, DescribeDBParameters, ModifyDBParameterGroup, CreateDBClusterParameterGroup, DescribeDBClusterParameterGroups, DeleteDBClusterParameterGroup, DescribeDBClusterParameters, ModifyDBClusterParameterGroup, CreateOptionGroup, DeleteOptionGroup, DescribeOptionGroups, DescribeOptionGroupOptions, ListTagsForResource, AddTagsToResource, RemoveTagsFromResource, DescribeDBEngineVersions, DescribeOrderableDBInstanceOptions, CreateGlobalCluster, DescribeGlobalClusters, DeleteGlobalCluster, RemoveFromGlobalCluster, ModifyGlobalCluster, CreateDBProxy, DescribeDBProxies, ModifyDBProxy, DeleteDBProxy, CreateDBProxyEndpoint, DescribeDBProxyEndpoints, ModifyDBProxyEndpoint, DeleteDBProxyEndpoint, DescribeDBProxyTargetGroups, ModifyDBProxyTargetGroup, RegisterDBProxyTargets, DeregisterDBProxyTargets, DescribeDBProxyTargets | `CreateDBInstance` spins up real Postgres/MySQL Docker container, returns actual `host:port` endpoint; a cluster's endpoint is its stable AWS-shaped name, registered as a Docker network alias on the backing container (on user-defined networks; the default bridge keeps the address form), so a stored endpoint keeps resolving after the container is replaced; Aurora MySQL 8.0/8.4 compatibility objects (`AWSAuthenticationPlugin`, RDS/Aurora stored procedures, predefined S3 roles) install automatically for supported MySQL images so Aurora provisioning workflows run — the plugin rejects authentication (non-enforcing). RDS Proxy is control plane only — proxies, proxy endpoints, the default target group and registered targets are stored and returned with the AWS shapes, but nothing listens on the proxy endpoint hostname; clients keep connecting to the instance or cluster endpoint |
| **Aurora DSQL** | CreateCluster, GetCluster, UpdateCluster, DeleteCluster, ListClusters, PutClusterPolicy, GetClusterPolicy, DeleteClusterPolicy, CreateStream, GetStream, ListStreams, DeleteStream, GetVpcEndpointServiceName, TagResource, UntagResource, ListTagsForResource | `CreateCluster` spins up a real `postgres:16-alpine` Docker container fronted by a validating PG wire-protocol proxy that **fails like real DSQL** — rejects unsupported SQL (`serial`, extensions, temp tables, ...), accepts `CREATE INDEX ASYNC` + `sys.jobs`; endpoint is `localhost:<port>`; any password accepted |
| **ElastiCache** | CreateCacheCluster, DeleteCacheCluster, DescribeCacheClusters, ModifyCacheCluster, RebootCacheCluster, CreateReplicationGroup, DeleteReplicationGroup, DescribeReplicationGroups, ModifyReplicationGroup, IncreaseReplicaCount, DecreaseReplicaCount, CreateCacheSubnetGroup, DescribeCacheSubnetGroups, ModifyCacheSubnetGroup, DeleteCacheSubnetGroup, CreateCacheParameterGroup, DescribeCacheParameterGroups, ModifyCacheParameterGroup, ResetCacheParameterGroup, DeleteCacheParameterGroup, DescribeCacheParameters, DescribeCacheEngineVersions, CreateUser, DescribeUsers, DeleteUser, ModifyUser, CreateUserGroup, DescribeUserGroups, DeleteUserGroup, ModifyUserGroup, ListTagsForResource, AddTagsToResource, RemoveTagsFromResource, CreateSnapshot, DeleteSnapshot, DescribeSnapshots, DescribeEvents | `CreateCacheCluster` spins up a real Redis, Valkey, or Memcached Docker container (`Engine=redis`/`valkey`/`memcached`) |
| **Glue** | CreateDatabase, DeleteDatabase, GetDatabase, GetDatabases, UpdateDatabase, CreateTable, DeleteTable, GetTable, GetTables, UpdateTable, BatchDeleteTable, CreatePartition, DeletePartition, GetPartition, GetPartitions, BatchCreatePartition, BatchGetPartition, CreatePartitionIndex, GetPartitionIndexes, GetColumnStatisticsForTable, UpdateColumnStatisticsForTable, DeleteColumnStatisticsForTable, GetColumnStatisticsForPartition, UpdateColumnStatisticsForPartition, DeleteColumnStatisticsForPartition, CreateConnection, DeleteConnection, GetConnection, GetConnections, CreateCrawler, DeleteCrawler, GetCrawler, GetCrawlers, UpdateCrawler, StartCrawler, StopCrawler, GetCrawlerMetrics, CreateJob, DeleteJob, GetJob, GetJobs, UpdateJob, StartJobRun, GetJobRun, GetJobRuns, BatchStopJobRun, CreateTrigger, GetTrigger, DeleteTrigger, UpdateTrigger, StartTrigger, StopTrigger, ListTriggers, BatchGetTriggers, GetTriggers, CreateWorkflow, GetWorkflow, DeleteWorkflow, UpdateWorkflow, StartWorkflowRun, CreateUserDefinedFunction, UpdateUserDefinedFunction, DeleteUserDefinedFunction, GetUserDefinedFunction, GetUserDefinedFunctions, CreateSecurityConfiguration, DeleteSecurityConfiguration, GetSecurityConfiguration, GetSecurityConfigurations, CreateClassifier, GetClassifier, GetClassifiers, DeleteClassifier, TagResource, UntagResource, GetTags | Python shell jobs actually execute via subprocess; Spark jobs run on the official `amazon/aws-glue-libs` PySpark image (`GlueVersion: 4.0` / `3.0`) on MiniStack's Docker network; Glue-backed Iceberg REST catalog at `/iceberg` (signed with the `glue` credential scope, prefix `catalogs/{catalog}`) — read path for tables whose `metadata_location` parameter points at a metadata.json on MiniStack S3, e.g. DuckDB `ATTACH ... (TYPE iceberg, ENDPOINT 'localhost:4566/iceberg', AUTHORIZATION_TYPE 'sigv4')` with `USE_SSL=1` |