Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

fix(elasticache): serverless caches run in cluster mode (#1982)

Jeff Hellerac79e65aaa13
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7f9cdab6..e97573cf 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,7 @@ Versioning follows [Semantic Versioning](https://semver.org/).
### Fixed
+- **ElastiCache — serverless caches run in cluster mode** — AWS serves a serverless cache as a cluster-mode cache, so a multi-key command whose keys hash to different slots fails with `CROSSSLOT`; MiniStack's container ran without cluster mode and accepted it, so such code passed locally and failed on AWS. The container now runs as one cluster-mode shard holding every slot, and `CLUSTER SLOTS` reports the cache's endpoint, as AWS's single virtual shard does. Contributed by @skialpine.
- **CloudTrail — `PutEventSelectors` stores `AdvancedEventSelectors` and sets `HasCustomEventSelectors`** — advanced selectors are now stored and returned in place of `EventSelectors` instead of being dropped, a request with both kinds, neither, or an empty or oversized list fails with `InvalidEventSelectorsException`, basic selectors read back with their omitted members filled in, and `GetTrail` and `DescribeTrails` report `HasCustomEventSelectors: true` while a trail has selectors other than the default one. Contributed by @iot-rocket.
- **CloudTrail — `GetEventSelectors` returns the default selector of a new trail** — a trail that never had `PutEventSelectors` answered an empty `EventSelectors` list instead of the default selector that logs all read and write management events. Contributed by @iot-rocket.
- **CloudTrail — `GetEventSelectors` returns the default selector of a new trail** — a trail that never had `PutEventSelectors` answered an empty `EventSelectors` list instead of the default selector that logs all read and write management events. Contributed by @iot-rocket.
diff --git a/README.md b/README.md
index f23d8f23..3c03e4e0 100644
--- a/README.md
+++ b/README.md
@@ -610,7 +610,7 @@ MiniStack expands AWS SAM transforms (`Transform: AWS::Serverless-2016-10-31`) i
| **ECS** | CreateCluster, DeleteCluster, DescribeClusters, ListClusters, UpdateCluster, UpdateClusterSettings, PutClusterCapacityProviders, RegisterTaskDefinition, DeregisterTaskDefinition, DescribeTaskDefinition, ListTaskDefinitions, ListTaskDefinitionFamilies, DeleteTaskDefinitions, CreateService, DeleteService, DescribeServices, UpdateService, ListServices, ListServicesByNamespace, RunTask, StopTask, DescribeTasks, ListTasks, ExecuteCommand, UpdateTaskProtection, GetTaskProtection, CreateCapacityProvider, UpdateCapacityProvider, DeleteCapacityProvider, DescribeCapacityProviders, TagResource, UntagResource, ListTagsForResource, ListAccountSettings, PutAccountSetting, PutAccountSettingDefault, DeleteAccountSetting, PutAttributes, DeleteAttributes, ListAttributes, DescribeServiceDeployments, ListServiceDeployments, DescribeServiceRevisions, SubmitTaskStateChange, SubmitContainerStateChange, SubmitAttachmentStateChanges, DiscoverPollEndpoint | 47 actions; `RunTask` starts real Docker containers via Docker socket; full Terraform ECS coverage |
| **RDS** | CreateDBInstance, DeleteDBInstance, DescribeDBInstances, ModifyDBInstance, StartDBInstance, StopDBInstance, RebootDBInstance, CreateDBInstanceReadReplica, RestoreDBInstanceFromDBSnapshot, CreateDBCluster, DeleteDBCluster, DescribeDBClusters, ModifyDBCluster, StartDBCluster, StopDBCluster, FailoverDBCluster, SwitchoverGlobalCluster, FailoverGlobalCluster, CreateDBSnapshot, DeleteDBSnapshot, DescribeDBSnapshots, CreateDBClusterSnapshot, DescribeDBClusterSnapshots, DescribeDBClusterSnapshotAttributes, DeleteDBClusterSnapshot, CreateDBSubnetGroup, DeleteDBSubnetGroup, DescribeDBSubnetGroups, ModifyDBSubnetGroup, CreateDBParameterGroup, DeleteDBParameterGroup, DescribeDBParameterGroups, DescribeDBParameters, ModifyDBParameterGroup, CreateDBClusterParameterGroup, DescribeDBClusterParameterGroups, DeleteDBClusterParameterGroup, DescribeDBClusterParameters, ModifyDBClusterParameterGroup, CreateOptionGroup, DeleteOptionGroup, DescribeOptionGroups, DescribeOptionGroupOptions, ListTagsForResource, AddTagsToResource, RemoveTagsFromResource, DescribeDBEngineVersions, DescribeOrderableDBInstanceOptions, CreateGlobalCluster, DescribeGlobalClusters, DeleteGlobalCluster, RemoveFromGlobalCluster, ModifyGlobalCluster, CreateDBProxy, DescribeDBProxies, ModifyDBProxy, DeleteDBProxy, CreateDBProxyEndpoint, DescribeDBProxyEndpoints, ModifyDBProxyEndpoint, DeleteDBProxyEndpoint, DescribeDBProxyTargetGroups, ModifyDBProxyTargetGroup, RegisterDBProxyTargets, DeregisterDBProxyTargets, DescribeDBProxyTargets | `CreateDBInstance` spins up real Postgres/MySQL Docker container, returns actual `host:port` endpoint; a cluster's endpoint is its stable AWS-shaped name, registered as a Docker network alias on the backing container (on user-defined networks; the default bridge keeps the address form), so a stored endpoint keeps resolving after the container is replaced; Aurora MySQL 8.0/8.4 compatibility objects (`AWSAuthenticationPlugin`, RDS/Aurora stored procedures, predefined S3 roles) install automatically for supported MySQL images so Aurora provisioning workflows run. IAM database authentication on MySQL and Aurora MySQL 8.0/8.4 (not MariaDB): a user created `IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS'` logs in with an SDK-generated auth token over `mysql_clear_password` when the instance or cluster has `IAMDatabaseAuthenticationEnabled`; with `AUTH=true` the token and the `rds-db:connect` policy are verified too. Planned Aurora MySQL switchover (`SwitchoverGlobalCluster`, and `FailoverGlobalCluster` without data loss allowed) performs a coordinated no-data-loss writer exchange on provisioned two-member MySQL 8 global clusters; other configurations keep the metadata-only path. RDS Proxy is control plane only — proxies, proxy endpoints, the default target group and registered targets are stored and returned with the AWS shapes, but nothing listens on the proxy endpoint hostname; clients keep connecting to the instance or cluster endpoint |
| **Aurora DSQL** | CreateCluster, GetCluster, UpdateCluster, DeleteCluster, ListClusters, PutClusterPolicy, GetClusterPolicy, DeleteClusterPolicy, CreateStream, GetStream, ListStreams, DeleteStream, GetVpcEndpointServiceName, TagResource, UntagResource, ListTagsForResource | `CreateCluster` spins up a real `postgres:16-alpine` Docker container fronted by a validating PG wire-protocol proxy that **fails like real DSQL** — rejects unsupported SQL (`serial`, extensions, temp tables, ...), accepts `CREATE INDEX ASYNC` + `sys.jobs`; endpoint is `localhost:<port>`; any password accepted |
-| **ElastiCache** | CreateCacheCluster, DeleteCacheCluster, DescribeCacheClusters, ModifyCacheCluster, RebootCacheCluster, CreateReplicationGroup, DeleteReplicationGroup, DescribeReplicationGroups, ModifyReplicationGroup, IncreaseReplicaCount, DecreaseReplicaCount, CreateCacheSubnetGroup, DescribeCacheSubnetGroups, ModifyCacheSubnetGroup, DeleteCacheSubnetGroup, CreateCacheParameterGroup, DescribeCacheParameterGroups, ModifyCacheParameterGroup, ResetCacheParameterGroup, DeleteCacheParameterGroup, DescribeCacheParameters, DescribeCacheEngineVersions, CreateUser, DescribeUsers, DeleteUser, ModifyUser, CreateUserGroup, DescribeUserGroups, DeleteUserGroup, ModifyUserGroup, ListTagsForResource, AddTagsToResource, RemoveTagsFromResource, CreateSnapshot, DeleteSnapshot, DescribeSnapshots, CreateServerlessCache, DescribeServerlessCaches, ModifyServerlessCache, DeleteServerlessCache, DescribeEvents | `CreateCacheCluster` spins up a real Redis, Valkey, or Memcached Docker container (`Engine=redis`/`valkey`/`memcached`); a serverless cache gets a TLS-only Valkey/Redis container whose certificate chains to the CA at `GET /_ministack/elasticache/ca.pem` |
+| **ElastiCache** | CreateCacheCluster, DeleteCacheCluster, DescribeCacheClusters, ModifyCacheCluster, RebootCacheCluster, CreateReplicationGroup, DeleteReplicationGroup, DescribeReplicationGroups, ModifyReplicationGroup, IncreaseReplicaCount, DecreaseReplicaCount, CreateCacheSubnetGroup, DescribeCacheSubnetGroups, ModifyCacheSubnetGroup, DeleteCacheSubnetGroup, CreateCacheParameterGroup, DescribeCacheParameterGroups, ModifyCacheParameterGroup, ResetCacheParameterGroup, DeleteCacheParameterGroup, DescribeCacheParameters, DescribeCacheEngineVersions, CreateUser, DescribeUsers, DeleteUser, ModifyUser, CreateUserGroup, DescribeUserGroups, DeleteUserGroup, ModifyUserGroup, ListTagsForResource, AddTagsToResource, RemoveTagsFromResource, CreateSnapshot, DeleteSnapshot, DescribeSnapshots, CreateServerlessCache, DescribeServerlessCaches, ModifyServerlessCache, DeleteServerlessCache, DescribeEvents | `CreateCacheCluster` spins up a real Redis, Valkey, or Memcached Docker container (`Engine=redis`/`valkey`/`memcached`); a serverless cache gets a TLS-only, single-shard cluster-mode Valkey/Redis container whose certificate chains to the CA at `GET /_ministack/elasticache/ca.pem` |
| **Glue** | CreateDatabase, DeleteDatabase, GetDatabase, GetDatabases, UpdateDatabase, CreateTable, DeleteTable, GetTable, GetTables, UpdateTable, BatchDeleteTable, CreatePartition, DeletePartition, GetPartition, GetPartitions, BatchCreatePartition, BatchGetPartition, CreatePartitionIndex, GetPartitionIndexes, GetColumnStatisticsForTable, UpdateColumnStatisticsForTable, DeleteColumnStatisticsForTable, GetColumnStatisticsForPartition, UpdateColumnStatisticsForPartition, DeleteColumnStatisticsForPartition, CreateConnection, DeleteConnection, GetConnection, GetConnections, CreateCrawler, DeleteCrawler, GetCrawler, GetCrawlers, UpdateCrawler, StartCrawler, StopCrawler, GetCrawlerMetrics, CreateJob, DeleteJob, GetJob, GetJobs, UpdateJob, StartJobRun, GetJobRun, GetJobRuns, BatchStopJobRun, CreateTrigger, GetTrigger, DeleteTrigger, UpdateTrigger, StartTrigger, StopTrigger, ListTriggers, BatchGetTriggers, GetTriggers, CreateWorkflow, GetWorkflow, DeleteWorkflow, UpdateWorkflow, StartWorkflowRun, CreateUserDefinedFunction, UpdateUserDefinedFunction, DeleteUserDefinedFunction, GetUserDefinedFunction, GetUserDefinedFunctions, CreateSecurityConfiguration, DeleteSecurityConfiguration, GetSecurityConfiguration, GetSecurityConfigurations, CreateClassifier, GetClassifier, GetClassifiers, DeleteClassifier, TagResource, UntagResource, GetTags | Python shell jobs actually execute via subprocess; Spark jobs run on the official `amazon/aws-glue-libs` PySpark image (`GlueVersion: 4.0` / `3.0`) on MiniStack's Docker network; Glue-backed Iceberg REST catalog at `/iceberg` (signed with the `glue` credential scope, prefix `catalogs/{catalog}`) — read path for tables whose `metadata_location` parameter points at a metadata.json on MiniStack S3, e.g. DuckDB `ATTACH ... (TYPE iceberg, ENDPOINT 'localhost:4566/iceberg', AUTHORIZATION_TYPE 'sigv4')` with `USE_SSL=1` |
| **S3 Tables** | CreateTableBucket, ListTableBuckets, GetTableBucket, DeleteTableBucket, CreateNamespace, ListNamespaces, GetNamespace, DeleteNamespace, CreateTable, ListTables, GetTable, DeleteTable, GetTableMetadataLocation, UpdateTableMetadataLocation | Iceberg-format tables; embedded Iceberg REST catalog at `/iceberg` so Spark jobs (`spark.sql.catalog.*.type=rest`) can create / load / commit without an external catalog; data files in S3, metadata in memory |
| **Athena** | StartQueryExecution, GetQueryExecution, GetQueryResults, StopQueryExecution, ListQueryExecutions, BatchGetQueryExecution, CreateWorkGroup, DeleteWorkGroup, GetWorkGroup, ListWorkGroups, UpdateWorkGroup, CreateNamedQuery, DeleteNamedQuery, GetNamedQuery, ListNamedQueries, BatchGetNamedQuery, CreateDataCatalog, GetDataCatalog, ListDataCatalogs, DeleteDataCatalog, UpdateDataCatalog, CreatePreparedStatement, GetPreparedStatement, DeletePreparedStatement, ListPreparedStatements, GetDatabase, ListDatabases, GetTableMetadata, ListTableMetadata, TagResource, UntagResource, ListTagsForResource | Real SQL via **DuckDB** when installed (`pip install duckdb`), otherwise returns mock results; databases and tables come from the Glue Data Catalog, and `CREATE EXTERNAL TABLE` / `DROP TABLE` change it; result pagination; column type metadata |
@@ -733,7 +733,7 @@ A Redis sidecar is also always available at `localhost:6379` when using Docker C
`Engine="valkey"` works the same way — a real `valkey/valkey` container is spawned (versions 7.2, 8.0, 8.1), the default parameter group becomes `default.valkey8`, and the advertised endpoint speaks the same protocol (`redis.Redis(...)` connects unchanged).
-Serverless caches are TLS-only, as on AWS. Trust MiniStack's ElastiCache CA and connect with TLS:
+Serverless caches are TLS-only and run in cluster mode, as on AWS: the keys of a multi-key command must share a hash slot. Trust MiniStack's ElastiCache CA and connect with TLS:
```python
import time, urllib.request
diff --git a/ministack/services/elasticache.py b/ministack/services/elasticache.py
index 8197e5a3..37b8d3ce 100644
--- a/ministack/services/elasticache.py
+++ b/ministack/services/elasticache.py
@@ -23,7 +23,8 @@ Supports: CreateCacheCluster, DeleteCacheCluster, DescribeCacheClusters,
When Docker is available, CreateCacheCluster spins up a real Redis/Valkey/Memcached
container. Otherwise returns localhost:6379 (assumes Redis sidecar in docker-compose).
A serverless cache gets its own Valkey/Redis container that, like AWS's, only
-speaks TLS; its certificate chains to the CA served at
+speaks TLS and runs in cluster mode as a single shard announcing the cache's
+endpoint; its certificate chains to the CA served at
``GET /_ministack/elasticache/ca.pem``.
"""
@@ -659,12 +660,15 @@ def _bootstrap_redis_cluster(bootstrap_container, node_addrs, replicas_per_shard
return False
-def _wait_cluster_ok(container, timeout=15):
- """Poll ``CLUSTER INFO`` until ``cluster_state:ok`` is reported."""
+def _wait_cluster_ok(container, timeout=15, cli_args=()):
+ """Poll ``CLUSTER INFO`` until ``cluster_state:ok`` is reported.
+
+ ``cli_args`` are as for ``_wait_redis_ready``.
+ """
deadline = time.time() + timeout
while time.time() < deadline:
try:
- result = container.exec_run(["redis-cli", "-p", "6379", "CLUSTER", "INFO"])
+ result = container.exec_run(["redis-cli", *cli_args, "-p", "6379", "CLUSTER", "INFO"])
if result.exit_code == 0 and b"cluster_state:ok" in result.output:
return True
except Exception:
@@ -2116,14 +2120,18 @@ _SERVERLESS_ENGINE_VERSIONS = {
_SERVERLESS_DEFAULT_MAJOR = {"valkey": "8", "redis": "7"}
_SERVERLESS_TLS_DIR = "/ministack-elasticache-tls"
_SERVERLESS_TLS_CLI_ARGS = ("--tls", "--cacert", f"{_SERVERLESS_TLS_DIR}/ca.crt", "-h", "127.0.0.1")
-# The certificate has to name the container's address, which is only known once
-# it runs, so the server waits for the material; `ready` is the archive's last
-# member. The image's entrypoint then drops to the uid that owns /data.
+# The certificate and the announced endpoint name the container's address, which
+# is only known once it runs, so the server waits for the material; `ready` is
+# the archive's last member. AWS serves a serverless cache as one cluster-mode
+# shard, so once the server answers, it takes every slot. The image's entrypoint
+# then drops to the uid that owns /data.
_SERVERLESS_TLS_WRAPPER = (
f"d={_SERVERLESS_TLS_DIR}; "
'while [ ! -e "$d/ready" ]; do sleep 0.1; done; '
'chown -R "$(stat -c %u:%g /data)" "$d"; '
- 'exec docker-entrypoint.sh "$@"'
+ f'(until redis-cli {" ".join(_SERVERLESS_TLS_CLI_ARGS)} PING >/dev/null 2>&1; do sleep 0.1; done; '
+ f'redis-cli {" ".join(_SERVERLESS_TLS_CLI_ARGS)} CLUSTER ADDSLOTSRANGE 0 16383 >/dev/null) & '
+ 'exec docker-entrypoint.sh "$@" $(cat "$d/announce")'
)
_serverless_ca_lock = _threading.Lock()
_serverless_ca = None
@@ -2161,7 +2169,17 @@ def _is_ip(value):
return False
-def _serverless_tls_archive(ca_pem, cert_pem, key_pem):
+def _serverless_announce_flags(address, port):
+ """Server flags that make CLUSTER SLOTS name the cache endpoint. A hostname
+ Redis refuses (it allows only letters, digits, '-' and '.') is left out, and
+ the node then announces its own address."""
+ flags = f"--cluster-announce-tls-port {port}"
+ if all(c.isascii() and (c.isalnum() or c in "-.") for c in address):
+ flags += f" --cluster-announce-hostname {address} --cluster-preferred-endpoint-type hostname"
+ return flags
+
+
+def _serverless_tls_archive(ca_pem, cert_pem, key_pem, announce):
root = _SERVERLESS_TLS_DIR.lstrip("/")
archive = io.BytesIO()
with tarfile.open(fileobj=archive, mode="w") as bundle:
@@ -2170,7 +2188,8 @@ def _serverless_tls_archive(ca_pem, cert_pem, key_pem):
entry.mode = 0o755
bundle.addfile(entry)
for filename, content, mode in (("ca.crt", ca_pem, 0o644), ("server.crt", cert_pem, 0o644),
- ("server.key", key_pem, 0o600), ("ready", "", 0o644)):
+ ("server.key", key_pem, 0o600), ("announce", announce, 0o644),
+ ("ready", "", 0o644)):
data = content.encode()
entry = tarfile.TarInfo(f"{root}/{filename}")
entry.mode = mode
@@ -2202,6 +2221,7 @@ def _spawn_serverless_container(name, engine, full_version, host_port, labels, w
"--tls-ca-cert-file", f"{_SERVERLESS_TLS_DIR}/ca.crt",
"--tls-auth-clients", "no",
"--protected-mode", "no",
+ "--cluster-enabled", "yes", "--tls-cluster", "yes",
],
)
if DOCKER_NETWORK:
@@ -2221,10 +2241,11 @@ def _spawn_serverless_container(name, engine, full_version, host_port, labels, w
ca_pem, ca_key = _ensure_serverless_ca()
cert_pem, key_pem, _public = sign_leaf_certificate(
ca_pem, ca_key, common_name=dns_names[0], san_dns=dns_names, san_ips=ips)
- if not container.put_archive("/", _serverless_tls_archive(ca_pem, cert_pem, key_pem)):
+ announce = _serverless_announce_flags(address, port)
+ if not container.put_archive("/", _serverless_tls_archive(ca_pem, cert_pem, key_pem, announce)):
raise RuntimeError("Docker rejected the TLS material")
- if wait and not _wait_redis_ready(container, timeout=30, cli_args=_SERVERLESS_TLS_CLI_ARGS):
- raise RuntimeError(f"{engine} in {name} never answered a TLS PING")
+ if wait and not _wait_cluster_ok(container, timeout=30, cli_args=_SERVERLESS_TLS_CLI_ARGS):
+ raise RuntimeError(f"{engine} in {name} never reported cluster_state:ok over TLS")
except Exception:
_teardown_containers(docker_client, [container.id])
raise
diff --git a/tests/test_elasticache.py b/tests/test_elasticache.py
index 12133153..f0dd92dd 100644
--- a/tests/test_elasticache.py
+++ b/tests/test_elasticache.py
@@ -1631,6 +1631,24 @@ def test_serverless_cache_serves_tls_only(ec, tmp_path):
_tls_command(endpoint, ca_file, "PING")
+def test_serverless_cache_is_one_cluster_mode_shard(ec, tmp_path):
+ """AWS serves a serverless cache in cluster mode: keys in different slots
+ fail together with CROSSSLOT, and CLUSTER SLOTS names the cache endpoint."""
+ ca_file = _serverless_ca_file(tmp_path)
+ name = f"sls-slots-{_uid()}"
+ ec.create_serverless_cache(ServerlessCacheName=name, Engine="valkey")
+ try:
+ endpoint = _wait_serverless(ec, name)["Endpoint"]
+ assert _tls_command(endpoint, ca_file, "MGET {a}x {b}y").startswith(b"-CROSSSLOT")
+ assert _tls_command(endpoint, ca_file, "MGET {a}x {a}y") == b"*2\r\n$-1\r\n$-1\r\n"
+ address = endpoint["Address"]
+ assert _tls_command(endpoint, ca_file, "CLUSTER SLOTS").startswith(
+ f"*1\r\n*3\r\n:0\r\n:16383\r\n*4\r\n${len(address)}\r\n{address}\r\n:{endpoint['Port']}\r\n".encode())
+ finally:
+ ec.delete_serverless_cache(ServerlessCacheName=name)
+
+
@pytest.mark.data_plane
def test_serverless_cache_engine_change_replaces_container_on_same_endpoint(ec, tmp_path):
ca_file = _serverless_ca_file(tmp_path)
@@ -2133,6 +2151,13 @@ def test_elasticache_cluster_create_time_present_and_parsed(ec):
# Serverless caches — provisioning state, not liveness
# ---------------------------------------------------------------------------
+def test_serverless_announce_flags_skip_a_hostname_redis_refuses():
+ assert elasticache._serverless_announce_flags("localhost", 26500) == (
+ "--cluster-announce-tls-port 26500 --cluster-announce-hostname localhost "
+ "--cluster-preferred-endpoint-type hostname")
+ assert elasticache._serverless_announce_flags("ministack_app_1", 26500) == "--cluster-announce-tls-port 26500"
+
+
def _serverless_create_params(name, engine="valkey"):
return {"ServerlessCacheName": [name], "Engine": [engine]}