Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.
Python awsaws-emulatoraws-localaws-sdkdevtoolsdockerdynamodbec2emulatorlambdalocalstacklocalstack-alternativeministackmock-awsopen-sourcepythons3sqsterraform
fix(elasticache): serverless caches run in cluster mode (#1982)
Jeff Hellerac79e65aaa13
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7f9cdab6..e97573cf 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,7 @@ Versioning follows [Semantic Versioning](https://semver.org/).### Fixed+- **ElastiCache — serverless caches run in cluster mode** — AWS serves a serverless cache as a cluster-mode cache, so a multi-key command whose keys hash to different slots fails with `CROSSSLOT`; MiniStack's container ran without cluster mode and accepted it, so such code passed locally and failed on AWS. The container now runs as one cluster-mode shard holding every slot, and `CLUSTER SLOTS` reports the cache's endpoint, as AWS's single virtual shard does. Contributed by @skialpine.- **CloudTrail — `PutEventSelectors` stores `AdvancedEventSelectors` and sets `HasCustomEventSelectors`** — advanced selectors are now stored and returned in place of `EventSelectors` instead of being dropped, a request with both kinds, neither, or an empty or oversized list fails with `InvalidEventSelectorsException`, basic selectors read back with their omitted members filled in, and `GetTrail` and `DescribeTrails` report `HasCustomEventSelectors: true` while a trail has selectors other than the default one. Contributed by @iot-rocket.- **CloudTrail — `GetEventSelectors` returns the default selector of a new trail** — a trail that never had `PutEventSelectors` answered an empty `EventSelectors` list instead of the default selector that logs all read and write management events. Contributed by @iot-rocket.- **CloudTrail — `GetEventSelectors` returns the default selector of a new trail** — a trail that never had `PutEventSelectors` answered an empty `EventSelectors` list instead of the default selector that logs all read and write management events. Contributed by @iot-rocket.diff --git a/README.md b/README.mdindex f23d8f23..3c03e4e0 100644--- a/README.md+++ b/README.md@@ -610,7 +610,7 @@ MiniStack expands AWS SAM transforms (`Transform: AWS::Serverless-2016-10-31`) i| **ECS** | CreateCluster, DeleteCluster, DescribeClusters, ListClusters, UpdateCluster, UpdateClusterSettings, PutClusterCapacityProviders, RegisterTaskDefinition, DeregisterTaskDefinition, DescribeTaskDefinition, ListTaskDefinitions, ListTaskDefinitionFamilies, DeleteTaskDefinitions, CreateService, DeleteService, DescribeServices, UpdateService, ListServices, ListServicesByNamespace, RunTask, StopTask, DescribeTasks, ListTasks, ExecuteCommand, UpdateTaskProtection, GetTaskProtection, CreateCapacityProvider, UpdateCapacityProvider, DeleteCapacityProvider, DescribeCapacityProviders, TagResource, UntagResource, ListTagsForResource, ListAccountSettings, PutAccountSetting, PutAccountSettingDefault, DeleteAccountSetting, PutAttributes, DeleteAttributes, ListAttributes, DescribeServiceDeployments, ListServiceDeployments, DescribeServiceRevisions, SubmitTaskStateChange, SubmitContainerStateChange, SubmitAttachmentStateChanges, DiscoverPollEndpoint | 47 actions; `RunTask` starts real Docker containers via Docker socket; full Terraform ECS coverage || **RDS** | CreateDBInstance, DeleteDBInstance, DescribeDBInstances, ModifyDBInstance, StartDBInstance, StopDBInstance, RebootDBInstance, CreateDBInstanceReadReplica, RestoreDBInstanceFromDBSnapshot, CreateDBCluster, DeleteDBCluster, DescribeDBClusters, ModifyDBCluster, StartDBCluster, StopDBCluster, FailoverDBCluster, SwitchoverGlobalCluster, FailoverGlobalCluster, CreateDBSnapshot, DeleteDBSnapshot, DescribeDBSnapshots, CreateDBClusterSnapshot, DescribeDBClusterSnapshots, DescribeDBClusterSnapshotAttributes, DeleteDBClusterSnapshot, CreateDBSubnetGroup, DeleteDBSubnetGroup, DescribeDBSubnetGroups, ModifyDBSubnetGroup, CreateDBParameterGroup, DeleteDBParameterGroup, DescribeDBParameterGroups, DescribeDBParameters, ModifyDBParameterGroup, CreateDBClusterParameterGroup, DescribeDBClusterParameterGroups, DeleteDBClusterParameterGroup, DescribeDBClusterParameters, ModifyDBClusterParameterGroup, CreateOptionGroup, DeleteOptionGroup, DescribeOptionGroups, DescribeOptionGroupOptions, ListTagsForResource, AddTagsToResource, RemoveTagsFromResource, DescribeDBEngineVersions, DescribeOrderableDBInstanceOptions, CreateGlobalCluster, DescribeGlobalClusters, DeleteGlobalCluster, RemoveFromGlobalCluster, ModifyGlobalCluster, CreateDBProxy, DescribeDBProxies, ModifyDBProxy, DeleteDBProxy, CreateDBProxyEndpoint, DescribeDBProxyEndpoints, ModifyDBProxyEndpoint, DeleteDBProxyEndpoint, DescribeDBProxyTargetGroups, ModifyDBProxyTargetGroup, RegisterDBProxyTargets, DeregisterDBProxyTargets, DescribeDBProxyTargets | `CreateDBInstance` spins up real Postgres/MySQL Docker container, returns actual `host:port` endpoint; a cluster's endpoint is its stable AWS-shaped name, registered as a Docker network alias on the backing container (on user-defined networks; the default bridge keeps the address form), so a stored endpoint keeps resolving after the container is replaced; Aurora MySQL 8.0/8.4 compatibility objects (`AWSAuthenticationPlugin`, RDS/Aurora stored procedures, predefined S3 roles) install automatically for supported MySQL images so Aurora provisioning workflows run. IAM database authentication on MySQL and Aurora MySQL 8.0/8.4 (not MariaDB): a user created `IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS'` logs in with an SDK-generated auth token over `mysql_clear_password` when the instance or cluster has `IAMDatabaseAuthenticationEnabled`; with `AUTH=true` the token and the `rds-db:connect` policy are verified too. Planned Aurora MySQL switchover (`SwitchoverGlobalCluster`, and `FailoverGlobalCluster` without data loss allowed) performs a coordinated no-data-loss writer exchange on provisioned two-member MySQL 8 global clusters; other configurations keep the metadata-only path. RDS Proxy is control plane only — proxies, proxy endpoints, the default target group and registered targets are stored and returned with the AWS shapes, but nothing listens on the proxy endpoint hostname; clients keep connecting to the instance or cluster endpoint || **Aurora DSQL** | CreateCluster, GetCluster, UpdateCluster, DeleteCluster, ListClusters, PutClusterPolicy, GetClusterPolicy, DeleteClusterPolicy, CreateStream, GetStream, ListStreams, DeleteStream, GetVpcEndpointServiceName, TagResource, UntagResource, ListTagsForResource | `CreateCluster` spins up a real `postgres:16-alpine` Docker container fronted by a validating PG wire-protocol proxy that **fails like real DSQL** — rejects unsupported SQL (`serial`, extensions, temp tables, ...), accepts `CREATE INDEX ASYNC` + `sys.jobs`; endpoint is `localhost:<port>`; any password accepted |-| **ElastiCache** | CreateCacheCluster, DeleteCacheCluster, DescribeCacheClusters, ModifyCacheCluster, RebootCacheCluster, CreateReplicationGroup, DeleteReplicationGroup, DescribeReplicationGroups, ModifyReplicationGroup, IncreaseReplicaCount, DecreaseReplicaCount, CreateCacheSubnetGroup, DescribeCacheSubnetGroups, ModifyCacheSubnetGroup, DeleteCacheSubnetGroup, CreateCacheParameterGroup, DescribeCacheParameterGroups, ModifyCacheParameterGroup, ResetCacheParameterGroup, DeleteCacheParameterGroup, DescribeCacheParameters, DescribeCacheEngineVersions, CreateUser, DescribeUsers, DeleteUser, ModifyUser, CreateUserGroup, DescribeUserGroups, DeleteUserGroup, ModifyUserGroup, ListTagsForResource, AddTagsToResource, RemoveTagsFromResource, CreateSnapshot, DeleteSnapshot, DescribeSnapshots, CreateServerlessCache, DescribeServerlessCaches, ModifyServerlessCache, DeleteServerlessCache, DescribeEvents | `CreateCacheCluster` spins up a real Redis, Valkey, or Memcached Docker container (`Engine=redis`/`valkey`/`memcached`); a serverless cache gets a TLS-only Valkey/Redis container whose certificate chains to the CA at `GET /_ministack/elasticache/ca.pem` |+| **ElastiCache** | CreateCacheCluster, DeleteCacheCluster, DescribeCacheClusters, ModifyCacheCluster, RebootCacheCluster, CreateReplicationGroup, DeleteReplicationGroup, DescribeReplicationGroups, ModifyReplicationGroup, IncreaseReplicaCount, DecreaseReplicaCount, CreateCacheSubnetGroup, DescribeCacheSubnetGroups, ModifyCacheSubnetGroup, DeleteCacheSubnetGroup, CreateCacheParameterGroup, DescribeCacheParameterGroups, ModifyCacheParameterGroup, ResetCacheParameterGroup, DeleteCacheParameterGroup, DescribeCacheParameters, DescribeCacheEngineVersions, CreateUser, DescribeUsers, DeleteUser, ModifyUser, CreateUserGroup, DescribeUserGroups, DeleteUserGroup, ModifyUserGroup, ListTagsForResource, AddTagsToResource, RemoveTagsFromResource, CreateSnapshot, DeleteSnapshot, DescribeSnapshots, CreateServerlessCache, DescribeServerlessCaches, ModifyServerlessCache, DeleteServerlessCache, DescribeEvents | `CreateCacheCluster` spins up a real Redis, Valkey, or Memcached Docker container (`Engine=redis`/`valkey`/`memcached`); a serverless cache gets a TLS-only, single-shard cluster-mode Valkey/Redis container whose certificate chains to the CA at `GET /_ministack/elasticache/ca.pem` || **Glue** | CreateDatabase, DeleteDatabase, GetDatabase, GetDatabases, UpdateDatabase, CreateTable, DeleteTable, GetTable, GetTables, UpdateTable, BatchDeleteTable, CreatePartition, DeletePartition, GetPartition, GetPartitions, BatchCreatePartition, BatchGetPartition, CreatePartitionIndex, GetPartitionIndexes, GetColumnStatisticsForTable, UpdateColumnStatisticsForTable, DeleteColumnStatisticsForTable, GetColumnStatisticsForPartition, UpdateColumnStatisticsForPartition, DeleteColumnStatisticsForPartition, CreateConnection, DeleteConnection, GetConnection, GetConnections, CreateCrawler, DeleteCrawler, GetCrawler, GetCrawlers, UpdateCrawler, StartCrawler, StopCrawler, GetCrawlerMetrics, CreateJob, DeleteJob, GetJob, GetJobs, UpdateJob, StartJobRun, GetJobRun, GetJobRuns, BatchStopJobRun, CreateTrigger, GetTrigger, DeleteTrigger, UpdateTrigger, StartTrigger, StopTrigger, ListTriggers, BatchGetTriggers, GetTriggers, CreateWorkflow, GetWorkflow, DeleteWorkflow, UpdateWorkflow, StartWorkflowRun, CreateUserDefinedFunction, UpdateUserDefinedFunction, DeleteUserDefinedFunction, GetUserDefinedFunction, GetUserDefinedFunctions, CreateSecurityConfiguration, DeleteSecurityConfiguration, GetSecurityConfiguration, GetSecurityConfigurations, CreateClassifier, GetClassifier, GetClassifiers, DeleteClassifier, TagResource, UntagResource, GetTags | Python shell jobs actually execute via subprocess; Spark jobs run on the official `amazon/aws-glue-libs` PySpark image (`GlueVersion: 4.0` / `3.0`) on MiniStack's Docker network; Glue-backed Iceberg REST catalog at `/iceberg` (signed with the `glue` credential scope, prefix `catalogs/{catalog}`) — read path for tables whose `metadata_location` parameter points at a metadata.json on MiniStack S3, e.g. DuckDB `ATTACH ... (TYPE iceberg, ENDPOINT 'localhost:4566/iceberg', AUTHORIZATION_TYPE 'sigv4')` with `USE_SSL=1` || **S3 Tables** | CreateTableBucket, ListTableBuckets, GetTableBucket, DeleteTableBucket, CreateNamespace, ListNamespaces, GetNamespace, DeleteNamespace, CreateTable, ListTables, GetTable, DeleteTable, GetTableMetadataLocation, UpdateTableMetadataLocation | Iceberg-format tables; embedded Iceberg REST catalog at `/iceberg` so Spark jobs (`spark.sql.catalog.*.type=rest`) can create / load / commit without an external catalog; data files in S3, metadata in memory || **Athena** | StartQueryExecution, GetQueryExecution, GetQueryResults, StopQueryExecution, ListQueryExecutions, BatchGetQueryExecution, CreateWorkGroup, DeleteWorkGroup, GetWorkGroup, ListWorkGroups, UpdateWorkGroup, CreateNamedQuery, DeleteNamedQuery, GetNamedQuery, ListNamedQueries, BatchGetNamedQuery, CreateDataCatalog, GetDataCatalog, ListDataCatalogs, DeleteDataCatalog, UpdateDataCatalog, CreatePreparedStatement, GetPreparedStatement, DeletePreparedStatement, ListPreparedStatements, GetDatabase, ListDatabases, GetTableMetadata, ListTableMetadata, TagResource, UntagResource, ListTagsForResource | Real SQL via **DuckDB** when installed (`pip install duckdb`), otherwise returns mock results; databases and tables come from the Glue Data Catalog, and `CREATE EXTERNAL TABLE` / `DROP TABLE` change it; result pagination; column type metadata |@@ -733,7 +733,7 @@ A Redis sidecar is also always available at `localhost:6379` when using Docker C`Engine="valkey"` works the same way — a real `valkey/valkey` container is spawned (versions 7.2, 8.0, 8.1), the default parameter group becomes `default.valkey8`, and the advertised endpoint speaks the same protocol (`redis.Redis(...)` connects unchanged).-Serverless caches are TLS-only, as on AWS. Trust MiniStack's ElastiCache CA and connect with TLS:+Serverless caches are TLS-only and run in cluster mode, as on AWS: the keys of a multi-key command must share a hash slot. Trust MiniStack's ElastiCache CA and connect with TLS:```pythonimport time, urllib.requestdiff --git a/ministack/services/elasticache.py b/ministack/services/elasticache.pyindex 8197e5a3..37b8d3ce 100644--- a/ministack/services/elasticache.py+++ b/ministack/services/elasticache.py@@ -23,7 +23,8 @@ Supports: CreateCacheCluster, DeleteCacheCluster, DescribeCacheClusters,When Docker is available, CreateCacheCluster spins up a real Redis/Valkey/Memcachedcontainer. Otherwise returns localhost:6379 (assumes Redis sidecar in docker-compose).A serverless cache gets its own Valkey/Redis container that, like AWS's, only-speaks TLS; its certificate chains to the CA served at+speaks TLS and runs in cluster mode as a single shard announcing the cache's+endpoint; its certificate chains to the CA served at``GET /_ministack/elasticache/ca.pem``."""@@ -659,12 +660,15 @@ def _bootstrap_redis_cluster(bootstrap_container, node_addrs, replicas_per_shardreturn False-def _wait_cluster_ok(container, timeout=15):- """Poll ``CLUSTER INFO`` until ``cluster_state:ok`` is reported."""+def _wait_cluster_ok(container, timeout=15, cli_args=()):+ """Poll ``CLUSTER INFO`` until ``cluster_state:ok`` is reported.++ ``cli_args`` are as for ``_wait_redis_ready``.+ """deadline = time.time() + timeoutwhile time.time() < deadline:try:- result = container.exec_run(["redis-cli", "-p", "6379", "CLUSTER", "INFO"])+ result = container.exec_run(["redis-cli", *cli_args, "-p", "6379", "CLUSTER", "INFO"])if result.exit_code == 0 and b"cluster_state:ok" in result.output:return Trueexcept Exception:@@ -2116,14 +2120,18 @@ _SERVERLESS_ENGINE_VERSIONS = {_SERVERLESS_DEFAULT_MAJOR = {"valkey": "8", "redis": "7"}_SERVERLESS_TLS_DIR = "/ministack-elasticache-tls"_SERVERLESS_TLS_CLI_ARGS = ("--tls", "--cacert", f"{_SERVERLESS_TLS_DIR}/ca.crt", "-h", "127.0.0.1")-# The certificate has to name the container's address, which is only known once-# it runs, so the server waits for the material; `ready` is the archive's last-# member. The image's entrypoint then drops to the uid that owns /data.+# The certificate and the announced endpoint name the container's address, which+# is only known once it runs, so the server waits for the material; `ready` is+# the archive's last member. AWS serves a serverless cache as one cluster-mode+# shard, so once the server answers, it takes every slot. The image's entrypoint+# then drops to the uid that owns /data._SERVERLESS_TLS_WRAPPER = (f"d={_SERVERLESS_TLS_DIR}; "'while [ ! -e "$d/ready" ]; do sleep 0.1; done; ''chown -R "$(stat -c %u:%g /data)" "$d"; '- 'exec docker-entrypoint.sh "$@"'+ f'(until redis-cli {" ".join(_SERVERLESS_TLS_CLI_ARGS)} PING >/dev/null 2>&1; do sleep 0.1; done; '+ f'redis-cli {" ".join(_SERVERLESS_TLS_CLI_ARGS)} CLUSTER ADDSLOTSRANGE 0 16383 >/dev/null) & '+ 'exec docker-entrypoint.sh "$@" $(cat "$d/announce")')_serverless_ca_lock = _threading.Lock()_serverless_ca = None@@ -2161,7 +2169,17 @@ def _is_ip(value):return False-def _serverless_tls_archive(ca_pem, cert_pem, key_pem):+def _serverless_announce_flags(address, port):+ """Server flags that make CLUSTER SLOTS name the cache endpoint. A hostname+ Redis refuses (it allows only letters, digits, '-' and '.') is left out, and+ the node then announces its own address."""+ flags = f"--cluster-announce-tls-port {port}"+ if all(c.isascii() and (c.isalnum() or c in "-.") for c in address):+ flags += f" --cluster-announce-hostname {address} --cluster-preferred-endpoint-type hostname"+ return flags+++def _serverless_tls_archive(ca_pem, cert_pem, key_pem, announce):root = _SERVERLESS_TLS_DIR.lstrip("/")archive = io.BytesIO()with tarfile.open(fileobj=archive, mode="w") as bundle:@@ -2170,7 +2188,8 @@ def _serverless_tls_archive(ca_pem, cert_pem, key_pem):entry.mode = 0o755bundle.addfile(entry)for filename, content, mode in (("ca.crt", ca_pem, 0o644), ("server.crt", cert_pem, 0o644),- ("server.key", key_pem, 0o600), ("ready", "", 0o644)):+ ("server.key", key_pem, 0o600), ("announce", announce, 0o644),+ ("ready", "", 0o644)):data = content.encode()entry = tarfile.TarInfo(f"{root}/{filename}")entry.mode = mode@@ -2202,6 +2221,7 @@ def _spawn_serverless_container(name, engine, full_version, host_port, labels, w"--tls-ca-cert-file", f"{_SERVERLESS_TLS_DIR}/ca.crt","--tls-auth-clients", "no","--protected-mode", "no",+ "--cluster-enabled", "yes", "--tls-cluster", "yes",],)if DOCKER_NETWORK:@@ -2221,10 +2241,11 @@ def _spawn_serverless_container(name, engine, full_version, host_port, labels, wca_pem, ca_key = _ensure_serverless_ca()cert_pem, key_pem, _public = sign_leaf_certificate(ca_pem, ca_key, common_name=dns_names[0], san_dns=dns_names, san_ips=ips)- if not container.put_archive("/", _serverless_tls_archive(ca_pem, cert_pem, key_pem)):+ announce = _serverless_announce_flags(address, port)+ if not container.put_archive("/", _serverless_tls_archive(ca_pem, cert_pem, key_pem, announce)):raise RuntimeError("Docker rejected the TLS material")- if wait and not _wait_redis_ready(container, timeout=30, cli_args=_SERVERLESS_TLS_CLI_ARGS):- raise RuntimeError(f"{engine} in {name} never answered a TLS PING")+ if wait and not _wait_cluster_ok(container, timeout=30, cli_args=_SERVERLESS_TLS_CLI_ARGS):+ raise RuntimeError(f"{engine} in {name} never reported cluster_state:ok over TLS")except Exception:_teardown_containers(docker_client, [container.id])raisediff --git a/tests/test_elasticache.py b/tests/test_elasticache.pyindex 12133153..f0dd92dd 100644--- a/tests/test_elasticache.py+++ b/tests/test_elasticache.py@@ -1631,6 +1631,24 @@ def test_serverless_cache_serves_tls_only(ec, tmp_path):_tls_command(endpoint, ca_file, "PING")[email protected]_plane+def test_serverless_cache_is_one_cluster_mode_shard(ec, tmp_path):+ """AWS serves a serverless cache in cluster mode: keys in different slots+ fail together with CROSSSLOT, and CLUSTER SLOTS names the cache endpoint."""+ ca_file = _serverless_ca_file(tmp_path)+ name = f"sls-slots-{_uid()}"+ ec.create_serverless_cache(ServerlessCacheName=name, Engine="valkey")+ try:+ endpoint = _wait_serverless(ec, name)["Endpoint"]+ assert _tls_command(endpoint, ca_file, "MGET {a}x {b}y").startswith(b"-CROSSSLOT")+ assert _tls_command(endpoint, ca_file, "MGET {a}x {a}y") == b"*2\r\n$-1\r\n$-1\r\n"+ address = endpoint["Address"]+ assert _tls_command(endpoint, ca_file, "CLUSTER SLOTS").startswith(+ f"*1\r\n*3\r\n:0\r\n:16383\r\n*4\r\n${len(address)}\r\n{address}\r\n:{endpoint['Port']}\r\n".encode())+ finally:+ ec.delete_serverless_cache(ServerlessCacheName=name)++@pytest.mark.data_planedef test_serverless_cache_engine_change_replaces_container_on_same_endpoint(ec, tmp_path):ca_file = _serverless_ca_file(tmp_path)@@ -2133,6 +2151,13 @@ def test_elasticache_cluster_create_time_present_and_parsed(ec):# Serverless caches — provisioning state, not liveness# ---------------------------------------------------------------------------+def test_serverless_announce_flags_skip_a_hostname_redis_refuses():+ assert elasticache._serverless_announce_flags("localhost", 26500) == (+ "--cluster-announce-tls-port 26500 --cluster-announce-hostname localhost "+ "--cluster-preferred-endpoint-type hostname")+ assert elasticache._serverless_announce_flags("ministack_app_1", 26500) == "--cluster-announce-tls-port 26500"++def _serverless_create_params(name, engine="valkey"):return {"ServerlessCacheName": [name], "Engine": [engine]}