Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.
Python awsaws-emulatoraws-localaws-sdkdevtoolsdockerdynamodbec2emulatorlambdalocalstacklocalstack-alternativeministackmock-awsopen-sourcepythons3sqsterraform
ALB: data-plane forwarding to instance/ip targets (#1113)
Jason Campf519b9424bc7
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex fa8e3647..29c8d35d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,6 +8,7 @@ Versioning follows [Semantic Versioning](https://semver.org/).## [Unreleased]### Added+- **ALB — data-plane forwarding to `instance`/`ip` targets** — the ALB data plane previously routed live traffic to Lambda targets only; forwarding to an `instance` or `ip` target group returned `502 Target type not supported`. Requests are now proxied over HTTP to the registered target, using the target `Id` as the host (an IP address for `ip` target groups; for `instance` target groups the `Id` is resolved as a hostname, since an emulator has no EC2 metadata to resolve instance ids against). Mirrors real ALB behavior: hop-by-hop headers are stripped, `X-Forwarded-For`/`X-Forwarded-Proto`/`X-Forwarded-Port` and a per-request `X-Amzn-Trace-Id` are injected, the target's HTTP error responses pass through unchanged, and connection failures surface as `502 Bad Gateway`. Multiple registered targets are balanced per-request. Existing listener rule conditions and `forward`/`redirect`/`fixed-response` actions apply unchanged. Contributed by @jasondcamp.- **ElastiCache — Valkey engine support** — `Engine=valkey` previously fell into the memcached branch of the container spawner: a nonexistent `memcached:<version>-alpine` image was requested, the spawn failed, and the cluster silently fell back to the shared sidecar advertising memcached's port 11211. `CreateCacheCluster` and `CreateReplicationGroup` now spawn a real `valkey/valkey:<major.minor>-alpine` container (AWS-supported versions 7.2, 8.0, 8.1) on port 6379, the no-Docker fallback advertises the Redis sidecar port, the default `EngineVersion` for valkey is `8.0` (previously memcached's `1.6.17`), and `DescribeCacheEngineVersions` lists valkey versions with their `valkey7`/`valkey8` parameter-group families (the `default.valkey*` groups already existed). `ELASTICACHE_CLUSTER_MODE_REAL` replication groups accept valkey too — the official valkey image ships `redis-*` compatibility symlinks, so the cluster bootstrap (`redis-cli --cluster create`) works unchanged.- **IoT — device shadow operations** — `GetThingShadow`, `UpdateThingShadow`, and `DeleteThingShadow` on the `iot-data` endpoint (classic and named shadows via `?name=`) previously returned `InvalidRequestException: Unsupported iot-data path: .../shadow`. Shadows now store `desired`/`reported` state: updates deep-merge into the stored document (a `null` value removes the field), bump the version, and stamp per-field `metadata` timestamps; the `/accepted` response echoes only the sections sent, while `GetThingShadow` returns the full document including the computed `state.delta` (desired fields that differ from reported). A stale `version` in an update is rejected with a `409`, and reading or deleting a nonexistent shadow returns `ResourceNotFoundException`. Shadows are account-scoped and persisted with the rest of the IoT state.- **KMS — `GenerateDataKeyPair` and `GenerateDataKeyPairWithoutPlaintext`** — the emulator implemented `GenerateDataKey` and `GenerateDataKeyWithoutPlaintext` but no data key *pair* operation, so both returned `InvalidAction: Unknown action`. These provision an asymmetric data key pair wrapped under a symmetric CMK: `GenerateDataKeyPair` returns `PrivateKeyPlaintext`, `PrivateKeyCiphertextBlob` and `PublicKey`, while the `WithoutPlaintext` variant omits `PrivateKeyPlaintext` so the private key never reaches the caller in the clear — the caller persists the ciphertext plus public key and nothing else. Supported `KeyPairSpec` values follow the real AWS enum — `RSA_2048/3072/4096`, `ECC_NIST_P256/P384/P521`, `ECC_SECG_P256K1`, `ECC_NIST_EDWARDS25519` (`SM2` is excluded, as `CreateKey` does not implement it either). Note this is a superset of what `CreateKey` accepts today: `CreateKey` still lacks `RSA_3072`, a pre-existing gap these operations do not share. The wrapping CMK must be symmetric, `EncryptionContext` binds the wrapped private key as it does for `Encrypt`, and the blob decrypts back to a PKCS#8 DER private key that pairs with the returned public key. The wrapped private key uses the emulator's ciphertext layout, now extended with a per-blob random nonce (`key_id(36) + context_hash(32) + nonce(16) + data`) that is mixed into key derivation; `Decrypt` reads these blobs exactly as it reads `Encrypt`'s and `GenerateDataKey`'s. The nonce is required for correctness of the `WithoutPlaintext` guarantee: without it the XOR keystream repeats across every blob sharing a CMK and `EncryptionContext`, and because a key pair's plaintext embeds its (public) modulus, the wrapped private key could be recovered from the public outputs alone. `Encrypt` and `GenerateDataKey` adopt the same nonced layout so all three wrap through an identical format.diff --git a/ministack/services/alb.py b/ministack/services/alb.pyindex 09aae109..c7649e6b 100644--- a/ministack/services/alb.py+++ b/ministack/services/alb.py@@ -1157,6 +1157,11 @@ async def _forward_to_tg(tg_arn, method, path, headers, body, query_params):return await _invoke_lambda_target(func_id, tg_arn, method, path,headers, body, query_params)+ if target_type in ("instance", "ip"):+ target = random.choice(registered)+ return await _proxy_http_target(target, tg, method, path,+ headers, body, query_params)+return (502, {"Content-Type": "application/json"},json.dumps({"message": f"Target type '{target_type}' not supported."}).encode())@@ -1259,6 +1264,53 @@ async def _invoke_lambda_target(function_ref, tg_arn, method, path, headers, bodreturn 200, {"Content-Type": "text/plain"}, raw+async def _proxy_http_target(target, tg, method, path, headers, body, query_params):+ """Forward a data-plane request to an instance/ip target over HTTP.++ The target Id is used as the host to connect to — an IP address for+ ip target groups, or any resolvable hostname for instance target+ groups (an emulator has no EC2 metadata to resolve instance ids+ against, so a hostname is the useful interpretation). Mirrors ALB+ behavior: hop-by-hop headers are stripped, X-Forwarded-* and+ X-Amzn-Trace-Id are injected, target HTTP errors pass through, and+ connection failures surface as 502.+ """+ import urllib.error+ import urllib.request+ from urllib.parse import urlencode++ host = str(target.get("Id", ""))+ port = int(target.get("Port") or tg.get("Port") or 80)+ qs = {k: (v[0] if isinstance(v, list) else v) for k, v in (query_params or {}).items()}+ url = f"http://{host}:{port}{path}" + (f"?{urlencode(qs)}" if qs else "")++ hop_by_hop = {"host", "content-length", "connection", "transfer-encoding",+ "accept-encoding"}+ fwd = {k: v for k, v in (headers or {}).items() if k.lower() not in hop_by_hop}+ fwd["X-Forwarded-For"] = (headers or {}).get("x-forwarded-for") or "127.0.0.1"+ fwd.setdefault("X-Forwarded-Proto", "http")+ fwd.setdefault("X-Forwarded-Port", "80")+ fwd["X-Amzn-Trace-Id"] = "Root=1-%08x-%s" % (+ int(time.time()),+ "".join(random.choices("0123456789abcdef", k=24)),+ )++ data = body if isinstance(body, (bytes, bytearray)) else (body.encode() if body else None)++ def _do():+ req = urllib.request.Request(url, data=data, method=method.upper(), headers=fwd)+ try:+ with urllib.request.urlopen(req, timeout=15) as resp:+ return resp.status, dict(resp.headers), resp.read()+ except urllib.error.HTTPError as e:+ return e.code, dict(e.headers), e.read()+ except Exception as e:+ return (502, {"Content-Type": "application/json"},+ json.dumps({"message": f"Target {host}:{port} connect error: {e}"}).encode())++ return await asyncio.to_thread(_do)++# ---------------------------------------------------------------------------# Data-plane: main dispatcher# ---------------------------------------------------------------------------diff --git a/tests/test_alb.py b/tests/test_alb.pyindex 41dc7ffc..2099de6e 100644--- a/tests/test_alb.py+++ b/tests/test_alb.py@@ -786,3 +786,96 @@ def test_alb_set_security_groups(elbv2):)["LoadBalancers"][0]["LoadBalancerArn"]resp = elbv2.set_security_groups(LoadBalancerArn=arn, SecurityGroups=["sg-bbb", "sg-ccc"])assert resp["SecurityGroupIds"] == ["sg-bbb", "sg-ccc"]+++def _alb_http_target_setup(elbv2, lb_name, target_id, target_port, target_type="ip"):+ """Create LB + instance/ip TG + listener + register an HTTP target.+ Returns (lb_arn, tg_arn, l_arn).+ """+ lb_arn = elbv2.create_load_balancer(Name=lb_name)["LoadBalancers"][0]["LoadBalancerArn"]+ tg_arn = elbv2.create_target_group(+ Name=f"{lb_name}-tg",+ Protocol="HTTP",+ Port=target_port,+ VpcId="vpc-00000001",+ TargetType=target_type,+ )["TargetGroups"][0]["TargetGroupArn"]+ elbv2.register_targets(TargetGroupArn=tg_arn, Targets=[{"Id": target_id, "Port": target_port}])+ l_arn = elbv2.create_listener(+ LoadBalancerArn=lb_arn,+ Protocol="HTTP",+ Port=80,+ DefaultActions=[{"Type": "forward", "TargetGroupArn": tg_arn}],+ )["Listeners"][0]["ListenerArn"]+ return lb_arn, tg_arn, l_arn+++def _alb_http_target_teardown(elbv2, lb_arn, tg_arn, l_arn):+ for fn, kwargs in (+ (elbv2.delete_listener, {"ListenerArn": l_arn}),+ (elbv2.delete_target_group, {"TargetGroupArn": tg_arn}),+ (elbv2.delete_load_balancer, {"LoadBalancerArn": lb_arn}),+ ):+ try:+ fn(**kwargs)+ except Exception:+ pass+++def test_elbv2_dataplane_forward_ip_target(elbv2):+ """ALB data plane proxies instance/ip targets over HTTP.++ The emulator's own health endpoint (127.0.0.1:4566 from inside the+ server process) doubles as the backend, so the test needs no external+ HTTP server and works both in-container and locally.+ """+ import urllib.request as _req++ lb_name = "dp-alb-ip"+ lb_arn, tg_arn, l_arn = _alb_http_target_setup(elbv2, lb_name, "127.0.0.1", 4566)+ try:+ url = f"{_endpoint}/_alb/{lb_name}/_ministack/health"+ resp = _req.urlopen(_req.Request(url, method="GET"))+ assert resp.status == 200+ body = json.loads(resp.read())+ assert "services" in body+ finally:+ _alb_http_target_teardown(elbv2, lb_arn, tg_arn, l_arn)+++def test_elbv2_dataplane_forward_instance_target_hostname(elbv2):+ """Instance targets resolve the Id as a hostname (no EC2 metadata in an+ emulator), so localhost works as an instance Id too."""+ import urllib.request as _req++ lb_name = "dp-alb-inst"+ lb_arn, tg_arn, l_arn = _alb_http_target_setup(+ elbv2, lb_name, "localhost", 4566, target_type="instance"+ )+ try:+ url = f"{_endpoint}/_alb/{lb_name}/_ministack/health"+ resp = _req.urlopen(_req.Request(url, method="GET"))+ assert resp.status == 200+ finally:+ _alb_http_target_teardown(elbv2, lb_arn, tg_arn, l_arn)+++def test_elbv2_dataplane_ip_target_unreachable_returns_502(elbv2):+ """Connection failures surface as 502 Bad Gateway, like a real ALB."""+ import urllib.error as _err+ import urllib.request as _req++ lb_name = "dp-alb-dead"+ lb_arn, tg_arn, l_arn = _alb_http_target_setup(elbv2, lb_name, "127.0.0.1", 59999)+ try:+ url = f"{_endpoint}/_alb/{lb_name}/anything"+ with pytest.raises(_err.HTTPError) as exc_info:+ _req.urlopen(_req.Request(url, method="GET"))+ assert exc_info.value.code == 502+ body = json.loads(exc_info.value.read())+ assert "connect error" in body["message"]+ finally:+ _alb_http_target_teardown(elbv2, lb_arn, tg_arn, l_arn)