Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.
Python awsaws-emulatoraws-localaws-sdkdevtoolsdockerdynamodbec2emulatorlambdalocalstacklocalstack-alternativeministackmock-awsopen-sourcepythons3sqsterraform
fix(lambda): sign the local Node executor's SDK stub calls (#2354)
iot-rocket35782d1ec718
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex de680a4d..625dd288 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -14,6 +14,7 @@ Versioning follows [Semantic Versioning](https://semver.org/).- **CloudFront — VPC origins** — `CreateVpcOrigin`, `GetVpcOrigin`, `UpdateVpcOrigin`, `DeleteVpcOrigin` and `ListVpcOrigins` replace the empty `ListVpcOrigins` stub, so Terraform's `aws_cloudfront_vpc_origin` and a distribution's `vpc_origin_config` apply. As on AWS, a create, update or delete answers `Deploying`; MiniStack then finishes it at once, so the next read shows `Deployed` with the next `ETag`, or 404 after a delete. Names are unique per account, endpoints are not, and an origin a distribution names can't be deleted or changed. Validation, its order, the `TLSv1`/`TLSv1.1`/`TLSv1.2` default, `IpAddressType`, the list paging (no `Marker` or `IsTruncated`) and the IAM resources follow AWS captures. The endpoint has to be a load balancer or instance that exists in MiniStack, and viewer requests are not routed to it. Contributed by @kharabasz.### Fixed+- **Lambda — the local Node executor's SDK stubs sign their calls** — the `@aws-sdk/client-lambda` stub and the JSON-RPC `@aws-sdk/client-*` stubs sent their requests without credentials, so a call landed in the default account whatever account the function belongs to, and under `AUTH=true` it ran as the account root. They now sign with the function's credentials, as the real SDK does: with `AUTH` unset a call lands in the function's own account, under `AUTH=true` it runs as the execution role. Contributed by @iot-rocket.- **IoT — `#` does not match the level above it** — a subscription or topic rule with a filter ending in `/#` (`a/c2/#`) also got messages published to the level above (`a/c2`), as the MQTT specification allows; AWS IoT Core delivers only `a/c2/` and deeper, and so does the broker now, for subscriptions and rules. A republish topic at that level is still refused, as on AWS. Contributed by @iot-rocket.- **EventBridge — archive KMS keys** — `CreateArchive` and `UpdateArchive` dropped `KmsKeyIdentifier`, so `DescribeArchive` never returned it and Terraform re-planned `kms_key_identifier` on every run; the key is now stored and returned, an update that omits it keeps it, and an empty identifier on update clears it; `ListArchives` entries carry only the `Archive` summary fields. Contributed by @kharabasz.- **WAF v2 — `CreateIPSet` and `CreateRuleGroup` refuse a duplicate name** — a second IP set or rule group with the name and scope of an existing one is now refused with `WAFDuplicateItemException` (`AWS WAF couldn’t perform the operation because some resource in your request is a duplicate of an existing one.`, also from `CreateWebACL` and `CreateRegexPatternSet`). Both stored the duplicate silently, so a client that retries a create after losing the response ended up with two. Contributed by @OS-kiranmalsetty.diff --git a/ministack/core/lambda_runtime.py b/ministack/core/lambda_runtime.pyindex 5eee9ddf..9efaf428 100644--- a/ministack/core/lambda_runtime.py+++ b/ministack/core/lambda_runtime.py@@ -370,6 +370,38 @@ fs.write = function(fd, ...args) {// a Lambda Layer with the actual SDK takes precedence; fall back to a stub// that routes through AWS_ENDPOINT_URL (Ministack).(function _installAwsSdkV3Stubs() {+ // SigV4-sign a stub request with the function's credentials, as the real SDK does.+ function _signedHeaders(service, method, ep, requestPath, body, headers) {+ const crypto = require("crypto");+ const accessKey = process.env.AWS_ACCESS_KEY_ID;+ const secretKey = process.env.AWS_SECRET_ACCESS_KEY;+ if (!accessKey || !secretKey) return headers;+ const region = process.env.AWS_REGION || process.env.AWS_DEFAULT_REGION || "us-east-1";+ const amzDate = new Date().toISOString().replace(/[-:]|\.\d{3}/g, "");+ const scope = amzDate.slice(0, 8) + "/" + region + "/" + service + "/aws4_request";+ const signed = Object.assign({}, headers, { host: ep.host, "x-amz-date": amzDate });+ if (process.env.AWS_SESSION_TOKEN) signed["x-amz-security-token"] = process.env.AWS_SESSION_TOKEN;+ const values = {};+ for (const name of Object.keys(signed)) values[name.toLowerCase()] = String(signed[name]).trim();+ const names = Object.keys(values).sort();+ const encode = (v) => encodeURIComponent(v)+ .replace(/[!'()*]/g, (c) => "%" + c.charCodeAt(0).toString(16).toUpperCase());+ const [rawPath, rawQuery] = requestPath.split("?");+ const query = new URLSearchParams(rawQuery || "");+ const canonicalQuery = [...query.entries()].map(([k, v]) => encode(k) + "=" + encode(v)).sort().join("&");+ const hash = (data) => crypto.createHash("sha256").update(data).digest("hex");+ const hmac = (key, data) => crypto.createHmac("sha256", key).update(data).digest();+ const canonical = [method, rawPath.split("/").map(encode).join("/"), canonicalQuery,+ names.map((n) => n + ":" + values[n] + "\n").join(""), names.join(";"), hash(body || "")].join("\n");+ let key = hmac("AWS4" + secretKey, amzDate.slice(0, 8));+ for (const part of [region, service, "aws4_request"]) key = hmac(key, part);+ const signature = crypto.createHmac("sha256", key)+ .update(["AWS4-HMAC-SHA256", amzDate, scope, hash(canonical)].join("\n")).digest("hex");+ signed["Authorization"] = "AWS4-HMAC-SHA256 Credential=" + accessKey + "/" + scope+ + ", SignedHeaders=" + names.join(";") + ", Signature=" + signature;+ return signed;+ }+// ── Lambda stub (REST-based, not JSON-RPC) ─────────────────────────────function _lambdaInvoke(params) {const ep = new URL(process.env.AWS_ENDPOINT_URL || "http://127.0.0.1:4566");@@ -387,14 +419,15 @@ fs.write = function(fd, ...args) {if (process.env.__DEPTH_ENV__) {headers["__DEPTH_HEADER__"] = process.env.__DEPTH_ENV__;}+ const requestPath = "/2015-03-31/functions/" + fn + "/invocations" + qs;return new Promise((resolve, reject) => {const req = (ep.protocol === "https:" ? https : http).request({hostname: ep.hostname,port: parseInt(ep.port || "4566", 10),method: "POST",- path: "/2015-03-31/functions/" + fn + "/invocations" + qs,- headers: headers,+ path: requestPath,+ headers: _signedHeaders("lambda", "POST", ep, requestPath, body, headers),},(res) => {const chunks = [];@@ -657,7 +690,7 @@ fs.write = function(fd, ...args) {"@aws-sdk/client-translate": "AWSShineFrontendService_20170701",};- function _jsonRpcRequest(targetPrefix, opName, params) {+ function _jsonRpcRequest(targetPrefix, opName, params, service) {const ep = new URL(process.env.AWS_ENDPOINT_URL || "http://127.0.0.1:4566");const body = JSON.stringify(params || {});return new Promise((resolve, reject) => {@@ -667,11 +700,11 @@ fs.write = function(fd, ...args) {port: parseInt(ep.port || "4566", 10),method: "POST",path: "/",- headers: {+ headers: _signedHeaders(service, "POST", ep, "/", body, {"Content-Type": "application/x-amz-json-1.1","X-Amz-Target": targetPrefix + "." + opName,"Content-Length": Buffer.byteLength(body),- },+ }),},(res) => {const chunks = [];@@ -700,12 +733,21 @@ fs.write = function(fd, ...args) {});}- function _makeGenericJsonServiceModule(targetPrefix) {+ // Signing names that differ from the package name.+ const _SIGNING_NAMES = {+ "sfn": "states", "cloudwatch-logs": "logs", "secrets-manager": "secretsmanager", "eventbridge": "events",+ "dynamodb-streams": "dynamodb", "cognito-identity-provider": "cognito-idp", "emr": "elasticmapreduce",+ "resource-groups-tagging-api": "tagging",+ };++ function _makeGenericJsonServiceModule(targetPrefix, id) {+ const pkg = id.replace(/^@aws-sdk\/client-/, "");+ const service = _SIGNING_NAMES[pkg] || pkg;// Command class factory: new PutParameterCommand(params) → has _run()function _cmdClass(opName) {return class {constructor(params) { this._params = params; }- _run() { return _jsonRpcRequest(targetPrefix, opName, this._params); }+ _run() { return _jsonRpcRequest(targetPrefix, opName, this._params, service); }};}@@ -722,7 +764,7 @@ fs.write = function(fd, ...args) {get(_, prop) {if (typeof prop !== "string") return undefined;const opName = prop[0].toUpperCase() + prop.slice(1);- return (params) => _jsonRpcRequest(targetPrefix, opName, params);+ return (params) => _jsonRpcRequest(targetPrefix, opName, params, service);},});},@@ -772,7 +814,7 @@ fs.write = function(fd, ...args) {requireError = err;}const prefix = _JSON_RPC_TARGETS[id];- if (prefix) return _makeGenericJsonServiceModule(prefix);+ if (prefix) return _makeGenericJsonServiceModule(prefix, id);if (requireError.code !== "MODULE_NOT_FOUND"|| !requireError.message.includes("'" + id + "'")diff --git a/tests/test_lambda.py b/tests/test_lambda.pyindex bf0498c1..6a9a9b5d 100644--- a/tests/test_lambda.py+++ b/tests/test_lambda.py@@ -8861,6 +8861,108 @@ def test_nodejs_worker_aws_sdk_v3_stub_wire_roundtrip(lam, ssm):pass[email protected](os.environ.get("LAMBDA_EXECUTOR", "").lower() == "docker",+ reason="the SDK stubs belong to the local Node executor")+def test_nodejs_worker_aws_sdk_v3_stub_calls_the_functions_own_account():+ """A stub call signs with the function's credentials, so it lands in the function's account."""+ from conftest import make_client++ if not shutil.which("node"):+ pytest.skip("node not found on PATH")+ account = "123456789012"+ own = {name: boto3.client(name, endpoint_url=ENDPOINT, region_name="us-east-1", aws_access_key_id=account,+ aws_secret_access_key="test") for name in ("lambda", "ssm", "iam")}+ fname = f"sdk-account-{_uuid_mod.uuid4().hex[:8]}"+ param_name = f"/ministack-test/{fname}"+ code = (+ "const { SSMClient, PutParameterCommand } = require('@aws-sdk/client-ssm');\n"+ "exports.handler = async (event) => {\n"+ " await new SSMClient({}).send(new PutParameterCommand({\n"+ " Name: event.name, Value: 'v', Type: 'String', Overwrite: true }));\n"+ " return { ok: true };\n"+ "};\n"+ )+ role = own["iam"].create_role(RoleName=fname, AssumeRolePolicyDocument=json.dumps({+ "Version": "2012-10-17", "Statement": [{+ "Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]}))["Role"]+ own["iam"].put_role_policy(RoleName=fname, PolicyName="ssm", PolicyDocument=json.dumps({+ "Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": "ssm:PutParameter", "Resource": "*"}]}))+ own["lambda"].create_function(FunctionName=fname, Runtime="nodejs20.x", Role=role["Arn"],+ Handler="index.handler", Code={"ZipFile": _make_zip_js(code, "index.js")})+ try:+ resp = own["lambda"].invoke(FunctionName=fname, Payload=json.dumps({"name": param_name}).encode())+ assert json.loads(resp["Payload"].read()) == {"ok": True}+ assert own["ssm"].get_parameter(Name=param_name)["Parameter"]["Value"] == "v"+ with pytest.raises(ClientError):+ make_client("ssm").get_parameter(Name=param_name)+ finally:+ own["lambda"].delete_function(FunctionName=fname)+ with contextlib.suppress(ClientError):+ own["ssm"].delete_parameter(Name=param_name)+ own["iam"].delete_role_policy(RoleName=fname, PolicyName="ssm")+ own["iam"].delete_role(RoleName=fname)++[email protected]_contract[email protected](os.environ.get("LAMBDA_EXECUTOR", "").lower() == "docker",+ reason="the SDK stubs belong to the local Node executor")+def test_nodejs_sdk_stub_calls_run_as_the_execution_role_contract(auth_enabled, lam):+ """The Node SDK stubs sign with the function's credentials, so the execution role's policy decides."""+ from conftest import make_client++ if not shutil.which("node"):+ pytest.skip("node not found on PATH")+ iam, suffix = make_client("iam"), _uuid_mod.uuid4().hex[:8]+ trust = json.dumps({"Version": "2012-10-17", "Statement": [{+ "Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]})+ roles = {}+ for kind in ("allowed", "bare"):+ roles[kind] = iam.create_role(RoleName=f"stub-{kind}-{suffix}", AssumeRolePolicyDocument=trust)["Role"]+ iam.put_role_policy(RoleName=roles["allowed"]["RoleName"], PolicyName="calls", PolicyDocument=json.dumps({+ "Version": "2012-10-17", "Statement": [+ {"Effect": "Allow", "Action": ["lambda:InvokeFunction", "ssm:PutParameter"], "Resource": "*"}]}))+ callee = f"stub-callee-{suffix}"+ code = (+ "const { LambdaClient, InvokeCommand } = require('@aws-sdk/client-lambda');\n"+ "const { SSMClient, PutParameterCommand } = require('@aws-sdk/client-ssm');\n"+ "exports.handler = async (event) => {\n"+ " const r = await new LambdaClient({}).send(\n"+ " new InvokeCommand({ FunctionName: event.target, Payload: '{}' }));\n"+ " let ssm = 'ok';\n"+ " try {\n"+ " await new SSMClient({}).send(new PutParameterCommand({\n"+ " Name: event.name, Value: 'v', Type: 'String', Overwrite: true }));\n"+ " } catch (e) { ssm = e.name; }\n"+ " return { status: r.StatusCode, ssm: ssm };\n"+ "};\n"+ )+ callers = {kind: f"stub-{kind}-{suffix}" for kind in roles}+ try:+ lam.create_function(FunctionName=callee, Runtime="python3.12", Role=roles["allowed"]["Arn"],+ Handler="index.handler",+ Code={"ZipFile": _make_zip("def handler(event, context):\n return 1\n")})+ answers = {}+ for kind, name in callers.items():+ lam.create_function(FunctionName=name, Runtime="nodejs20.x", Role=roles[kind]["Arn"],+ Handler="index.handler", Code={"ZipFile": _make_zip_js(code, "index.js")})+ resp = lam.invoke(FunctionName=name, Payload=json.dumps(+ {"target": callee, "name": f"/stub-roles/{suffix}/{kind}"}).encode())+ answers[kind] = json.loads(resp["Payload"].read())+ assert answers["allowed"] == {"status": 200, "ssm": "ok"}+ assert answers["bare"] == ({"status": 403, "ssm": "AccessDeniedException"} if auth_enabled+ else {"status": 200, "ssm": "ok"})+ finally:+ for name in (callee, *callers.values()):+ with contextlib.suppress(Exception):+ lam.delete_function(FunctionName=name)+ for kind in callers:+ with contextlib.suppress(Exception):+ make_client("ssm").delete_parameter(Name=f"/stub-roles/{suffix}/{kind}")+ iam.delete_role_policy(RoleName=roles["allowed"]["RoleName"], PolicyName="calls")+ for role in roles.values():+ iam.delete_role(RoleName=role["RoleName"])++def test_lambda_ruby_4_0_runtime_maps_to_official_image():"""Lambda Ruby 4.0 runtime support (botocore 1.42.94 added the runtime).Maps to AWS's official Lambda Ruby 4.0 base image."""