Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

fix(lambda): event source mappings answer as AWS does (#2442)

Luke Dutton6c385477a80f
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d14e9543..2108452d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -15,6 +15,8 @@ Versioning follows [Semantic Versioning](https://semver.org/).
- **Athena — the result file is saved at the reported `OutputLocation`** — a query's `<id>.csv` and `<id>.csv.metadata` are now saved directly under the result location, at the path that `GetQueryExecution` reports, as AWS does. Before, they were saved one folder too deep, at `<location>/<id>.csv/<id>.csv`, so a client that read the reported path got `NoSuchKey`.
- **Athena — an enforced workgroup overrides the client-side result location** — when a workgroup has `EnforceWorkGroupConfiguration: true`, `StartQueryExecution` now writes results to the workgroup's `OutputLocation`, even when the request gives its own `ResultConfiguration.OutputLocation`, as AWS does. Before, the request's location was used.
- **Lambda — a partial ARN as the `FunctionName`** — `123456789012:function:my-function[:qualifier]` resolved to the account ID as the function name, so `Invoke` and the other function operations failed with `Function not found: ...:function:123456789012`; it now resolves as the full ARN in the caller's region, as on AWS, and gives the same result as that full ARN.
+- **Lambda — event source mappings answer as AWS does** — a Kinesis or DynamoDB Streams mapping now reports AWS's defaults (`BatchSize` 100, `MaximumRetryAttempts` and `MaximumRecordAgeInSeconds` -1, `BisectBatchOnFunctionError` false, `ParallelizationFactor` 1, `TumblingWindowInSeconds` 0, `DestinationConfig` `{"OnFailure": {}}`), `StateTransitionReason` `User action`, and `LastProcessingResult` `OK` or `PROBLEM: Function call failed` instead of `OK - N records` and `FAILED`. An SQS mapping has no `LastProcessingResult`, as on AWS. Create, update and delete answer the transitional state (`Creating`, `Updating`, `Enabling`, `Disabling`, `Deleting`), `UpdateEventSourceMapping` answers 202, a second mapping for the same source and function is refused with `ResourceConflictException`, and an unknown mapping's message is AWS's. MiniStack's states settle at once, where AWS's SQS mappings stay `Creating` for a few seconds.
+
- **Glue — Glue 5 file writes to the bucket that holds `--extra-py-files`** — a CSV (or other plain-file) write to that bucket stayed an open multipart upload, so the job succeeded but the output folder had only a 0-byte `part-*` marker under `__magic_job-…/`. `spark-submit` downloads `s3://` `--extra-py-files` before Spark starts, and the S3A filesystem it makes for that bucket saved the magicv2 committer's upload list in S3, where the commit did not look. Every S3A filesystem now tracks these uploads in memory, so the commit completes them. Glue 4 jobs use `DirectFileOutputCommitter` and do not change.
- **Step Functions — state events inside a Map or a Parallel branch** — the states in an inline Map item or a Parallel branch wrote no `<Type>StateEntered` or `<Type>StateExited` events, so `GetExecutionHistory` did not show them. They are now in the execution history, as on AWS; for a Distributed Map they are in each child execution's history and not in the parent's. Events added at the same time from several Map items or branches can no longer get the same `id`.
## [1.5.33] — 2026-10-26
diff --git a/ministack/services/lambda_svc.py b/ministack/services/lambda_svc.py
index a05c6a90..065f40b7 100644
--- a/ministack/services/lambda_svc.py
+++ b/ministack/services/lambda_svc.py
@@ -7138,15 +7138,25 @@ def _delete_provisioned_concurrency(func_name: str, qualifier: str):
# ---------------------------------------------------------------------------
-def _esm_response(esm: dict) -> dict:
+def _esm_response(esm: dict, state: str | None = None, modified_now: bool = False) -> dict:
"""Return ESM dict without internal-only fields.
+ *state* is the transitional state a create, update or delete answers with
+ (``Creating``, ``Updating``, ``Enabling``, ``Disabling``, ``Deleting``); the
+ record keeps the settled one. AWS reports a stream mapping's LastModified in
+ whole seconds except in the answer to the call that changed it.
+
EventSourceMappingArn is derived here rather than stored on the record so
that mappings created before it existed — restored state included — report
it too. Callers cannot construct this ARN themselves, and ListTags is keyed
on it, so omitting it leaves tags unreadable.
"""
out = {k: v for k, v in esm.items() if k not in ("FunctionName", "Enabled")}
+ if state:
+ out["State"] = state
+ if (not modified_now and isinstance(out.get("LastModified"), float)
+ and not str(esm.get("EventSourceArn", "")).startswith("arn:aws:sqs:")):
+ out["LastModified"] = int(out["LastModified"])
esm_id = esm.get("UUID")
if esm_id:
out["EventSourceMappingArn"] = _esm_arn(esm_id)
@@ -7253,9 +7263,24 @@ def _create_esm(data: dict):
if err:
return err
+ is_sqs = event_source_spec.service == "sqs"
+ for other in _esms.values():
+ if other["EventSourceArn"] == event_source_arn and other["FunctionName"] == func_name:
+ message = (
+ f'An event source mapping with SQS arn (" {event_source_arn} ") and function (" {func_name} ") '
+ f'already exists. Please update or delete the existing mapping with UUID {other["UUID"]}'
+ if is_sqs else
+ f'The event source arn (" {event_source_arn} ") and function (" {func_name} ") provided mapping '
+ f'already exists. Please update or delete the existing mapping with UUID {other["UUID"]}'
+ )
+ return error_response_json("ResourceConflictException", message, 409)
+
enabled = data.get("Enabled", True)
if isinstance(enabled, str):
enabled = enabled.lower() != "false"
+ # Defaults and fields as AWS answers them: a stream mapping reports its
+ # retry, age, batching and destination settings; an SQS one does not, and
+ # has no LastProcessingResult.
esm = {
"UUID": esm_id,
"EventSourceArn": event_source_arn,
@@ -7263,20 +7288,29 @@ def _create_esm(data: dict):
"FunctionName": func_name,
"Qualifier": qualifier,
"State": "Enabled" if enabled else "Disabled",
- "StateTransitionReason": "USER_INITIATED",
- "BatchSize": data.get("BatchSize", 10),
+ "StateTransitionReason": "USER_INITIATED" if is_sqs else "User action",
+ "BatchSize": data.get("BatchSize", 10 if is_sqs else 100),
"MaximumBatchingWindowInSeconds": data.get("MaximumBatchingWindowInSeconds", 0),
- "LastModified": int(time.time()),
- "LastProcessingResult": "No records processed",
+ "LastModified": time.time(),
"Enabled": enabled,
"FunctionResponseTypes": data.get("FunctionResponseTypes", []),
}
+ if not is_sqs:
+ esm.update({
+ "LastProcessingResult": "No records processed",
+ "MaximumRetryAttempts": -1,
+ "MaximumRecordAgeInSeconds": -1,
+ "BisectBatchOnFunctionError": False,
+ "ParallelizationFactor": 1,
+ "TumblingWindowInSeconds": 0,
+ "DestinationConfig": {"OnFailure": {}},
+ })
# Optional stream/poll settings — stored on create (previously only picked
# up on UpdateEventSourceMapping), so GetEventSourceMapping echoes them and
# the poller can honor MaximumRecordAgeInSeconds + the OnFailure destination.
for _opt in ("MaximumRetryAttempts", "MaximumRecordAgeInSeconds",
"BisectBatchOnFunctionError", "ParallelizationFactor",
- "DestinationConfig"):
+ "DestinationConfig", "TumblingWindowInSeconds"):
if _opt in data:
esm[_opt] = data[_opt]
if event_source_spec.service != "sqs":
@@ -7293,7 +7327,8 @@ def _create_esm(data: dict):
esm["Tags"] = dict(tags)
_esms[esm_id] = esm
_ensure_poller()
- return json_response(_esm_response(esm), 202)
+ creating = "Creating" if (enabled or not is_sqs) else "Disabled"
+ return json_response(_esm_response(esm, state=creating, modified_now=True), 202)
def _get_esm(esm_id: str):
@@ -7301,7 +7336,7 @@ def _get_esm(esm_id: str):
if not esm:
return error_response_json(
"ResourceNotFoundException",
- f"The resource you requested does not exist. (Service: Lambda, Status Code: 404, Request ID: {new_uuid()})",
+ "The resource you requested does not exist.",
404,
)
return json_response(_esm_response(esm))
@@ -7358,6 +7393,7 @@ def _update_esm(esm_id: str, data: dict):
):
if key in data:
esm[key] = data[key]
+ was_enabled = esm.get("State") == "Enabled"
if "Enabled" in data:
esm["Enabled"] = data["Enabled"]
esm["State"] = "Enabled" if data["Enabled"] else "Disabled"
@@ -7368,8 +7404,15 @@ def _update_esm(esm_id: str, data: dict):
esm["FunctionName"] = new_name
esm["Qualifier"] = qualifier
esm["FunctionArn"] = _func_arn(new_name) + (f":{qualifier}" if qualifier else "")
- esm["LastModified"] = int(time.time())
- return json_response(_esm_response(esm))
+ esm["LastModified"] = time.time()
+ now_enabled = esm.get("State") == "Enabled"
+ if now_enabled != was_enabled:
+ state = "Enabling" if now_enabled else "Disabling"
+ elif now_enabled or not str(esm.get("EventSourceArn", "")).startswith("arn:aws:sqs:"):
+ state = "Updating"
+ else:
+ state = "Disabled" # an SQS mapping that stays disabled
+ return json_response(_esm_response(esm, state=state, modified_now=True), 202)
def _delete_esm(esm_id: str):
@@ -7380,9 +7423,8 @@ def _delete_esm(esm_id: str):
f"Event source mapping not found: {esm_id}",
404,
)
- esm["State"] = "Deleting"
_release_esm_poll_state(esm_id)
- return json_response(_esm_response(esm), 202)
+ return json_response(_esm_response(esm, state="Deleting"), 202)
def _release_esm_poll_state(esm_id: str) -> None:
@@ -7663,7 +7705,6 @@ def _run_sqs_batch(esm, func_rec, event, batch, queue_url, queue_name):
err_body = result.get("body") or {}
err_type = err_body.get("errorType") if isinstance(err_body, dict) else None
err_msg = err_body.get("errorMessage") if isinstance(err_body, dict) else None
- esm["LastProcessingResult"] = "FAILED"
logger.warning(
"ESM: Lambda %s failed processing SQS batch from %s (errorType=%s errorMessage=%s)\n%s",
func_name, queue_name, err_type, err_msg, result.get("log", ""),
@@ -7701,11 +7742,9 @@ def _run_sqs_batch(esm, func_rec, event, batch, queue_url, queue_name):
n_failed = len(batch) - len(succeeded)
if n_failed:
- esm["LastProcessingResult"] = f"OK - {len(succeeded)} records, {n_failed} partial failures"
logger.info("ESM: Lambda %s processed %d SQS messages from %s (%d partial failures)",
func_name, len(succeeded), queue_name, n_failed)
else:
- esm["LastProcessingResult"] = f"OK - {len(batch)} records"
logger.info("ESM: Lambda %s processed %d SQS messages from %s", func_name, len(batch), queue_name)
log_output = result.get("log", "")
if log_output:
@@ -7834,7 +7873,7 @@ def _poll_kinesis():
err_body = result.get("body") or {}
err_type = err_body.get("errorType") if isinstance(err_body, dict) else None
err_msg = err_body.get("errorMessage") if isinstance(err_body, dict) else None
- esm["LastProcessingResult"] = "FAILED"
+ esm["LastProcessingResult"] = "PROBLEM: Function call failed"
logger.warning(
"ESM: Lambda %s failed processing Kinesis batch from %s/%s (errorType=%s errorMessage=%s)\n%s",
func_name, stream_name, shard_id, err_type, err_msg, result.get("log", ""),
@@ -7847,7 +7886,7 @@ def _poll_kinesis():
processed_any = True
_esm_backoff_until.pop(esm_id, None)
positions[shard_id] = pos + len(raw_records)
- esm["LastProcessingResult"] = f"OK - {len(raw_records)} records"
+ esm["LastProcessingResult"] = "OK"
log_output = result.get("log", "")
if log_output:
logger.info("ESM: Lambda %s output:\n%s", func_name, log_output)
@@ -8014,7 +8053,7 @@ def _poll_dynamodb_streams():
err_body = result.get("body") or {}
err_type = err_body.get("errorType") if isinstance(err_body, dict) else None
err_msg = err_body.get("errorMessage") if isinstance(err_body, dict) else None
- esm["LastProcessingResult"] = "FAILED"
+ esm["LastProcessingResult"] = "PROBLEM: Function call failed"
logger.warning(
"ESM: Lambda %s failed processing DynamoDB stream batch from %s (errorType=%s errorMessage=%s)\n%s",
func_name, table_name, err_type, err_msg, result.get("log", ""),
@@ -8046,7 +8085,7 @@ def _poll_dynamodb_streams():
_esm_backoff_until.pop(esm_id, None)
with _dynamodb_stream_positions_lock:
_dynamodb_stream_positions[esm_id] = pos + raw_len
- esm["LastProcessingResult"] = f"OK - {len(batch)} records"
+ esm["LastProcessingResult"] = "OK"
log_output = result.get("log", "")
if log_output:
logger.info("ESM: Lambda %s output:\n%s", func_name, log_output)
diff --git a/tests/test_lambda.py b/tests/test_lambda.py
index 3d24810e..0846fa85 100644
--- a/tests/test_lambda.py
+++ b/tests/test_lambda.py
@@ -947,7 +947,7 @@ def test_lambda_esm_sqs(lam, sqs):
Enabled=True,
)
esm_uuid = resp["UUID"]
- assert resp["State"] == "Enabled"
+ assert resp["State"] == "Creating" # as AWS answers CreateEventSourceMapping
# Send a message to SQS
sqs.send_message(QueueUrl=q_url, MessageBody="trigger-lambda")
@@ -1427,7 +1427,8 @@ def test_lambda_esm_sqs_comprehensive(lam, sqs):
Enabled=True,
)
esm_uuid = resp["UUID"]
- assert resp["State"] == "Enabled"
+ assert resp["State"] == "Creating" # as AWS answers CreateEventSourceMapping
+ assert lam.get_event_source_mapping(UUID=esm_uuid)["State"] == "Enabled"
assert resp["BatchSize"] == 5
assert resp["EventSourceArn"] == q_arn
@@ -15824,3 +15825,107 @@ def test_lambda_create_esm_kinesis_scope_messages_follow_aws(lam, event_source_a
assert (err["Code"], err["Message"]) == ("InvalidParameterValueException", message)
finally:
lam.delete_function(FunctionName=name)
+
+
+# ---------------------------------------------------------------------------
+# Event source mapping responses, states and results as AWS answers them
+# (recorded against Lambda in us-east-1)
+# ---------------------------------------------------------------------------
+
+def _esm_function(lam, code):
+ name = f"esm-aws-{_uuid_mod.uuid4().hex[:8]}"
+ buf = io.BytesIO()
+ with zipfile.ZipFile(buf, "w") as zf:
+ zf.writestr("index.py", code)
+ lam.create_function(FunctionName=name, Runtime="python3.12", Role=_LAMBDA_ROLE, Handler="index.handler",
+ Code={"ZipFile": buf.getvalue()})
+ return name
+
+
+def test_esm_stream_and_sqs_mappings_have_aws_shape(lam, kin, sqs):
+ fn = _esm_function(lam, "def handler(e, c):\n return None\n")
+ stream = f"esm-aws-{_uuid_mod.uuid4().hex[:8]}"
+ kin.create_stream(StreamName=stream, ShardCount=1)
+ stream_arn = kin.describe_stream_summary(StreamName=stream)["StreamDescriptionSummary"]["StreamARN"]
+ queue_url = sqs.create_queue(QueueName=f"esm-aws-{_uuid_mod.uuid4().hex[:8]}")["QueueUrl"]
+ queue_arn = sqs.get_queue_attributes(QueueUrl=queue_url, AttributeNames=["QueueArn"])["Attributes"]["QueueArn"]
+ try:
+ created = lam.create_event_source_mapping(FunctionName=fn, EventSourceArn=stream_arn,
+ StartingPosition="LATEST")
+ assert created["ResponseMetadata"]["HTTPStatusCode"] == 202
+ assert {k: created[k] for k in (
+ "State", "StateTransitionReason", "BatchSize", "MaximumRetryAttempts", "MaximumRecordAgeInSeconds",
+ "BisectBatchOnFunctionError", "ParallelizationFactor", "TumblingWindowInSeconds", "DestinationConfig",
+ "LastProcessingResult", "MaximumBatchingWindowInSeconds")} == {
+ "State": "Creating", "StateTransitionReason": "User action", "BatchSize": 100,
+ "MaximumRetryAttempts": -1, "MaximumRecordAgeInSeconds": -1, "BisectBatchOnFunctionError": False,
+ "ParallelizationFactor": 1, "TumblingWindowInSeconds": 0, "DestinationConfig": {"OnFailure": {}},
+ "LastProcessingResult": "No records processed", "MaximumBatchingWindowInSeconds": 0}
+ uuid = created["UUID"]
+ assert lam.get_event_source_mapping(UUID=uuid)["State"] == "Enabled"
+ assert lam.update_event_source_mapping(UUID=uuid, BatchSize=5)["State"] == "Updating"
+ assert lam.update_event_source_mapping(UUID=uuid, Enabled=False)["State"] == "Disabling"
+ assert lam.update_event_source_mapping(UUID=uuid, BatchSize=7)["State"] == "Updating"
+ assert lam.get_event_source_mapping(UUID=uuid)["State"] == "Disabled"
+ assert lam.update_event_source_mapping(UUID=uuid, Enabled=True)["State"] == "Enabling"
+ with pytest.raises(ClientError) as exc:
+ lam.create_event_source_mapping(FunctionName=fn, EventSourceArn=stream_arn, StartingPosition="LATEST")
+ assert exc.value.response["ResponseMetadata"]["HTTPStatusCode"] == 409
+ assert exc.value.response["Error"] == {
+ "Code": "ResourceConflictException",
+ "Message": f'The event source arn (" {stream_arn} ") and function (" {fn} ") provided mapping already '
+ f"exists. Please update or delete the existing mapping with UUID {uuid}"}
+ assert lam.delete_event_source_mapping(UUID=uuid)["State"] == "Deleting"
+ with pytest.raises(ClientError) as exc:
+ lam.get_event_source_mapping(UUID=uuid)
+ assert exc.value.response["Error"] == {"Code": "ResourceNotFoundException",
+ "Message": "The resource you requested does not exist."}
+
+ created = lam.create_event_source_mapping(FunctionName=fn, EventSourceArn=queue_arn)
+ assert created["State"] == "Creating"
+ assert created["StateTransitionReason"] == "USER_INITIATED"
+ assert created["BatchSize"] == 10
+ for absent in ("LastProcessingResult", "MaximumRetryAttempts", "ParallelizationFactor", "StartingPosition"):
+ assert absent not in created
+ uuid = created["UUID"]
+ assert lam.update_event_source_mapping(UUID=uuid, Enabled=False)["State"] == "Disabling"
+ assert lam.update_event_source_mapping(UUID=uuid, BatchSize=5)["State"] == "Disabled"
+ with pytest.raises(ClientError) as exc:
+ lam.create_event_source_mapping(FunctionName=fn, EventSourceArn=queue_arn)
+ assert exc.value.response["Error"]["Message"] == (
+ f'An event source mapping with SQS arn (" {queue_arn} ") and function (" {fn} ") already exists. '
+ f"Please update or delete the existing mapping with UUID {uuid}")
+ lam.delete_event_source_mapping(UUID=uuid)
+ created = lam.create_event_source_mapping(FunctionName=fn, EventSourceArn=queue_arn, Enabled=False)
+ assert created["State"] == "Disabled"
+ lam.delete_event_source_mapping(UUID=created["UUID"])
+ finally:
+ lam.delete_function(FunctionName=fn)
+ kin.delete_stream(StreamName=stream)
+ sqs.delete_queue(QueueUrl=queue_url)
+
+
[email protected]("code,result", [
+ ("def handler(e, c):\n return None\n", "OK"),
+ ("def handler(e, c):\n raise Exception('boom')\n", "PROBLEM: Function call failed"),
+])
+def test_esm_kinesis_last_processing_result_is_aws_wording(lam, kin, code, result):
+ fn = _esm_function(lam, code)
+ stream = f"esm-aws-{_uuid_mod.uuid4().hex[:8]}"
+ kin.create_stream(StreamName=stream, ShardCount=1)
+ arn = kin.describe_stream_summary(StreamName=stream)["StreamDescriptionSummary"]["StreamARN"]
+ kin.put_record(StreamName=stream, Data=b"x", PartitionKey="pk")
+ try:
+ uuid = lam.create_event_source_mapping(FunctionName=fn, EventSourceArn=arn, StartingPosition="TRIM_HORIZON",
+ MaximumRetryAttempts=0)["UUID"]
+ got, deadline = "", time.time() + 30
+ while time.time() < deadline:
+ got = lam.get_event_source_mapping(UUID=uuid)["LastProcessingResult"]
+ if got != "No records processed":
+ break
+ time.sleep(0.5)
+ assert got == result
+ lam.delete_event_source_mapping(UUID=uuid)
+ finally:
+ lam.delete_function(FunctionName=fn)
+ kin.delete_stream(StreamName=stream)