Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

feat: EMR control plane — RunJobFlow, steps, instance groups/fleets, 656 tests (v1.1.2) (#22)

Nahuel Nucera581fe91e6ba3
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 23a3c2a9..6abd58f8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,29 @@ Versioning follows [Semantic Versioning](https://semver.org/).
---
+## [1.1.2] — 2026-03-29
+
+### Added
+- **Amazon EMR** (`ministack/services/emr.py`) — full control plane emulation (no real Spark/Hadoop)
+ - **Clusters**: `RunJobFlow`, `DescribeCluster`, `ListClusters`, `TerminateJobFlows`, `ModifyCluster`, `SetTerminationProtection`, `SetVisibleToAllUsers`
+ - **Steps**: `AddJobFlowSteps`, `DescribeStep`, `ListSteps`, `CancelSteps` — steps stored as COMPLETED immediately (emulator behaviour)
+ - **Instance Fleets**: `AddInstanceFleet`, `ListInstanceFleets`, `ModifyInstanceFleet`
+ - **Instance Groups**: `AddInstanceGroups`, `ListInstanceGroups`, `ModifyInstanceGroups`
+ - **Bootstrap Actions**: `ListBootstrapActions`
+ - **Tags**: `AddTags`, `RemoveTags`
+ - **Block Public Access**: `GetBlockPublicAccessConfiguration`, `PutBlockPublicAccessConfiguration`
+ - All three instance config modes: simple (`MasterInstanceType`/`SlaveInstanceType`/`InstanceCount`), `InstanceGroups`, `InstanceFleets`
+ - `KeepJobFlowAliveWhenNoSteps=True` → `WAITING`; `False` → `TERMINATED`
+ - `TerminationProtected=True` raises `ValidationException` on `TerminateJobFlows`
+ - JSON protocol via `X-Amz-Target: ElasticMapReduce.{Op}`, credential scope `elasticmapreduce`
+ - Pro-only on LocalStack — free in MiniStack
+ - 12 integration tests
+
+### Tests
+- 656 integration tests — all passing
+
+---
+
## [1.1.1] — 2026-03-29
### Added
diff --git a/README.md b/README.md
index f2f1f633..7d696d3d 100644
--- a/README.md
+++ b/README.md
@@ -236,6 +236,7 @@ subnet = ec2.create_subnet(
| **Firehose** | CreateDeliveryStream, DeleteDeliveryStream, DescribeDeliveryStream, ListDeliveryStreams, PutRecord, PutRecordBatch, UpdateDestination, TagDeliveryStream, UntagDeliveryStream, ListTagsForDeliveryStream, StartDeliveryStreamEncryption, StopDeliveryStreamEncryption | S3 destinations write records to the local S3 emulator; all other destination types buffer in-memory; concurrency-safe `UpdateDestination` via `VersionId` |
| **Route53** | CreateHostedZone, GetHostedZone, DeleteHostedZone, ListHostedZones, ListHostedZonesByName, UpdateHostedZoneComment, ChangeResourceRecordSets (CREATE/UPSERT/DELETE), ListResourceRecordSets, GetChange, CreateHealthCheck, GetHealthCheck, DeleteHealthCheck, ListHealthChecks, UpdateHealthCheck, ChangeTagsForResource, ListTagsForResource | REST/XML protocol; SOA + NS records auto-created; CallerReference idempotency; alias records, weighted/failover/latency routing; marker-based pagination |
| **EC2** | RunInstances, DescribeInstances, TerminateInstances, StopInstances, StartInstances, RebootInstances, DescribeImages, CreateSecurityGroup, DeleteSecurityGroup, DescribeSecurityGroups, AuthorizeSecurityGroupIngress, RevokeSecurityGroupIngress, AuthorizeSecurityGroupEgress, RevokeSecurityGroupEgress, CreateKeyPair, DeleteKeyPair, DescribeKeyPairs, ImportKeyPair, CreateVpc, DeleteVpc, DescribeVpcs, ModifyVpcAttribute, CreateSubnet, DeleteSubnet, DescribeSubnets, ModifySubnetAttribute, CreateInternetGateway, DeleteInternetGateway, DescribeInternetGateways, AttachInternetGateway, DetachInternetGateway, CreateRouteTable, DeleteRouteTable, DescribeRouteTables, AssociateRouteTable, DisassociateRouteTable, CreateRoute, ReplaceRoute, DeleteRoute, CreateNetworkInterface, DeleteNetworkInterface, DescribeNetworkInterfaces, AttachNetworkInterface, DetachNetworkInterface, CreateVpcEndpoint, DeleteVpcEndpoints, DescribeVpcEndpoints, DescribeAvailabilityZones, AllocateAddress, ReleaseAddress, AssociateAddress, DisassociateAddress, DescribeAddresses, CreateTags, DeleteTags, DescribeTags | In-memory state only — no real VMs; default VPC, subnet, security group, internet gateway, and route table always present; rules stored but not enforced (matches LocalStack) |
+| **EMR** | RunJobFlow, DescribeCluster, ListClusters, TerminateJobFlows, ModifyCluster, SetTerminationProtection, SetVisibleToAllUsers, AddJobFlowSteps, DescribeStep, ListSteps, CancelSteps, AddInstanceFleet, ListInstanceFleets, ModifyInstanceFleet, AddInstanceGroups, ListInstanceGroups, ModifyInstanceGroups, ListBootstrapActions, AddTags, RemoveTags, GetBlockPublicAccessConfiguration, PutBlockPublicAccessConfiguration | Control plane only — no real Spark/Hadoop; clusters start in WAITING (KeepAlive=true) or TERMINATED (KeepAlive=false); steps stored as COMPLETED immediately; all three instance modes (simple, InstanceGroups, InstanceFleets); TerminationProtected enforced; Pro-only on LocalStack — free here |
| **Cognito** | **User Pools**: CreateUserPool, DeleteUserPool, DescribeUserPool, ListUserPools, UpdateUserPool, CreateUserPoolClient, DeleteUserPoolClient, DescribeUserPoolClient, ListUserPoolClients, UpdateUserPoolClient, AdminCreateUser, AdminDeleteUser, AdminGetUser, ListUsers, AdminSetUserPassword, AdminUpdateUserAttributes, AdminConfirmSignUp, AdminDisableUser, AdminEnableUser, AdminResetUserPassword, AdminUserGlobalSignOut, AdminAddUserToGroup, AdminRemoveUserFromGroup, AdminListGroupsForUser, AdminListUserAuthEvents, AdminInitiateAuth, AdminRespondToAuthChallenge, InitiateAuth, RespondToAuthChallenge, GlobalSignOut, RevokeToken, SignUp, ConfirmSignUp, ForgotPassword, ConfirmForgotPassword, ChangePassword, GetUser, UpdateUserAttributes, DeleteUser, CreateGroup, DeleteGroup, GetGroup, ListGroups, ListUsersInGroup, CreateUserPoolDomain, DeleteUserPoolDomain, DescribeUserPoolDomain, GetUserPoolMfaConfig, SetUserPoolMfaConfig, AssociateSoftwareToken, VerifySoftwareToken, TagResource, UntagResource, ListTagsForResource; **Identity Pools**: CreateIdentityPool, DeleteIdentityPool, DescribeIdentityPool, ListIdentityPools, UpdateIdentityPool, GetId, GetCredentialsForIdentity, GetOpenIdToken, SetIdentityPoolRoles, GetIdentityPoolRoles, ListIdentities, DescribeIdentity, MergeDeveloperIdentities, UnlinkDeveloperIdentity, UnlinkIdentity, TagResource, UntagResource, ListTagsForResource; **OAuth2**: /oauth2/token (client_credentials) | Stub JWT tokens (structurally valid base64url JWTs); SRP auth returns PASSWORD_VERIFIER challenge; confirmation codes hardcoded (signup: 123456, forgot-password: 654321); MFA config stored in-memory |
---
@@ -506,6 +507,7 @@ provider "aws" {
cognitoidp = "http://localhost:4566"
cognitoidentity = "http://localhost:4566"
ec2 = "http://localhost:4566"
+ emr = "http://localhost:4566"
}
}
```
@@ -556,6 +558,7 @@ config:
| **Route53** | ✅ | ✅ | ✅ |
| **Cognito** | ✅ | ✅ | ✅ |
| **EC2** | ✅ | ✅ | ✅ |
+| **EMR** | ✅ | Paid | ✅ |
| CloudFormation | ❌ | partial | ✅ |
| Cost | **Free** | Was free, now paid | $35+/mo |
| Docker image size | ~150MB | ~1GB | ~1GB |
diff --git a/ministack/app.py b/ministack/app.py
index 81b6cf37..bb6febf6 100644
--- a/ministack/app.py
+++ b/ministack/app.py
@@ -31,6 +31,7 @@ from ministack.services import firehose
from ministack.services import apigateway_v1
from ministack.services import route53
from ministack.services import cognito
+from ministack.services import emr
from ministack.services.iam_sts import handle_iam_request, handle_sts_request
LOG_LEVEL = os.environ.get("LOG_LEVEL", "INFO").upper()
@@ -68,6 +69,7 @@ SERVICE_HANDLERS = {
"cognito-idp": cognito.handle_request,
"cognito-identity": cognito.handle_request,
"ec2": ec2.handle_request,
+ "elasticmapreduce": emr.handle_request,
}
SERVICE_NAME_ALIASES = {
@@ -118,7 +120,7 @@ BANNER = r"""
Services: S3, SQS, SNS, DynamoDB, Lambda, IAM, STS, SecretsManager, CloudWatch Logs,
SSM, EventBridge, Kinesis, CloudWatch, SES, Step Functions,
ECS, RDS, ElastiCache, Glue, Athena, API Gateway, Firehose, Route53,
- Cognito
+ Cognito, EC2, EMR
"""
@@ -375,6 +377,7 @@ def _reset_all_state():
(route53, route53.reset),
(cognito, cognito.reset),
(ec2, ec2.reset),
+ (emr, emr.reset),
]:
try:
fn()
diff --git a/ministack/core/router.py b/ministack/core/router.py
index 303cf058..b8377a80 100644
--- a/ministack/core/router.py
+++ b/ministack/core/router.py
@@ -116,6 +116,10 @@ SERVICE_PATTERNS = {
"target_prefixes": ["AWSCognitoIdentityService"],
"host_patterns": [r"cognito-identity\."],
},
+ "elasticmapreduce": {
+ "target_prefixes": ["ElasticMapReduce"],
+ "host_patterns": [r"elasticmapreduce\."],
+ },
"ec2": {
"host_patterns": [r"ec2\."],
"path_patterns": [r"Action=.*Instance", r"Action=.*Security", r"Action=.*KeyPair",
@@ -165,6 +169,7 @@ def detect_service(method: str, path: str, headers: dict, query_params: dict) ->
"route53": "route53",
"cognito-idp": "cognito-idp",
"cognito-identity": "cognito-identity",
+ "elasticmapreduce": "elasticmapreduce",
}
if svc_name in scope_map:
return scope_map[svc_name]
diff --git a/ministack/services/emr.py b/ministack/services/emr.py
new file mode 100644
index 00000000..93c7f978
--- /dev/null
+++ b/ministack/services/emr.py
@@ -0,0 +1,579 @@
+"""
+EMR (Elastic MapReduce) Service Emulator.
+JSON protocol via X-Amz-Target: ElasticMapReduce.{Operation}
+In-memory only — no real Spark/Hadoop execution.
+
+Supports:
+ Clusters: RunJobFlow, DescribeCluster, ListClusters, TerminateJobFlows,
+ ModifyCluster, SetTerminationProtection, SetVisibleToAllUsers
+ Steps: AddJobFlowSteps, DescribeStep, ListSteps, CancelSteps
+ Instance Fleets: AddInstanceFleet, ListInstanceFleets, ModifyInstanceFleet
+ Instance Groups: AddInstanceGroups, ListInstanceGroups, ModifyInstanceGroups
+ Bootstrap: ListBootstrapActions
+ Tags: AddTags, RemoveTags
+ Block Public Access: GetBlockPublicAccessConfiguration, PutBlockPublicAccessConfiguration
+"""
+
+import os
+import json
+import random
+import string
+import logging
+import time
+
+from ministack.core.responses import json_response, error_response_json, new_uuid
+
+logger = logging.getLogger("emr")
+
+ACCOUNT_ID = os.environ.get("MINISTACK_ACCOUNT_ID", "000000000000")
+REGION = os.environ.get("MINISTACK_REGION", "us-east-1")
+
+# ---------------------------------------------------------------------------
+# State
+# ---------------------------------------------------------------------------
+
+_clusters: dict = {} # cluster_id -> cluster record
+_steps: dict = {} # cluster_id -> [step records]
+_block_public_access: dict = {
+ "BlockPublicSecurityGroupRules": False,
+ "PermittedPublicSecurityGroupRuleRanges": [],
+}
+
+# ---------------------------------------------------------------------------
+# ID generators
+# ---------------------------------------------------------------------------
+
+def _cluster_id():
+ chars = string.ascii_uppercase + string.digits
+ return "j-" + "".join(random.choices(chars, k=13))
+
+def _step_id():
+ chars = string.ascii_uppercase + string.digits
+ return "s-" + "".join(random.choices(chars, k=13))
+
+def _fleet_id():
+ chars = string.ascii_uppercase + string.digits
+ return "if-" + "".join(random.choices(chars, k=13))
+
+def _group_id():
+ chars = string.ascii_uppercase + string.digits
+ return "ig-" + "".join(random.choices(chars, k=13))
+
+def _now_iso():
+ return time.strftime("%Y-%m-%dT%H:%M:%S.000Z", time.gmtime())
+
+# ---------------------------------------------------------------------------
+# Handlers
+# ---------------------------------------------------------------------------
+
+def _run_job_flow(data):
+ name = data.get("Name")
+ if not name:
+ return error_response_json("ValidationException", "Name is required", 400)
+
+ cluster_id = _cluster_id()
+ arn = f"arn:aws:elasticmapreduce:{REGION}:{ACCOUNT_ID}:cluster/{cluster_id}"
+ instances = data.get("Instances", {})
+ keep_alive = instances.get("KeepJobFlowAliveWhenNoSteps", False)
+ tags = data.get("Tags", [])
+ applications = data.get("Applications", [])
+ bootstrap_actions = data.get("BootstrapActions", [])
+ release_label = data.get("ReleaseLabel", "emr-6.10.0")
+ log_uri = data.get("LogUri", "")
+ service_role = data.get("ServiceRole", "EMR_DefaultRole")
+ job_flow_role = data.get("JobFlowRole", "EMR_EC2_DefaultRole")
+ visible_to_all = data.get("VisibleToAllUsers", True)
+ termination_protected = instances.get("TerminationProtected", False)
+ now = _now_iso()
+
+ # Derive initial state: WAITING if keep_alive, else TERMINATED (no real execution)
+ initial_state = "WAITING" if keep_alive else "TERMINATED"
+
+ # Build instance fleets/groups from Instances config
+ instance_fleets = []
+ instance_groups = []
+
+ if instances.get("InstanceFleets"):
+ for fleet in instances["InstanceFleets"]:
+ instance_fleets.append({
+ "Id": _fleet_id(),
+ "Name": fleet.get("Name", fleet.get("InstanceFleetType", "MASTER")),
+ "Status": {"State": "RUNNING", "StateChangeReason": {}, "Timeline": {"CreationDateTime": now}},
+ "InstanceFleetType": fleet.get("InstanceFleetType", "MASTER"),
+ "TargetOnDemandCapacity": fleet.get("TargetOnDemandCapacity", 0),
+ "TargetSpotCapacity": fleet.get("TargetSpotCapacity", 0),
+ "ProvisionedOnDemandCapacity": fleet.get("TargetOnDemandCapacity", 0),
+ "ProvisionedSpotCapacity": fleet.get("TargetSpotCapacity", 0),
+ "InstanceTypeSpecifications": fleet.get("InstanceTypeConfigs", []),
+ })
+ elif instances.get("InstanceGroups"):
+ for ig in instances["InstanceGroups"]:
+ instance_groups.append({
+ "Id": _group_id(),
+ "Name": ig.get("Name", ig.get("InstanceRole", "MASTER")),
+ "Market": ig.get("Market", "ON_DEMAND"),
+ "InstanceGroupType": ig.get("InstanceRole", "MASTER"),
+ "InstanceType": ig.get("InstanceType", "m5.xlarge"),
+ "RequestedInstanceCount": ig.get("InstanceCount", 1),
+ "RunningInstanceCount": ig.get("InstanceCount", 1),
+ "Status": {"State": "RUNNING", "StateChangeReason": {}, "Timeline": {"CreationDateTime": now}},
+ })
+ else:
+ # Simple mode: MasterInstanceType / SlaveInstanceType / InstanceCount
+ master_type = instances.get("MasterInstanceType", "m5.xlarge")
+ slave_type = instances.get("SlaveInstanceType", "m5.xlarge")
+ instance_count = instances.get("InstanceCount", 1)
+ instance_groups = [
+ {
+ "Id": _group_id(),
+ "Name": "Master",
+ "Market": "ON_DEMAND",
+ "InstanceGroupType": "MASTER",
+ "InstanceType": master_type,
+ "RequestedInstanceCount": 1,
+ "RunningInstanceCount": 1,
+ "Status": {"State": "RUNNING", "StateChangeReason": {}, "Timeline": {"CreationDateTime": now}},
+ },
+ ]
+ if instance_count > 1:
+ instance_groups.append({
+ "Id": _group_id(),
+ "Name": "Core",
+ "Market": "ON_DEMAND",
+ "InstanceGroupType": "CORE",
+ "InstanceType": slave_type,
+ "RequestedInstanceCount": instance_count - 1,
+ "RunningInstanceCount": instance_count - 1,
+ "Status": {"State": "RUNNING", "StateChangeReason": {}, "Timeline": {"CreationDateTime": now}},
+ })
+
+ collection_type = "INSTANCE_FLEET" if instance_fleets else "INSTANCE_GROUP"
+
+ _clusters[cluster_id] = {
+ "Id": cluster_id,
+ "Name": name,
+ "ClusterArn": arn,
+ "Status": {
+ "State": initial_state,
+ "StateChangeReason": {"Code": "", "Message": ""},
+ "Timeline": {"CreationDateTime": now, "ReadyDateTime": now},
+ },
+ "Ec2InstanceAttributes": {
+ "Ec2KeyName": instances.get("Ec2KeyName", ""),
+ "Ec2SubnetId": instances.get("Ec2SubnetId", ""),
+ "Ec2AvailabilityZone": f"{REGION}a",
+ "IamInstanceProfile": job_flow_role,
+ "EmrManagedMasterSecurityGroup": instances.get("EmrManagedMasterSecurityGroup", ""),
+ "EmrManagedSlaveSecurityGroup": instances.get("EmrManagedSlaveSecurityGroup", ""),
+ },
+ "InstanceCollectionType": collection_type,
+ "LogUri": log_uri,
+ "ReleaseLabel": release_label,
+ "AutoTerminate": not keep_alive,
+ "TerminationProtected": termination_protected,
+ "VisibleToAllUsers": visible_to_all,
+ "Applications": applications,
+ "Tags": tags,
+ "ServiceRole": service_role,
+ "NormalizedInstanceHours": 0,
+ "MasterPublicDnsName": f"ec2-0-0-0-0.compute-1.amazonaws.com",
+ "StepConcurrencyLevel": data.get("StepConcurrencyLevel", 1),
+ "BootstrapActions": bootstrap_actions,
+ "InstanceFleets": instance_fleets,
+ "InstanceGroups": instance_groups,
+ }
+
+ # Steps passed at creation time
+ steps_in = data.get("Steps", [])
+ _steps[cluster_id] = []
+ for step in steps_in:
+ _steps[cluster_id].append(_make_step(step))
+
+ return json_response({"JobFlowId": cluster_id, "ClusterArn": arn})
+
+
+def _describe_cluster(data):
+ cluster_id = data.get("ClusterId")
+ cluster = _clusters.get(cluster_id)
+ if not cluster:
+ return error_response_json("InvalidRequestException",
+ f"Cluster id '{cluster_id}' is not valid.", 400)
+ return json_response({"Cluster": cluster})
+
+
+def _list_clusters(data):
+ state_filter = data.get("ClusterStates", [])
+ result = []
+ for c in _clusters.values():
+ state = c["Status"]["State"]
+ if state_filter and state not in state_filter:
+ continue
+ result.append({
+ "Id": c["Id"],
+ "Name": c["Name"],
+ "Status": c["Status"],
+ "NormalizedInstanceHours": c["NormalizedInstanceHours"],
+ "ClusterArn": c["ClusterArn"],
+ })
+ return json_response({"Clusters": result})
+
+
+def _terminate_job_flows(data):
+ ids = data.get("JobFlowIds", [])
+ for cid in ids:
+ cluster = _clusters.get(cid)
+ if cluster:
+ if cluster.get("TerminationProtected"):
+ return error_response_json(
+ "ValidationException",
+ f"Cluster {cid} is protected from termination. Disable termination protection first.", 400
+ )
+ cluster["Status"]["State"] = "TERMINATED"
+ cluster["Status"]["StateChangeReason"] = {"Code": "USER_REQUEST", "Message": "User request"}
+ return json_response({})
+
+
+def _modify_cluster(data):
+ cluster_id = data.get("ClusterId")
+ cluster = _clusters.get(cluster_id)
+ if not cluster:
+ return error_response_json("InvalidRequestException",
+ f"Cluster id '{cluster_id}' is not valid.", 400)
+ if "StepConcurrencyLevel" in data:
+ cluster["StepConcurrencyLevel"] = data["StepConcurrencyLevel"]
+ return json_response({"StepConcurrencyLevel": cluster["StepConcurrencyLevel"]})
+
+
+def _set_termination_protection(data):
+ ids = data.get("JobFlowIds", [])
+ protected = data.get("TerminationProtected", False)
+ for cid in ids:
+ if cid in _clusters:
+ _clusters[cid]["TerminationProtected"] = protected
+ return json_response({})
+
+
+def _set_visible_to_all_users(data):
+ ids = data.get("JobFlowIds", [])
+ visible = data.get("VisibleToAllUsers", True)
+ for cid in ids:
+ if cid in _clusters:
+ _clusters[cid]["VisibleToAllUsers"] = visible
+ return json_response({})
+
+
+# ---------------------------------------------------------------------------
+# Steps
+# ---------------------------------------------------------------------------
+
+def _make_step(step_config):
+ now = _now_iso()
+ return {
+ "Id": _step_id(),
+ "Name": step_config.get("Name", ""),
+ "Config": {
+ "Jar": step_config.get("HadoopJarStep", {}).get("Jar", ""),
+ "Properties": {p["Key"]: p["Value"] for p in step_config.get("HadoopJarStep", {}).get("Properties", [])},
+ "MainClass": step_config.get("HadoopJarStep", {}).get("MainClass", ""),
+ "Args": step_config.get("HadoopJarStep", {}).get("Args", []),
+ },
+ "ActionOnFailure": step_config.get("ActionOnFailure", "CONTINUE"),
+ "Status": {
+ "State": "COMPLETED",
+ "StateChangeReason": {},
+ "Timeline": {"CreationDateTime": now, "StartDateTime": now, "EndDateTime": now},
+ },
+ }
+
+
+def _add_job_flow_steps(data):
+ cluster_id = data.get("JobFlowId")
+ if cluster_id not in _clusters:
+ return error_response_json("InvalidRequestException",
+ f"Cluster id '{cluster_id}' is not valid.", 400)
+ step_ids = []
+ for step_config in data.get("Steps", []):
+ step = _make_step(step_config)
+ _steps.setdefault(cluster_id, []).append(step)
+ step_ids.append(step["Id"])
+ return json_response({"StepIds": step_ids})
+
+
+def _describe_step(data):
+ cluster_id = data.get("ClusterId")
+ step_id = data.get("StepId")
+ for step in _steps.get(cluster_id, []):
+ if step["Id"] == step_id:
+ return json_response({"Step": step})
+ return error_response_json("InvalidRequestException",
+ f"Step id '{step_id}' is not valid.", 400)
+
+
+def _list_steps(data):
+ cluster_id = data.get("ClusterId")
+ state_filter = data.get("StepStates", [])
+ steps = _steps.get(cluster_id, [])
+ if state_filter:
+ steps = [s for s in steps if s["Status"]["State"] in state_filter]
+ return json_response({"Steps": steps})
+
+
+def _cancel_steps(data):
+ cluster_id = data.get("ClusterId")
+ step_ids = data.get("StepIds", [])
+ cancelled = []
+ for step in _steps.get(cluster_id, []):
+ if step["Id"] in step_ids and step["Status"]["State"] in ("PENDING", "RUNNING"):
+ step["Status"]["State"] = "CANCELLED"
+ cancelled.append({"StepId": step["Id"], "Status": "SUBMITTED"})
+ elif step["Id"] in step_ids:
+ cancelled.append({"StepId": step["Id"], "Status": "FAILED_TO_CANCEL",
+ "Reason": f"Step in state {step['Status']['State']} cannot be cancelled"})
+ return json_response({"CancelStepsInfoList": cancelled})
+
+
+# ---------------------------------------------------------------------------
+# Instance Fleets
+# ---------------------------------------------------------------------------
+
+def _add_instance_fleet(data):
+ cluster_id = data.get("ClusterId")
+ cluster = _clusters.get(cluster_id)
+ if not cluster:
+ return error_response_json("InvalidRequestException",
+ f"Cluster id '{cluster_id}' is not valid.", 400)
+ fleet = data.get("InstanceFleet", {})
+ now = _now_iso()
+ fleet_id = _fleet_id()
+ record = {
+ "Id": fleet_id,
+ "Name": fleet.get("Name", fleet.get("InstanceFleetType", "TASK")),
+ "Status": {"State": "RUNNING", "StateChangeReason": {}, "Timeline": {"CreationDateTime": now}},
+ "InstanceFleetType": fleet.get("InstanceFleetType", "TASK"),
+ "TargetOnDemandCapacity": fleet.get("TargetOnDemandCapacity", 0),
+ "TargetSpotCapacity": fleet.get("TargetSpotCapacity", 0),
+ "ProvisionedOnDemandCapacity": fleet.get("TargetOnDemandCapacity", 0),
+ "ProvisionedSpotCapacity": fleet.get("TargetSpotCapacity", 0),
+ "InstanceTypeSpecifications": fleet.get("InstanceTypeConfigs", []),
+ }
+ cluster["InstanceFleets"].append(record)
+ return json_response({"ClusterArn": cluster["ClusterArn"], "InstanceFleetId": fleet_id})
+
+
+def _list_instance_fleets(data):
+ cluster_id = data.get("ClusterId")
+ cluster = _clusters.get(cluster_id)
+ if not cluster:
+ return error_response_json("InvalidRequestException",
+ f"Cluster id '{cluster_id}' is not valid.", 400)
+ return json_response({"InstanceFleets": cluster.get("InstanceFleets", [])})
+
+
+def _modify_instance_fleet(data):
+ cluster_id = data.get("ClusterId")
+ cluster = _clusters.get(cluster_id)
+ if not cluster:
+ return error_response_json("InvalidRequestException",
+ f"Cluster id '{cluster_id}' is not valid.", 400)
+ fleet_mod = data.get("InstanceFleet", {})
+ fleet_id = fleet_mod.get("InstanceFleetId")
+ for fleet in cluster.get("InstanceFleets", []):
+ if fleet["Id"] == fleet_id:
+ if "TargetOnDemandCapacity" in fleet_mod:
+ fleet["TargetOnDemandCapacity"] = fleet_mod["TargetOnDemandCapacity"]
+ fleet["ProvisionedOnDemandCapacity"] = fleet_mod["TargetOnDemandCapacity"]
+ if "TargetSpotCapacity" in fleet_mod:
+ fleet["TargetSpotCapacity"] = fleet_mod["TargetSpotCapacity"]
+ fleet["ProvisionedSpotCapacity"] = fleet_mod["TargetSpotCapacity"]
+ break
+ return json_response({})
+
+
+# ---------------------------------------------------------------------------
+# Instance Groups
+# ---------------------------------------------------------------------------
+
+def _add_instance_groups(data):
+ cluster_id = data.get("JobFlowId")
+ cluster = _clusters.get(cluster_id)
+ if not cluster:
+ return error_response_json("InvalidRequestException",
+ f"Cluster id '{cluster_id}' is not valid.", 400)
+ now = _now_iso()
+ group_ids = []
+ for ig in data.get("InstanceGroups", []):
+ gid = _group_id()
+ record = {
+ "Id": gid,
+ "Name": ig.get("Name", ig.get("InstanceRole", "TASK")),
+ "Market": ig.get("Market", "ON_DEMAND"),
+ "InstanceGroupType": ig.get("InstanceRole", "TASK"),
+ "InstanceType": ig.get("InstanceType", "m5.xlarge"),
+ "RequestedInstanceCount": ig.get("InstanceCount", 1),
+ "RunningInstanceCount": ig.get("InstanceCount", 1),
+ "Status": {"State": "RUNNING", "StateChangeReason": {}, "Timeline": {"CreationDateTime": now}},
+ }
+ cluster["InstanceGroups"].append(record)
+ group_ids.append(gid)
+ return json_response({"JobFlowId": cluster_id, "InstanceGroupIds": group_ids})
+
+
+def _list_instance_groups(data):
+ cluster_id = data.get("ClusterId")
+ cluster = _clusters.get(cluster_id)
+ if not cluster:
+ return error_response_json("InvalidRequestException",
+ f"Cluster id '{cluster_id}' is not valid.", 400)
+ return json_response({"InstanceGroups": cluster.get("InstanceGroups", [])})
+
+
+def _modify_instance_groups(data):
+ cluster_id = data.get("ClusterId")
+ cluster = _clusters.get(cluster_id)
+ if not cluster:
+ return error_response_json("InvalidRequestException",
+ f"Cluster id '{cluster_id}' is not valid.", 400)
+ for mod in data.get("InstanceGroups", []):
+ gid = mod.get("InstanceGroupId")
+ for ig in cluster.get("InstanceGroups", []):
+ if ig["Id"] == gid:
+ if "InstanceCount" in mod:
+ ig["RequestedInstanceCount"] = mod["InstanceCount"]
+ ig["RunningInstanceCount"] = mod["InstanceCount"]
+ break
+ return json_response({})
+
+
+# ---------------------------------------------------------------------------
+# Bootstrap Actions
+# ---------------------------------------------------------------------------
+
+def _list_bootstrap_actions(data):
+ cluster_id = data.get("ClusterId")
+ cluster = _clusters.get(cluster_id)
+ if not cluster:
+ return error_response_json("InvalidRequestException",
+ f"Cluster id '{cluster_id}' is not valid.", 400)
+ actions = [
+ {
+ "Name": ba.get("Name", ""),
+ "ScriptPath": ba.get("ScriptBootstrapAction", {}).get("Path", ""),
+ "Args": ba.get("ScriptBootstrapAction", {}).get("Args", []),
+ }
+ for ba in cluster.get("BootstrapActions", [])
+ ]
+ return json_response({"BootstrapActions": actions})
+
+
+# ---------------------------------------------------------------------------
+# Tags
+# ---------------------------------------------------------------------------
+
+def _add_tags(data):
+ resource_id = data.get("ResourceId")
+ cluster = _clusters.get(resource_id)
+ if not cluster:
+ return error_response_json("InvalidRequestException",
+ f"Resource id '{resource_id}' is not valid.", 400)
+ new_tags = data.get("Tags", [])
+ existing = {t["Key"]: i for i, t in enumerate(cluster["Tags"])}
+ for tag in new_tags:
+ idx = existing.get(tag["Key"])
+ if idx is not None:
+ cluster["Tags"][idx] = tag
+ else:
+ cluster["Tags"].append(tag)
+ existing[tag["Key"]] = len(cluster["Tags"]) - 1
+ return json_response({})
+
+
+def _remove_tags(data):
+ resource_id = data.get("ResourceId")
+ cluster = _clusters.get(resource_id)
+ if not cluster:
+ return error_response_json("InvalidRequestException",
+ f"Resource id '{resource_id}' is not valid.", 400)
+ keys = set(data.get("TagKeys", []))
+ cluster["Tags"] = [t for t in cluster["Tags"] if t["Key"] not in keys]
+ return json_response({})
+
+
+# ---------------------------------------------------------------------------
+# Block Public Access
+# ---------------------------------------------------------------------------
+
+def _get_block_public_access_configuration(data):
+ return json_response({
+ "BlockPublicAccessConfiguration": _block_public_access,
+ "BlockPublicAccessConfigurationMetadata": {
+ "CreationDateTime": _now_iso(),
+ "CreatedByArn": f"arn:aws:iam::{ACCOUNT_ID}:root",
+ },
+ })
+
+
+def _put_block_public_access_configuration(data):
+ config = data.get("BlockPublicAccessConfiguration", {})
+ _block_public_access["BlockPublicSecurityGroupRules"] = config.get(
+ "BlockPublicSecurityGroupRules", False
+ )
+ _block_public_access["PermittedPublicSecurityGroupRuleRanges"] = config.get(
+ "PermittedPublicSecurityGroupRuleRanges", []
+ )
+ return json_response({})
+
+
+# ---------------------------------------------------------------------------
+# Request routing
+# ---------------------------------------------------------------------------
+
+_HANDLERS = {
+ "RunJobFlow": _run_job_flow,
+ "DescribeCluster": _describe_cluster,
+ "ListClusters": _list_clusters,
+ "TerminateJobFlows": _terminate_job_flows,
+ "ModifyCluster": _modify_cluster,
+ "SetTerminationProtection": _set_termination_protection,
+ "SetVisibleToAllUsers": _set_visible_to_all_users,
+ "AddJobFlowSteps": _add_job_flow_steps,
+ "DescribeStep": _describe_step,
+ "ListSteps": _list_steps,
+ "CancelSteps": _cancel_steps,
+ "AddInstanceFleet": _add_instance_fleet,
+ "ListInstanceFleets": _list_instance_fleets,
+ "ModifyInstanceFleet": _modify_instance_fleet,
+ "AddInstanceGroups": _add_instance_groups,
+ "ListInstanceGroups": _list_instance_groups,
+ "ModifyInstanceGroups": _modify_instance_groups,
+ "ListBootstrapActions": _list_bootstrap_actions,
+ "AddTags": _add_tags,
+ "RemoveTags": _remove_tags,
+ "GetBlockPublicAccessConfiguration": _get_block_public_access_configuration,
+ "PutBlockPublicAccessConfiguration": _put_block_public_access_configuration,
+}
+
+
+async def handle_request(method, path, headers, body, query_params):
+ target = headers.get("x-amz-target", "")
+ action = target.split(".")[-1] if "." in target else ""
+
+ try:
+ data = json.loads(body) if body else {}
+ except json.JSONDecodeError:
+ return error_response_json("SerializationException", "Invalid JSON", 400)
+
+ handler = _HANDLERS.get(action)
+ if not handler:
+ return error_response_json("InvalidAction", f"Unknown EMR action: {action}", 400)
+ return handler(data)
+
+
+# ---------------------------------------------------------------------------
+# Reset
+# ---------------------------------------------------------------------------
+
+def reset():
+ _clusters.clear()
+ _steps.clear()
+ _block_public_access["BlockPublicSecurityGroupRules"] = False
+ _block_public_access["PermittedPublicSecurityGroupRuleRanges"] = []
diff --git a/pyproject.toml b/pyproject.toml
index 068d54ff..a3d93110 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "ministack"
-version = "1.1.1"
+version = "1.1.2"
description = "Free, open-source local AWS emulator — drop-in LocalStack replacement"
readme = "README.md"
license = { file = "LICENSE" }
diff --git a/tests/conftest.py b/tests/conftest.py
index 46e4a663..639cc672 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -145,6 +145,10 @@ def cognito_identity():
def ec2():
return make_client("ec2")
[email protected](scope="session")
+def emr():
+ return make_client("emr")
+
@pytest.fixture(scope="session")
def sfn_sync():
"""SFN client for StartSyncExecution — forces same endpoint (boto3 normally prefixes sync-)."""
diff --git a/tests/test_services.py b/tests/test_services.py
index 68b6303d..7b6f2ebe 100644
--- a/tests/test_services.py
+++ b/tests/test_services.py
@@ -11116,3 +11116,232 @@ def test_ec2_vpc_endpoint_crud(ec2):
def test_ec2_describe_route_tables_default(ec2):
desc = ec2.describe_route_tables()
assert any(rt["VpcId"] == "vpc-00000001" for rt in desc["RouteTables"])
+
+
+# ---------------------------------------------------------------------------
+# EMR
+# ---------------------------------------------------------------------------
+
+def test_emr_run_job_flow_simple(emr):
+ resp = emr.run_job_flow(
+ Name="test-cluster",
+ ReleaseLabel="emr-6.10.0",
+ Instances={
+ "MasterInstanceType": "m5.xlarge",
+ "SlaveInstanceType": "m5.xlarge",
+ "InstanceCount": 3,
+ "KeepJobFlowAliveWhenNoSteps": True,
+ },
+ JobFlowRole="EMR_EC2_DefaultRole",
+ ServiceRole="EMR_DefaultRole",
+ )
+ assert resp["JobFlowId"].startswith("j-")
+ assert "ClusterArn" in resp
+ assert "elasticmapreduce" in resp["ClusterArn"]
+
+
+def test_emr_describe_cluster(emr):
+ jf = emr.run_job_flow(
+ Name="describe-test",
+ ReleaseLabel="emr-6.10.0",
+ Instances={
+ "MasterInstanceType": "m5.xlarge",
+ "InstanceCount": 1,
+ "KeepJobFlowAliveWhenNoSteps": True,
+ },
+ JobFlowRole="EMR_EC2_DefaultRole",
+ ServiceRole="EMR_DefaultRole",
+ )
+ cluster_id = jf["JobFlowId"]
+ desc = emr.describe_cluster(ClusterId=cluster_id)
+ cluster = desc["Cluster"]
+ assert cluster["Id"] == cluster_id
+ assert cluster["Name"] == "describe-test"
+ assert cluster["Status"]["State"] == "WAITING"
+ assert cluster["ReleaseLabel"] == "emr-6.10.0"
+
+
+def test_emr_list_clusters(emr):
+ emr.run_job_flow(
+ Name="list-test",
+ ReleaseLabel="emr-6.10.0",
+ Instances={"MasterInstanceType": "m5.xlarge", "InstanceCount": 1, "KeepJobFlowAliveWhenNoSteps": True},
+ JobFlowRole="EMR_EC2_DefaultRole",
+ ServiceRole="EMR_DefaultRole",
+ )
+ resp = emr.list_clusters()
+ assert len(resp["Clusters"]) >= 1
+ assert all("Id" in c for c in resp["Clusters"])
+
+
+def test_emr_terminate_job_flows(emr):
+ jf = emr.run_job_flow(
+ Name="terminate-test",
+ ReleaseLabel="emr-6.10.0",
+ Instances={"MasterInstanceType": "m5.xlarge", "InstanceCount": 1, "KeepJobFlowAliveWhenNoSteps": True},
+ JobFlowRole="EMR_EC2_DefaultRole",
+ ServiceRole="EMR_DefaultRole",
+ )
+ cluster_id = jf["JobFlowId"]
+ emr.terminate_job_flows(JobFlowIds=[cluster_id])
+ desc = emr.describe_cluster(ClusterId=cluster_id)
+ assert desc["Cluster"]["Status"]["State"] == "TERMINATED"
+
+
+def test_emr_termination_protection(emr):
+ jf = emr.run_job_flow(
+ Name="protected-cluster",
+ ReleaseLabel="emr-6.10.0",
+ Instances={"MasterInstanceType": "m5.xlarge", "InstanceCount": 1,
+ "KeepJobFlowAliveWhenNoSteps": True, "TerminationProtected": True},
+ JobFlowRole="EMR_EC2_DefaultRole",
+ ServiceRole="EMR_DefaultRole",
+ )
+ cluster_id = jf["JobFlowId"]
+ import botocore.exceptions
+ try:
+ emr.terminate_job_flows(JobFlowIds=[cluster_id])
+ assert False, "should have raised"
+ except botocore.exceptions.ClientError as e:
+ assert "ValidationException" in str(e) or "protected" in str(e).lower()
+
+
+def test_emr_add_and_list_steps(emr):
+ jf = emr.run_job_flow(
+ Name="steps-cluster",
+ ReleaseLabel="emr-6.10.0",
+ Instances={"MasterInstanceType": "m5.xlarge", "InstanceCount": 1, "KeepJobFlowAliveWhenNoSteps": True},
+ JobFlowRole="EMR_EC2_DefaultRole",
+ ServiceRole="EMR_DefaultRole",
+ )
+ cluster_id = jf["JobFlowId"]
+ step_resp = emr.add_job_flow_steps(
+ JobFlowId=cluster_id,
+ Steps=[{
+ "Name": "my-spark-step",
+ "ActionOnFailure": "CONTINUE",
+ "HadoopJarStep": {
+ "Jar": "command-runner.jar",
+ "Args": ["spark-submit", "--class", "com.example.Main", "s3://bucket/app.jar"],
+ },
+ }],
+ )
+ assert len(step_resp["StepIds"]) == 1
+ step_id = step_resp["StepIds"][0]
+ assert step_id.startswith("s-")
+
+ steps = emr.list_steps(ClusterId=cluster_id)
+ assert any(s["Id"] == step_id for s in steps["Steps"])
+
+
+def test_emr_describe_step(emr):
+ jf = emr.run_job_flow(
+ Name="describe-step-cluster",
+ ReleaseLabel="emr-6.10.0",
+ Instances={"MasterInstanceType": "m5.xlarge", "InstanceCount": 1, "KeepJobFlowAliveWhenNoSteps": True},
+ JobFlowRole="EMR_EC2_DefaultRole",
+ ServiceRole="EMR_DefaultRole",
+ )
+ cluster_id = jf["JobFlowId"]
+ step_resp = emr.add_job_flow_steps(
+ JobFlowId=cluster_id,
+ Steps=[{"Name": "step1", "ActionOnFailure": "CONTINUE",
+ "HadoopJarStep": {"Jar": "command-runner.jar", "Args": []}}],
+ )
+ step_id = step_resp["StepIds"][0]
+ desc = emr.describe_step(ClusterId=cluster_id, StepId=step_id)
+ assert desc["Step"]["Id"] == step_id
+ assert desc["Step"]["Status"]["State"] == "COMPLETED"
+
+
+def test_emr_tags(emr):
+ jf = emr.run_job_flow(
+ Name="tagged-cluster",
+ ReleaseLabel="emr-6.10.0",
+ Instances={"MasterInstanceType": "m5.xlarge", "InstanceCount": 1, "KeepJobFlowAliveWhenNoSteps": True},
+ JobFlowRole="EMR_EC2_DefaultRole",
+ ServiceRole="EMR_DefaultRole",
+ Tags=[{"Key": "env", "Value": "test"}],
+ )
+ cluster_id = jf["JobFlowId"]
+ emr.add_tags(ResourceId=cluster_id, Tags=[{"Key": "team", "Value": "data"}])
+ desc = emr.describe_cluster(ClusterId=cluster_id)
+ tag_map = {t["Key"]: t["Value"] for t in desc["Cluster"]["Tags"]}
+ assert tag_map["env"] == "test"
+ assert tag_map["team"] == "data"
+
+ emr.remove_tags(ResourceId=cluster_id, TagKeys=["env"])
+ desc2 = emr.describe_cluster(ClusterId=cluster_id)
+ tag_keys = [t["Key"] for t in desc2["Cluster"]["Tags"]]
+ assert "env" not in tag_keys
+ assert "team" in tag_keys
+
+
+def test_emr_auto_terminate_state(emr):
+ """Cluster with KeepJobFlowAliveWhenNoSteps=False starts as TERMINATED."""
+ jf = emr.run_job_flow(
+ Name="auto-terminate-cluster",
+ ReleaseLabel="emr-6.10.0",
+ Instances={"MasterInstanceType": "m5.xlarge", "InstanceCount": 1, "KeepJobFlowAliveWhenNoSteps": False},
+ JobFlowRole="EMR_EC2_DefaultRole",
+ ServiceRole="EMR_DefaultRole",
+ )
+ cluster_id = jf["JobFlowId"]
+ desc = emr.describe_cluster(ClusterId=cluster_id)
+ assert desc["Cluster"]["Status"]["State"] == "TERMINATED"
+ assert desc["Cluster"]["AutoTerminate"] is True
+
+
+def test_emr_modify_cluster(emr):
+ jf = emr.run_job_flow(
+ Name="modify-cluster",
+ ReleaseLabel="emr-6.10.0",
+ Instances={"MasterInstanceType": "m5.xlarge", "InstanceCount": 1, "KeepJobFlowAliveWhenNoSteps": True},
+ JobFlowRole="EMR_EC2_DefaultRole",
+ ServiceRole="EMR_DefaultRole",
+ )
+ cluster_id = jf["JobFlowId"]
+ resp = emr.modify_cluster(ClusterId=cluster_id, StepConcurrencyLevel=5)
+ assert resp["StepConcurrencyLevel"] == 5
+
+
+def test_emr_block_public_access(emr):
+ resp = emr.get_block_public_access_configuration()
+ assert "BlockPublicAccessConfiguration" in resp
+ assert resp["BlockPublicAccessConfiguration"]["BlockPublicSecurityGroupRules"] is False
+
+ emr.put_block_public_access_configuration(
+ BlockPublicAccessConfiguration={
+ "BlockPublicSecurityGroupRules": True,
+ "PermittedPublicSecurityGroupRuleRanges": [{"MinRange": 22, "MaxRange": 22}],
+ }
+ )
+ resp2 = emr.get_block_public_access_configuration()
+ assert resp2["BlockPublicAccessConfiguration"]["BlockPublicSecurityGroupRules"] is True
+
+
+def test_emr_instance_groups(emr):
+ jf = emr.run_job_flow(
+ Name="ig-cluster",
+ ReleaseLabel="emr-6.10.0",
+ Instances={
+ "InstanceGroups": [
+ {"Name": "Master", "InstanceRole": "MASTER", "InstanceType": "m5.xlarge", "InstanceCount": 1},
+ {"Name": "Core", "InstanceRole": "CORE", "InstanceType": "m5.xlarge", "InstanceCount": 2},
+ ],
+ "KeepJobFlowAliveWhenNoSteps": True,
+ },
+ JobFlowRole="EMR_EC2_DefaultRole",
+ ServiceRole="EMR_DefaultRole",
+ )
+ cluster_id = jf["JobFlowId"]
+ groups = emr.list_instance_groups(ClusterId=cluster_id)
+ assert len(groups["InstanceGroups"]) >= 2
+
+ new_group_resp = emr.add_instance_groups(
+ JobFlowId=cluster_id,
+ InstanceGroups=[{"Name": "Task", "InstanceRole": "TASK", "InstanceType": "m5.xlarge", "InstanceCount": 2}],
+ )
+ assert len(new_group_resp["InstanceGroupIds"]) == 1
+ groups2 = emr.list_instance_groups(ClusterId=cluster_id)
+ assert len(groups2["InstanceGroups"]) == 3