Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.
Python awsaws-emulatoraws-localaws-sdkdevtoolsdockerdynamodbec2emulatorlambdalocalstacklocalstack-alternativeministackmock-awsopen-sourcepythons3sqsterraform
feat: EMR control plane — RunJobFlow, steps, instance groups/fleets, 656 tests (v1.1.2) (#22)
Nahuel Nucera581fe91e6ba3
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 23a3c2a9..6abd58f8 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,29 @@ Versioning follows [Semantic Versioning](https://semver.org/).---+## [1.1.2] — 2026-03-29++### Added+- **Amazon EMR** (`ministack/services/emr.py`) — full control plane emulation (no real Spark/Hadoop)+ - **Clusters**: `RunJobFlow`, `DescribeCluster`, `ListClusters`, `TerminateJobFlows`, `ModifyCluster`, `SetTerminationProtection`, `SetVisibleToAllUsers`+ - **Steps**: `AddJobFlowSteps`, `DescribeStep`, `ListSteps`, `CancelSteps` — steps stored as COMPLETED immediately (emulator behaviour)+ - **Instance Fleets**: `AddInstanceFleet`, `ListInstanceFleets`, `ModifyInstanceFleet`+ - **Instance Groups**: `AddInstanceGroups`, `ListInstanceGroups`, `ModifyInstanceGroups`+ - **Bootstrap Actions**: `ListBootstrapActions`+ - **Tags**: `AddTags`, `RemoveTags`+ - **Block Public Access**: `GetBlockPublicAccessConfiguration`, `PutBlockPublicAccessConfiguration`+ - All three instance config modes: simple (`MasterInstanceType`/`SlaveInstanceType`/`InstanceCount`), `InstanceGroups`, `InstanceFleets`+ - `KeepJobFlowAliveWhenNoSteps=True` → `WAITING`; `False` → `TERMINATED`+ - `TerminationProtected=True` raises `ValidationException` on `TerminateJobFlows`+ - JSON protocol via `X-Amz-Target: ElasticMapReduce.{Op}`, credential scope `elasticmapreduce`+ - Pro-only on LocalStack — free in MiniStack+ - 12 integration tests++### Tests+- 656 integration tests — all passing++---+## [1.1.1] — 2026-03-29### Addeddiff --git a/README.md b/README.mdindex f2f1f633..7d696d3d 100644--- a/README.md+++ b/README.md@@ -236,6 +236,7 @@ subnet = ec2.create_subnet(| **Firehose** | CreateDeliveryStream, DeleteDeliveryStream, DescribeDeliveryStream, ListDeliveryStreams, PutRecord, PutRecordBatch, UpdateDestination, TagDeliveryStream, UntagDeliveryStream, ListTagsForDeliveryStream, StartDeliveryStreamEncryption, StopDeliveryStreamEncryption | S3 destinations write records to the local S3 emulator; all other destination types buffer in-memory; concurrency-safe `UpdateDestination` via `VersionId` || **Route53** | CreateHostedZone, GetHostedZone, DeleteHostedZone, ListHostedZones, ListHostedZonesByName, UpdateHostedZoneComment, ChangeResourceRecordSets (CREATE/UPSERT/DELETE), ListResourceRecordSets, GetChange, CreateHealthCheck, GetHealthCheck, DeleteHealthCheck, ListHealthChecks, UpdateHealthCheck, ChangeTagsForResource, ListTagsForResource | REST/XML protocol; SOA + NS records auto-created; CallerReference idempotency; alias records, weighted/failover/latency routing; marker-based pagination || **EC2** | RunInstances, DescribeInstances, TerminateInstances, StopInstances, StartInstances, RebootInstances, DescribeImages, CreateSecurityGroup, DeleteSecurityGroup, DescribeSecurityGroups, AuthorizeSecurityGroupIngress, RevokeSecurityGroupIngress, AuthorizeSecurityGroupEgress, RevokeSecurityGroupEgress, CreateKeyPair, DeleteKeyPair, DescribeKeyPairs, ImportKeyPair, CreateVpc, DeleteVpc, DescribeVpcs, ModifyVpcAttribute, CreateSubnet, DeleteSubnet, DescribeSubnets, ModifySubnetAttribute, CreateInternetGateway, DeleteInternetGateway, DescribeInternetGateways, AttachInternetGateway, DetachInternetGateway, CreateRouteTable, DeleteRouteTable, DescribeRouteTables, AssociateRouteTable, DisassociateRouteTable, CreateRoute, ReplaceRoute, DeleteRoute, CreateNetworkInterface, DeleteNetworkInterface, DescribeNetworkInterfaces, AttachNetworkInterface, DetachNetworkInterface, CreateVpcEndpoint, DeleteVpcEndpoints, DescribeVpcEndpoints, DescribeAvailabilityZones, AllocateAddress, ReleaseAddress, AssociateAddress, DisassociateAddress, DescribeAddresses, CreateTags, DeleteTags, DescribeTags | In-memory state only — no real VMs; default VPC, subnet, security group, internet gateway, and route table always present; rules stored but not enforced (matches LocalStack) |+| **EMR** | RunJobFlow, DescribeCluster, ListClusters, TerminateJobFlows, ModifyCluster, SetTerminationProtection, SetVisibleToAllUsers, AddJobFlowSteps, DescribeStep, ListSteps, CancelSteps, AddInstanceFleet, ListInstanceFleets, ModifyInstanceFleet, AddInstanceGroups, ListInstanceGroups, ModifyInstanceGroups, ListBootstrapActions, AddTags, RemoveTags, GetBlockPublicAccessConfiguration, PutBlockPublicAccessConfiguration | Control plane only — no real Spark/Hadoop; clusters start in WAITING (KeepAlive=true) or TERMINATED (KeepAlive=false); steps stored as COMPLETED immediately; all three instance modes (simple, InstanceGroups, InstanceFleets); TerminationProtected enforced; Pro-only on LocalStack — free here || **Cognito** | **User Pools**: CreateUserPool, DeleteUserPool, DescribeUserPool, ListUserPools, UpdateUserPool, CreateUserPoolClient, DeleteUserPoolClient, DescribeUserPoolClient, ListUserPoolClients, UpdateUserPoolClient, AdminCreateUser, AdminDeleteUser, AdminGetUser, ListUsers, AdminSetUserPassword, AdminUpdateUserAttributes, AdminConfirmSignUp, AdminDisableUser, AdminEnableUser, AdminResetUserPassword, AdminUserGlobalSignOut, AdminAddUserToGroup, AdminRemoveUserFromGroup, AdminListGroupsForUser, AdminListUserAuthEvents, AdminInitiateAuth, AdminRespondToAuthChallenge, InitiateAuth, RespondToAuthChallenge, GlobalSignOut, RevokeToken, SignUp, ConfirmSignUp, ForgotPassword, ConfirmForgotPassword, ChangePassword, GetUser, UpdateUserAttributes, DeleteUser, CreateGroup, DeleteGroup, GetGroup, ListGroups, ListUsersInGroup, CreateUserPoolDomain, DeleteUserPoolDomain, DescribeUserPoolDomain, GetUserPoolMfaConfig, SetUserPoolMfaConfig, AssociateSoftwareToken, VerifySoftwareToken, TagResource, UntagResource, ListTagsForResource; **Identity Pools**: CreateIdentityPool, DeleteIdentityPool, DescribeIdentityPool, ListIdentityPools, UpdateIdentityPool, GetId, GetCredentialsForIdentity, GetOpenIdToken, SetIdentityPoolRoles, GetIdentityPoolRoles, ListIdentities, DescribeIdentity, MergeDeveloperIdentities, UnlinkDeveloperIdentity, UnlinkIdentity, TagResource, UntagResource, ListTagsForResource; **OAuth2**: /oauth2/token (client_credentials) | Stub JWT tokens (structurally valid base64url JWTs); SRP auth returns PASSWORD_VERIFIER challenge; confirmation codes hardcoded (signup: 123456, forgot-password: 654321); MFA config stored in-memory |---@@ -506,6 +507,7 @@ provider "aws" {cognitoidp = "http://localhost:4566"cognitoidentity = "http://localhost:4566"ec2 = "http://localhost:4566"+ emr = "http://localhost:4566"}}```@@ -556,6 +558,7 @@ config:| **Route53** | ✅ | ✅ | ✅ || **Cognito** | ✅ | ✅ | ✅ || **EC2** | ✅ | ✅ | ✅ |+| **EMR** | ✅ | Paid | ✅ || CloudFormation | ❌ | partial | ✅ || Cost | **Free** | Was free, now paid | $35+/mo || Docker image size | ~150MB | ~1GB | ~1GB |diff --git a/ministack/app.py b/ministack/app.pyindex 81b6cf37..bb6febf6 100644--- a/ministack/app.py+++ b/ministack/app.py@@ -31,6 +31,7 @@ from ministack.services import firehosefrom ministack.services import apigateway_v1from ministack.services import route53from ministack.services import cognito+from ministack.services import emrfrom ministack.services.iam_sts import handle_iam_request, handle_sts_requestLOG_LEVEL = os.environ.get("LOG_LEVEL", "INFO").upper()@@ -68,6 +69,7 @@ SERVICE_HANDLERS = {"cognito-idp": cognito.handle_request,"cognito-identity": cognito.handle_request,"ec2": ec2.handle_request,+ "elasticmapreduce": emr.handle_request,}SERVICE_NAME_ALIASES = {@@ -118,7 +120,7 @@ BANNER = r"""Services: S3, SQS, SNS, DynamoDB, Lambda, IAM, STS, SecretsManager, CloudWatch Logs,SSM, EventBridge, Kinesis, CloudWatch, SES, Step Functions,ECS, RDS, ElastiCache, Glue, Athena, API Gateway, Firehose, Route53,- Cognito+ Cognito, EC2, EMR"""@@ -375,6 +377,7 @@ def _reset_all_state():(route53, route53.reset),(cognito, cognito.reset),(ec2, ec2.reset),+ (emr, emr.reset),]:try:fn()diff --git a/ministack/core/router.py b/ministack/core/router.pyindex 303cf058..b8377a80 100644--- a/ministack/core/router.py+++ b/ministack/core/router.py@@ -116,6 +116,10 @@ SERVICE_PATTERNS = {"target_prefixes": ["AWSCognitoIdentityService"],"host_patterns": [r"cognito-identity\."],},+ "elasticmapreduce": {+ "target_prefixes": ["ElasticMapReduce"],+ "host_patterns": [r"elasticmapreduce\."],+ },"ec2": {"host_patterns": [r"ec2\."],"path_patterns": [r"Action=.*Instance", r"Action=.*Security", r"Action=.*KeyPair",@@ -165,6 +169,7 @@ def detect_service(method: str, path: str, headers: dict, query_params: dict) ->"route53": "route53","cognito-idp": "cognito-idp","cognito-identity": "cognito-identity",+ "elasticmapreduce": "elasticmapreduce",}if svc_name in scope_map:return scope_map[svc_name]diff --git a/ministack/services/emr.py b/ministack/services/emr.pynew file mode 100644index 00000000..93c7f978--- /dev/null+++ b/ministack/services/emr.py@@ -0,0 +1,579 @@+"""+EMR (Elastic MapReduce) Service Emulator.+JSON protocol via X-Amz-Target: ElasticMapReduce.{Operation}+In-memory only — no real Spark/Hadoop execution.++Supports:+ Clusters: RunJobFlow, DescribeCluster, ListClusters, TerminateJobFlows,+ ModifyCluster, SetTerminationProtection, SetVisibleToAllUsers+ Steps: AddJobFlowSteps, DescribeStep, ListSteps, CancelSteps+ Instance Fleets: AddInstanceFleet, ListInstanceFleets, ModifyInstanceFleet+ Instance Groups: AddInstanceGroups, ListInstanceGroups, ModifyInstanceGroups+ Bootstrap: ListBootstrapActions+ Tags: AddTags, RemoveTags+ Block Public Access: GetBlockPublicAccessConfiguration, PutBlockPublicAccessConfiguration+"""++import os+import json+import random+import string+import logging+import time++from ministack.core.responses import json_response, error_response_json, new_uuid++logger = logging.getLogger("emr")++ACCOUNT_ID = os.environ.get("MINISTACK_ACCOUNT_ID", "000000000000")+REGION = os.environ.get("MINISTACK_REGION", "us-east-1")++# ---------------------------------------------------------------------------+# State+# ---------------------------------------------------------------------------++_clusters: dict = {} # cluster_id -> cluster record+_steps: dict = {} # cluster_id -> [step records]+_block_public_access: dict = {+ "BlockPublicSecurityGroupRules": False,+ "PermittedPublicSecurityGroupRuleRanges": [],+}++# ---------------------------------------------------------------------------+# ID generators+# ---------------------------------------------------------------------------++def _cluster_id():+ chars = string.ascii_uppercase + string.digits+ return "j-" + "".join(random.choices(chars, k=13))++def _step_id():+ chars = string.ascii_uppercase + string.digits+ return "s-" + "".join(random.choices(chars, k=13))++def _fleet_id():+ chars = string.ascii_uppercase + string.digits+ return "if-" + "".join(random.choices(chars, k=13))++def _group_id():+ chars = string.ascii_uppercase + string.digits+ return "ig-" + "".join(random.choices(chars, k=13))++def _now_iso():+ return time.strftime("%Y-%m-%dT%H:%M:%S.000Z", time.gmtime())++# ---------------------------------------------------------------------------+# Handlers+# ---------------------------------------------------------------------------++def _run_job_flow(data):+ name = data.get("Name")+ if not name:+ return error_response_json("ValidationException", "Name is required", 400)++ cluster_id = _cluster_id()+ arn = f"arn:aws:elasticmapreduce:{REGION}:{ACCOUNT_ID}:cluster/{cluster_id}"+ instances = data.get("Instances", {})+ keep_alive = instances.get("KeepJobFlowAliveWhenNoSteps", False)+ tags = data.get("Tags", [])+ applications = data.get("Applications", [])+ bootstrap_actions = data.get("BootstrapActions", [])+ release_label = data.get("ReleaseLabel", "emr-6.10.0")+ log_uri = data.get("LogUri", "")+ service_role = data.get("ServiceRole", "EMR_DefaultRole")+ job_flow_role = data.get("JobFlowRole", "EMR_EC2_DefaultRole")+ visible_to_all = data.get("VisibleToAllUsers", True)+ termination_protected = instances.get("TerminationProtected", False)+ now = _now_iso()++ # Derive initial state: WAITING if keep_alive, else TERMINATED (no real execution)+ initial_state = "WAITING" if keep_alive else "TERMINATED"++ # Build instance fleets/groups from Instances config+ instance_fleets = []+ instance_groups = []++ if instances.get("InstanceFleets"):+ for fleet in instances["InstanceFleets"]:+ instance_fleets.append({+ "Id": _fleet_id(),+ "Name": fleet.get("Name", fleet.get("InstanceFleetType", "MASTER")),+ "Status": {"State": "RUNNING", "StateChangeReason": {}, "Timeline": {"CreationDateTime": now}},+ "InstanceFleetType": fleet.get("InstanceFleetType", "MASTER"),+ "TargetOnDemandCapacity": fleet.get("TargetOnDemandCapacity", 0),+ "TargetSpotCapacity": fleet.get("TargetSpotCapacity", 0),+ "ProvisionedOnDemandCapacity": fleet.get("TargetOnDemandCapacity", 0),+ "ProvisionedSpotCapacity": fleet.get("TargetSpotCapacity", 0),+ "InstanceTypeSpecifications": fleet.get("InstanceTypeConfigs", []),+ })+ elif instances.get("InstanceGroups"):+ for ig in instances["InstanceGroups"]:+ instance_groups.append({+ "Id": _group_id(),+ "Name": ig.get("Name", ig.get("InstanceRole", "MASTER")),+ "Market": ig.get("Market", "ON_DEMAND"),+ "InstanceGroupType": ig.get("InstanceRole", "MASTER"),+ "InstanceType": ig.get("InstanceType", "m5.xlarge"),+ "RequestedInstanceCount": ig.get("InstanceCount", 1),+ "RunningInstanceCount": ig.get("InstanceCount", 1),+ "Status": {"State": "RUNNING", "StateChangeReason": {}, "Timeline": {"CreationDateTime": now}},+ })+ else:+ # Simple mode: MasterInstanceType / SlaveInstanceType / InstanceCount+ master_type = instances.get("MasterInstanceType", "m5.xlarge")+ slave_type = instances.get("SlaveInstanceType", "m5.xlarge")+ instance_count = instances.get("InstanceCount", 1)+ instance_groups = [+ {+ "Id": _group_id(),+ "Name": "Master",+ "Market": "ON_DEMAND",+ "InstanceGroupType": "MASTER",+ "InstanceType": master_type,+ "RequestedInstanceCount": 1,+ "RunningInstanceCount": 1,+ "Status": {"State": "RUNNING", "StateChangeReason": {}, "Timeline": {"CreationDateTime": now}},+ },+ ]+ if instance_count > 1:+ instance_groups.append({+ "Id": _group_id(),+ "Name": "Core",+ "Market": "ON_DEMAND",+ "InstanceGroupType": "CORE",+ "InstanceType": slave_type,+ "RequestedInstanceCount": instance_count - 1,+ "RunningInstanceCount": instance_count - 1,+ "Status": {"State": "RUNNING", "StateChangeReason": {}, "Timeline": {"CreationDateTime": now}},+ })++ collection_type = "INSTANCE_FLEET" if instance_fleets else "INSTANCE_GROUP"++ _clusters[cluster_id] = {+ "Id": cluster_id,+ "Name": name,+ "ClusterArn": arn,+ "Status": {+ "State": initial_state,+ "StateChangeReason": {"Code": "", "Message": ""},+ "Timeline": {"CreationDateTime": now, "ReadyDateTime": now},+ },+ "Ec2InstanceAttributes": {+ "Ec2KeyName": instances.get("Ec2KeyName", ""),+ "Ec2SubnetId": instances.get("Ec2SubnetId", ""),+ "Ec2AvailabilityZone": f"{REGION}a",+ "IamInstanceProfile": job_flow_role,+ "EmrManagedMasterSecurityGroup": instances.get("EmrManagedMasterSecurityGroup", ""),+ "EmrManagedSlaveSecurityGroup": instances.get("EmrManagedSlaveSecurityGroup", ""),+ },+ "InstanceCollectionType": collection_type,+ "LogUri": log_uri,+ "ReleaseLabel": release_label,+ "AutoTerminate": not keep_alive,+ "TerminationProtected": termination_protected,+ "VisibleToAllUsers": visible_to_all,+ "Applications": applications,+ "Tags": tags,+ "ServiceRole": service_role,+ "NormalizedInstanceHours": 0,+ "MasterPublicDnsName": f"ec2-0-0-0-0.compute-1.amazonaws.com",+ "StepConcurrencyLevel": data.get("StepConcurrencyLevel", 1),+ "BootstrapActions": bootstrap_actions,+ "InstanceFleets": instance_fleets,+ "InstanceGroups": instance_groups,+ }++ # Steps passed at creation time+ steps_in = data.get("Steps", [])+ _steps[cluster_id] = []+ for step in steps_in:+ _steps[cluster_id].append(_make_step(step))++ return json_response({"JobFlowId": cluster_id, "ClusterArn": arn})+++def _describe_cluster(data):+ cluster_id = data.get("ClusterId")+ cluster = _clusters.get(cluster_id)+ if not cluster:+ return error_response_json("InvalidRequestException",+ f"Cluster id '{cluster_id}' is not valid.", 400)+ return json_response({"Cluster": cluster})+++def _list_clusters(data):+ state_filter = data.get("ClusterStates", [])+ result = []+ for c in _clusters.values():+ state = c["Status"]["State"]+ if state_filter and state not in state_filter:+ continue+ result.append({+ "Id": c["Id"],+ "Name": c["Name"],+ "Status": c["Status"],+ "NormalizedInstanceHours": c["NormalizedInstanceHours"],+ "ClusterArn": c["ClusterArn"],+ })+ return json_response({"Clusters": result})+++def _terminate_job_flows(data):+ ids = data.get("JobFlowIds", [])+ for cid in ids:+ cluster = _clusters.get(cid)+ if cluster:+ if cluster.get("TerminationProtected"):+ return error_response_json(+ "ValidationException",+ f"Cluster {cid} is protected from termination. Disable termination protection first.", 400+ )+ cluster["Status"]["State"] = "TERMINATED"+ cluster["Status"]["StateChangeReason"] = {"Code": "USER_REQUEST", "Message": "User request"}+ return json_response({})+++def _modify_cluster(data):+ cluster_id = data.get("ClusterId")+ cluster = _clusters.get(cluster_id)+ if not cluster:+ return error_response_json("InvalidRequestException",+ f"Cluster id '{cluster_id}' is not valid.", 400)+ if "StepConcurrencyLevel" in data:+ cluster["StepConcurrencyLevel"] = data["StepConcurrencyLevel"]+ return json_response({"StepConcurrencyLevel": cluster["StepConcurrencyLevel"]})+++def _set_termination_protection(data):+ ids = data.get("JobFlowIds", [])+ protected = data.get("TerminationProtected", False)+ for cid in ids:+ if cid in _clusters:+ _clusters[cid]["TerminationProtected"] = protected+ return json_response({})+++def _set_visible_to_all_users(data):+ ids = data.get("JobFlowIds", [])+ visible = data.get("VisibleToAllUsers", True)+ for cid in ids:+ if cid in _clusters:+ _clusters[cid]["VisibleToAllUsers"] = visible+ return json_response({})+++# ---------------------------------------------------------------------------+# Steps+# ---------------------------------------------------------------------------++def _make_step(step_config):+ now = _now_iso()+ return {+ "Id": _step_id(),+ "Name": step_config.get("Name", ""),+ "Config": {+ "Jar": step_config.get("HadoopJarStep", {}).get("Jar", ""),+ "Properties": {p["Key"]: p["Value"] for p in step_config.get("HadoopJarStep", {}).get("Properties", [])},+ "MainClass": step_config.get("HadoopJarStep", {}).get("MainClass", ""),+ "Args": step_config.get("HadoopJarStep", {}).get("Args", []),+ },+ "ActionOnFailure": step_config.get("ActionOnFailure", "CONTINUE"),+ "Status": {+ "State": "COMPLETED",+ "StateChangeReason": {},+ "Timeline": {"CreationDateTime": now, "StartDateTime": now, "EndDateTime": now},+ },+ }+++def _add_job_flow_steps(data):+ cluster_id = data.get("JobFlowId")+ if cluster_id not in _clusters:+ return error_response_json("InvalidRequestException",+ f"Cluster id '{cluster_id}' is not valid.", 400)+ step_ids = []+ for step_config in data.get("Steps", []):+ step = _make_step(step_config)+ _steps.setdefault(cluster_id, []).append(step)+ step_ids.append(step["Id"])+ return json_response({"StepIds": step_ids})+++def _describe_step(data):+ cluster_id = data.get("ClusterId")+ step_id = data.get("StepId")+ for step in _steps.get(cluster_id, []):+ if step["Id"] == step_id:+ return json_response({"Step": step})+ return error_response_json("InvalidRequestException",+ f"Step id '{step_id}' is not valid.", 400)+++def _list_steps(data):+ cluster_id = data.get("ClusterId")+ state_filter = data.get("StepStates", [])+ steps = _steps.get(cluster_id, [])+ if state_filter:+ steps = [s for s in steps if s["Status"]["State"] in state_filter]+ return json_response({"Steps": steps})+++def _cancel_steps(data):+ cluster_id = data.get("ClusterId")+ step_ids = data.get("StepIds", [])+ cancelled = []+ for step in _steps.get(cluster_id, []):+ if step["Id"] in step_ids and step["Status"]["State"] in ("PENDING", "RUNNING"):+ step["Status"]["State"] = "CANCELLED"+ cancelled.append({"StepId": step["Id"], "Status": "SUBMITTED"})+ elif step["Id"] in step_ids:+ cancelled.append({"StepId": step["Id"], "Status": "FAILED_TO_CANCEL",+ "Reason": f"Step in state {step['Status']['State']} cannot be cancelled"})+ return json_response({"CancelStepsInfoList": cancelled})+++# ---------------------------------------------------------------------------+# Instance Fleets+# ---------------------------------------------------------------------------++def _add_instance_fleet(data):+ cluster_id = data.get("ClusterId")+ cluster = _clusters.get(cluster_id)+ if not cluster:+ return error_response_json("InvalidRequestException",+ f"Cluster id '{cluster_id}' is not valid.", 400)+ fleet = data.get("InstanceFleet", {})+ now = _now_iso()+ fleet_id = _fleet_id()+ record = {+ "Id": fleet_id,+ "Name": fleet.get("Name", fleet.get("InstanceFleetType", "TASK")),+ "Status": {"State": "RUNNING", "StateChangeReason": {}, "Timeline": {"CreationDateTime": now}},+ "InstanceFleetType": fleet.get("InstanceFleetType", "TASK"),+ "TargetOnDemandCapacity": fleet.get("TargetOnDemandCapacity", 0),+ "TargetSpotCapacity": fleet.get("TargetSpotCapacity", 0),+ "ProvisionedOnDemandCapacity": fleet.get("TargetOnDemandCapacity", 0),+ "ProvisionedSpotCapacity": fleet.get("TargetSpotCapacity", 0),+ "InstanceTypeSpecifications": fleet.get("InstanceTypeConfigs", []),+ }+ cluster["InstanceFleets"].append(record)+ return json_response({"ClusterArn": cluster["ClusterArn"], "InstanceFleetId": fleet_id})+++def _list_instance_fleets(data):+ cluster_id = data.get("ClusterId")+ cluster = _clusters.get(cluster_id)+ if not cluster:+ return error_response_json("InvalidRequestException",+ f"Cluster id '{cluster_id}' is not valid.", 400)+ return json_response({"InstanceFleets": cluster.get("InstanceFleets", [])})+++def _modify_instance_fleet(data):+ cluster_id = data.get("ClusterId")+ cluster = _clusters.get(cluster_id)+ if not cluster:+ return error_response_json("InvalidRequestException",+ f"Cluster id '{cluster_id}' is not valid.", 400)+ fleet_mod = data.get("InstanceFleet", {})+ fleet_id = fleet_mod.get("InstanceFleetId")+ for fleet in cluster.get("InstanceFleets", []):+ if fleet["Id"] == fleet_id:+ if "TargetOnDemandCapacity" in fleet_mod:+ fleet["TargetOnDemandCapacity"] = fleet_mod["TargetOnDemandCapacity"]+ fleet["ProvisionedOnDemandCapacity"] = fleet_mod["TargetOnDemandCapacity"]+ if "TargetSpotCapacity" in fleet_mod:+ fleet["TargetSpotCapacity"] = fleet_mod["TargetSpotCapacity"]+ fleet["ProvisionedSpotCapacity"] = fleet_mod["TargetSpotCapacity"]+ break+ return json_response({})+++# ---------------------------------------------------------------------------+# Instance Groups+# ---------------------------------------------------------------------------++def _add_instance_groups(data):+ cluster_id = data.get("JobFlowId")+ cluster = _clusters.get(cluster_id)+ if not cluster:+ return error_response_json("InvalidRequestException",+ f"Cluster id '{cluster_id}' is not valid.", 400)+ now = _now_iso()+ group_ids = []+ for ig in data.get("InstanceGroups", []):+ gid = _group_id()+ record = {+ "Id": gid,+ "Name": ig.get("Name", ig.get("InstanceRole", "TASK")),+ "Market": ig.get("Market", "ON_DEMAND"),+ "InstanceGroupType": ig.get("InstanceRole", "TASK"),+ "InstanceType": ig.get("InstanceType", "m5.xlarge"),+ "RequestedInstanceCount": ig.get("InstanceCount", 1),+ "RunningInstanceCount": ig.get("InstanceCount", 1),+ "Status": {"State": "RUNNING", "StateChangeReason": {}, "Timeline": {"CreationDateTime": now}},+ }+ cluster["InstanceGroups"].append(record)+ group_ids.append(gid)+ return json_response({"JobFlowId": cluster_id, "InstanceGroupIds": group_ids})+++def _list_instance_groups(data):+ cluster_id = data.get("ClusterId")+ cluster = _clusters.get(cluster_id)+ if not cluster:+ return error_response_json("InvalidRequestException",+ f"Cluster id '{cluster_id}' is not valid.", 400)+ return json_response({"InstanceGroups": cluster.get("InstanceGroups", [])})+++def _modify_instance_groups(data):+ cluster_id = data.get("ClusterId")+ cluster = _clusters.get(cluster_id)+ if not cluster:+ return error_response_json("InvalidRequestException",+ f"Cluster id '{cluster_id}' is not valid.", 400)+ for mod in data.get("InstanceGroups", []):+ gid = mod.get("InstanceGroupId")+ for ig in cluster.get("InstanceGroups", []):+ if ig["Id"] == gid:+ if "InstanceCount" in mod:+ ig["RequestedInstanceCount"] = mod["InstanceCount"]+ ig["RunningInstanceCount"] = mod["InstanceCount"]+ break+ return json_response({})+++# ---------------------------------------------------------------------------+# Bootstrap Actions+# ---------------------------------------------------------------------------++def _list_bootstrap_actions(data):+ cluster_id = data.get("ClusterId")+ cluster = _clusters.get(cluster_id)+ if not cluster:+ return error_response_json("InvalidRequestException",+ f"Cluster id '{cluster_id}' is not valid.", 400)+ actions = [+ {+ "Name": ba.get("Name", ""),+ "ScriptPath": ba.get("ScriptBootstrapAction", {}).get("Path", ""),+ "Args": ba.get("ScriptBootstrapAction", {}).get("Args", []),+ }+ for ba in cluster.get("BootstrapActions", [])+ ]+ return json_response({"BootstrapActions": actions})+++# ---------------------------------------------------------------------------+# Tags+# ---------------------------------------------------------------------------++def _add_tags(data):+ resource_id = data.get("ResourceId")+ cluster = _clusters.get(resource_id)+ if not cluster:+ return error_response_json("InvalidRequestException",+ f"Resource id '{resource_id}' is not valid.", 400)+ new_tags = data.get("Tags", [])+ existing = {t["Key"]: i for i, t in enumerate(cluster["Tags"])}+ for tag in new_tags:+ idx = existing.get(tag["Key"])+ if idx is not None:+ cluster["Tags"][idx] = tag+ else:+ cluster["Tags"].append(tag)+ existing[tag["Key"]] = len(cluster["Tags"]) - 1+ return json_response({})+++def _remove_tags(data):+ resource_id = data.get("ResourceId")+ cluster = _clusters.get(resource_id)+ if not cluster:+ return error_response_json("InvalidRequestException",+ f"Resource id '{resource_id}' is not valid.", 400)+ keys = set(data.get("TagKeys", []))+ cluster["Tags"] = [t for t in cluster["Tags"] if t["Key"] not in keys]+ return json_response({})+++# ---------------------------------------------------------------------------+# Block Public Access+# ---------------------------------------------------------------------------++def _get_block_public_access_configuration(data):+ return json_response({+ "BlockPublicAccessConfiguration": _block_public_access,+ "BlockPublicAccessConfigurationMetadata": {+ "CreationDateTime": _now_iso(),+ "CreatedByArn": f"arn:aws:iam::{ACCOUNT_ID}:root",+ },+ })+++def _put_block_public_access_configuration(data):+ config = data.get("BlockPublicAccessConfiguration", {})+ _block_public_access["BlockPublicSecurityGroupRules"] = config.get(+ "BlockPublicSecurityGroupRules", False+ )+ _block_public_access["PermittedPublicSecurityGroupRuleRanges"] = config.get(+ "PermittedPublicSecurityGroupRuleRanges", []+ )+ return json_response({})+++# ---------------------------------------------------------------------------+# Request routing+# ---------------------------------------------------------------------------++_HANDLERS = {+ "RunJobFlow": _run_job_flow,+ "DescribeCluster": _describe_cluster,+ "ListClusters": _list_clusters,+ "TerminateJobFlows": _terminate_job_flows,+ "ModifyCluster": _modify_cluster,+ "SetTerminationProtection": _set_termination_protection,+ "SetVisibleToAllUsers": _set_visible_to_all_users,+ "AddJobFlowSteps": _add_job_flow_steps,+ "DescribeStep": _describe_step,+ "ListSteps": _list_steps,+ "CancelSteps": _cancel_steps,+ "AddInstanceFleet": _add_instance_fleet,+ "ListInstanceFleets": _list_instance_fleets,+ "ModifyInstanceFleet": _modify_instance_fleet,+ "AddInstanceGroups": _add_instance_groups,+ "ListInstanceGroups": _list_instance_groups,+ "ModifyInstanceGroups": _modify_instance_groups,+ "ListBootstrapActions": _list_bootstrap_actions,+ "AddTags": _add_tags,+ "RemoveTags": _remove_tags,+ "GetBlockPublicAccessConfiguration": _get_block_public_access_configuration,+ "PutBlockPublicAccessConfiguration": _put_block_public_access_configuration,+}+++async def handle_request(method, path, headers, body, query_params):+ target = headers.get("x-amz-target", "")+ action = target.split(".")[-1] if "." in target else ""++ try:+ data = json.loads(body) if body else {}+ except json.JSONDecodeError:+ return error_response_json("SerializationException", "Invalid JSON", 400)++ handler = _HANDLERS.get(action)+ if not handler:+ return error_response_json("InvalidAction", f"Unknown EMR action: {action}", 400)+ return handler(data)+++# ---------------------------------------------------------------------------+# Reset+# ---------------------------------------------------------------------------++def reset():+ _clusters.clear()+ _steps.clear()+ _block_public_access["BlockPublicSecurityGroupRules"] = False+ _block_public_access["PermittedPublicSecurityGroupRuleRanges"] = []diff --git a/pyproject.toml b/pyproject.tomlindex 068d54ff..a3d93110 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"[project]name = "ministack"-version = "1.1.1"+version = "1.1.2"description = "Free, open-source local AWS emulator — drop-in LocalStack replacement"readme = "README.md"license = { file = "LICENSE" }diff --git a/tests/conftest.py b/tests/conftest.pyindex 46e4a663..639cc672 100644--- a/tests/conftest.py+++ b/tests/conftest.py@@ -145,6 +145,10 @@ def cognito_identity():def ec2():return make_client("ec2")[email protected](scope="session")+def emr():+ return make_client("emr")+@pytest.fixture(scope="session")def sfn_sync():"""SFN client for StartSyncExecution — forces same endpoint (boto3 normally prefixes sync-)."""diff --git a/tests/test_services.py b/tests/test_services.pyindex 68b6303d..7b6f2ebe 100644--- a/tests/test_services.py+++ b/tests/test_services.py@@ -11116,3 +11116,232 @@ def test_ec2_vpc_endpoint_crud(ec2):def test_ec2_describe_route_tables_default(ec2):desc = ec2.describe_route_tables()assert any(rt["VpcId"] == "vpc-00000001" for rt in desc["RouteTables"])+++# ---------------------------------------------------------------------------+# EMR+# ---------------------------------------------------------------------------++def test_emr_run_job_flow_simple(emr):+ resp = emr.run_job_flow(+ Name="test-cluster",+ ReleaseLabel="emr-6.10.0",+ Instances={+ "MasterInstanceType": "m5.xlarge",+ "SlaveInstanceType": "m5.xlarge",+ "InstanceCount": 3,+ "KeepJobFlowAliveWhenNoSteps": True,+ },+ JobFlowRole="EMR_EC2_DefaultRole",+ ServiceRole="EMR_DefaultRole",+ )+ assert resp["JobFlowId"].startswith("j-")+ assert "ClusterArn" in resp+ assert "elasticmapreduce" in resp["ClusterArn"]+++def test_emr_describe_cluster(emr):+ jf = emr.run_job_flow(+ Name="describe-test",+ ReleaseLabel="emr-6.10.0",+ Instances={+ "MasterInstanceType": "m5.xlarge",+ "InstanceCount": 1,+ "KeepJobFlowAliveWhenNoSteps": True,+ },+ JobFlowRole="EMR_EC2_DefaultRole",+ ServiceRole="EMR_DefaultRole",+ )+ cluster_id = jf["JobFlowId"]+ desc = emr.describe_cluster(ClusterId=cluster_id)+ cluster = desc["Cluster"]+ assert cluster["Id"] == cluster_id+ assert cluster["Name"] == "describe-test"+ assert cluster["Status"]["State"] == "WAITING"+ assert cluster["ReleaseLabel"] == "emr-6.10.0"+++def test_emr_list_clusters(emr):+ emr.run_job_flow(+ Name="list-test",+ ReleaseLabel="emr-6.10.0",+ Instances={"MasterInstanceType": "m5.xlarge", "InstanceCount": 1, "KeepJobFlowAliveWhenNoSteps": True},+ JobFlowRole="EMR_EC2_DefaultRole",+ ServiceRole="EMR_DefaultRole",+ )+ resp = emr.list_clusters()+ assert len(resp["Clusters"]) >= 1+ assert all("Id" in c for c in resp["Clusters"])+++def test_emr_terminate_job_flows(emr):+ jf = emr.run_job_flow(+ Name="terminate-test",+ ReleaseLabel="emr-6.10.0",+ Instances={"MasterInstanceType": "m5.xlarge", "InstanceCount": 1, "KeepJobFlowAliveWhenNoSteps": True},+ JobFlowRole="EMR_EC2_DefaultRole",+ ServiceRole="EMR_DefaultRole",+ )+ cluster_id = jf["JobFlowId"]+ emr.terminate_job_flows(JobFlowIds=[cluster_id])+ desc = emr.describe_cluster(ClusterId=cluster_id)+ assert desc["Cluster"]["Status"]["State"] == "TERMINATED"+++def test_emr_termination_protection(emr):+ jf = emr.run_job_flow(+ Name="protected-cluster",+ ReleaseLabel="emr-6.10.0",+ Instances={"MasterInstanceType": "m5.xlarge", "InstanceCount": 1,+ "KeepJobFlowAliveWhenNoSteps": True, "TerminationProtected": True},+ JobFlowRole="EMR_EC2_DefaultRole",+ ServiceRole="EMR_DefaultRole",+ )+ cluster_id = jf["JobFlowId"]+ import botocore.exceptions+ try:+ emr.terminate_job_flows(JobFlowIds=[cluster_id])+ assert False, "should have raised"+ except botocore.exceptions.ClientError as e:+ assert "ValidationException" in str(e) or "protected" in str(e).lower()+++def test_emr_add_and_list_steps(emr):+ jf = emr.run_job_flow(+ Name="steps-cluster",+ ReleaseLabel="emr-6.10.0",+ Instances={"MasterInstanceType": "m5.xlarge", "InstanceCount": 1, "KeepJobFlowAliveWhenNoSteps": True},+ JobFlowRole="EMR_EC2_DefaultRole",+ ServiceRole="EMR_DefaultRole",+ )+ cluster_id = jf["JobFlowId"]+ step_resp = emr.add_job_flow_steps(+ JobFlowId=cluster_id,+ Steps=[{+ "Name": "my-spark-step",+ "ActionOnFailure": "CONTINUE",+ "HadoopJarStep": {+ "Jar": "command-runner.jar",+ "Args": ["spark-submit", "--class", "com.example.Main", "s3://bucket/app.jar"],+ },+ }],+ )+ assert len(step_resp["StepIds"]) == 1+ step_id = step_resp["StepIds"][0]+ assert step_id.startswith("s-")++ steps = emr.list_steps(ClusterId=cluster_id)+ assert any(s["Id"] == step_id for s in steps["Steps"])+++def test_emr_describe_step(emr):+ jf = emr.run_job_flow(+ Name="describe-step-cluster",+ ReleaseLabel="emr-6.10.0",+ Instances={"MasterInstanceType": "m5.xlarge", "InstanceCount": 1, "KeepJobFlowAliveWhenNoSteps": True},+ JobFlowRole="EMR_EC2_DefaultRole",+ ServiceRole="EMR_DefaultRole",+ )+ cluster_id = jf["JobFlowId"]+ step_resp = emr.add_job_flow_steps(+ JobFlowId=cluster_id,+ Steps=[{"Name": "step1", "ActionOnFailure": "CONTINUE",+ "HadoopJarStep": {"Jar": "command-runner.jar", "Args": []}}],+ )+ step_id = step_resp["StepIds"][0]+ desc = emr.describe_step(ClusterId=cluster_id, StepId=step_id)+ assert desc["Step"]["Id"] == step_id+ assert desc["Step"]["Status"]["State"] == "COMPLETED"+++def test_emr_tags(emr):+ jf = emr.run_job_flow(+ Name="tagged-cluster",+ ReleaseLabel="emr-6.10.0",+ Instances={"MasterInstanceType": "m5.xlarge", "InstanceCount": 1, "KeepJobFlowAliveWhenNoSteps": True},+ JobFlowRole="EMR_EC2_DefaultRole",+ ServiceRole="EMR_DefaultRole",+ Tags=[{"Key": "env", "Value": "test"}],+ )+ cluster_id = jf["JobFlowId"]+ emr.add_tags(ResourceId=cluster_id, Tags=[{"Key": "team", "Value": "data"}])+ desc = emr.describe_cluster(ClusterId=cluster_id)+ tag_map = {t["Key"]: t["Value"] for t in desc["Cluster"]["Tags"]}+ assert tag_map["env"] == "test"+ assert tag_map["team"] == "data"++ emr.remove_tags(ResourceId=cluster_id, TagKeys=["env"])+ desc2 = emr.describe_cluster(ClusterId=cluster_id)+ tag_keys = [t["Key"] for t in desc2["Cluster"]["Tags"]]+ assert "env" not in tag_keys+ assert "team" in tag_keys+++def test_emr_auto_terminate_state(emr):+ """Cluster with KeepJobFlowAliveWhenNoSteps=False starts as TERMINATED."""+ jf = emr.run_job_flow(+ Name="auto-terminate-cluster",+ ReleaseLabel="emr-6.10.0",+ Instances={"MasterInstanceType": "m5.xlarge", "InstanceCount": 1, "KeepJobFlowAliveWhenNoSteps": False},+ JobFlowRole="EMR_EC2_DefaultRole",+ ServiceRole="EMR_DefaultRole",+ )+ cluster_id = jf["JobFlowId"]+ desc = emr.describe_cluster(ClusterId=cluster_id)+ assert desc["Cluster"]["Status"]["State"] == "TERMINATED"+ assert desc["Cluster"]["AutoTerminate"] is True+++def test_emr_modify_cluster(emr):+ jf = emr.run_job_flow(+ Name="modify-cluster",+ ReleaseLabel="emr-6.10.0",+ Instances={"MasterInstanceType": "m5.xlarge", "InstanceCount": 1, "KeepJobFlowAliveWhenNoSteps": True},+ JobFlowRole="EMR_EC2_DefaultRole",+ ServiceRole="EMR_DefaultRole",+ )+ cluster_id = jf["JobFlowId"]+ resp = emr.modify_cluster(ClusterId=cluster_id, StepConcurrencyLevel=5)+ assert resp["StepConcurrencyLevel"] == 5+++def test_emr_block_public_access(emr):+ resp = emr.get_block_public_access_configuration()+ assert "BlockPublicAccessConfiguration" in resp+ assert resp["BlockPublicAccessConfiguration"]["BlockPublicSecurityGroupRules"] is False++ emr.put_block_public_access_configuration(+ BlockPublicAccessConfiguration={+ "BlockPublicSecurityGroupRules": True,+ "PermittedPublicSecurityGroupRuleRanges": [{"MinRange": 22, "MaxRange": 22}],+ }+ )+ resp2 = emr.get_block_public_access_configuration()+ assert resp2["BlockPublicAccessConfiguration"]["BlockPublicSecurityGroupRules"] is True+++def test_emr_instance_groups(emr):+ jf = emr.run_job_flow(+ Name="ig-cluster",+ ReleaseLabel="emr-6.10.0",+ Instances={+ "InstanceGroups": [+ {"Name": "Master", "InstanceRole": "MASTER", "InstanceType": "m5.xlarge", "InstanceCount": 1},+ {"Name": "Core", "InstanceRole": "CORE", "InstanceType": "m5.xlarge", "InstanceCount": 2},+ ],+ "KeepJobFlowAliveWhenNoSteps": True,+ },+ JobFlowRole="EMR_EC2_DefaultRole",+ ServiceRole="EMR_DefaultRole",+ )+ cluster_id = jf["JobFlowId"]+ groups = emr.list_instance_groups(ClusterId=cluster_id)+ assert len(groups["InstanceGroups"]) >= 2++ new_group_resp = emr.add_instance_groups(+ JobFlowId=cluster_id,+ InstanceGroups=[{"Name": "Task", "InstanceRole": "TASK", "InstanceType": "m5.xlarge", "InstanceCount": 2}],+ )+ assert len(new_group_resp["InstanceGroupIds"]) == 1+ groups2 = emr.list_instance_groups(ClusterId=cluster_id)+ assert len(groups2["InstanceGroups"]) == 3