Light, fluffy, and always free - The AWS Local Emulator alternative
EMR Serverless
EMR Serverless is an AWS service that provides serverless data analytics environments. Floci provides an emulator for the management-plane API (REST JSON) which covers the lifecycle of an Application and of the job runs submitted to it.
Configuration
| Key | Description | Default |
|---|---|---|
floci.emrserverless.enabled |
Whether the EMR Serverless API is enabled | true |
floci.emrserverless.port |
The port the service is exposed on | 4566 |
floci.services.emrserverless.job-run-duration-seconds |
How long a BATCH job run stays RUNNING before it succeeds; 0 makes it SUCCESS as soon as it starts |
0 |
floci.services.emrserverless.runner-mode |
immediate moves job runs through their states without running anything; docker runs each Spark job run in a container (see Running Spark job runs) |
immediate |
floci.services.emrserverless.docker-network |
Docker network for job run containers | the global default |
floci.services.emrserverless.image-template |
Image for a docker job run, %s standing for the release label |
public.ecr.aws/emr-serverless/spark/%s:latest, or spark4/ in place of spark/ for an emr-spark-* release |
floci.services.emrserverless.reconcile-containers-on-startup |
Remove the drivers a previous run of this Floci left on the Docker daemon before the first job run starts; turn it off when two Floci instances share a daemon with the same port and no resource namespace | true |
Endpoints
The emulator implements the standard AWS emr-serverless service endpoints:
POST /applications(CreateApplication)GET /applications(ListApplications)GET /applications/{applicationId}(GetApplication)PATCH /applications/{applicationId}(UpdateApplication)DELETE /applications/{applicationId}(DeleteApplication)POST /applications/{applicationId}/start(StartApplication)POST /applications/{applicationId}/stop(StopApplication)POST /applications/{applicationId}/jobruns(StartJobRun)GET /applications/{applicationId}/jobruns(ListJobRuns)GET /applications/{applicationId}/jobruns/{jobRunId}(GetJobRun)DELETE /applications/{applicationId}/jobruns/{jobRunId}(CancelJobRun)GET /applications/{applicationId}/jobruns/{jobRunId}/attempts(ListJobRunAttempts)POST /tags/{resourceArn}(TagResource)DELETE /tags/{resourceArn}?tagKeys=...(UntagResource)GET /tags/{resourceArn}(ListTagsForResource)
The tagging operations work on application and job run ARNs, as CreateApplication and StartJobRun
return them (arn:aws:emr-serverless:<region>:<account>:/applications/<id> and
.../applications/<id>/jobruns/<id>). They share the tags those calls store, so GetApplication and
GetJobRun always report the current set. Following the EMR Serverless user guide, a resource holds
at most 50 tags, and aws: in any case is reserved as the prefix of a tag key or value. A session ARN
(.../sessions/<id>) answers ResourceNotFoundException, as sessions are not emulated. Once an
application is deleted, its ARN and its runs' ARNs answer ResourceNotFoundException too, while
GetApplication and GetJobRun still report them.
Applications, job runs, creation client tokens, and tags are scoped to the request's AWS Region. An application or run created in one Region is not returned by list, get, or tag requests signed for another.
ListApplications refuses a states filter naming a state that does not exist, or more than 7, and a
maxResults outside 1 to 50, with the ValidationException AWS's validation layer answers.
CreateApplication stores every configuration UpdateApplication does, runtimeConfiguration,
monitoringConfiguration, schedulerConfiguration, diskEncryptionConfiguration,
interactiveConfiguration, identityCenterConfiguration and jobLevelCostAllocationConfiguration
among them, and GetApplication returns each as it was written.
DeleteApplication leaves the application in TERMINATED state. GetApplication continues to
report that state so lifecycle waiters can observe it, while an unfiltered ListApplications
omits the terminated application. A terminated application cannot be started or stopped again.
Job runs
In the default immediate runner mode, job runs are metadata only: nothing executes. A run follows
the job run states and GetJobRun's response shape, and settles when it is read. In the docker
runner mode a Spark run executes; see Running Spark job runs.
- Lifecycle: a run is
RUNNINGfrom the moment it is submitted. With the defaultjob-run-duration-secondsof0, aBATCHrun isSUCCESSas soon as it starts; a positive duration keeps itRUNNINGthat long. A run whose duration would pass itsexecutionTimeoutMinutes(720 when not given, as AWS reports it) isCANCELLEDat the timeout. ASTREAMINGrun staysRUNNINGuntil it is cancelled. - Auto-start: a run submitted to a
CREATEDorSTOPPEDapplication starts the application, asautoStartConfiguration.enableddefaults totrue. With auto-start turned off, the run is refused withValidationExceptionuntil the application is started. A run on a terminated application is refused. - Idempotency:
clientTokenis the idempotency token. Repeating a request with the same token returns the original run; reusing the token with a different request answersConflictException(HTTP 409) "Request did not match the original". - Execution timeout:
executionTimeoutMinutesis 0 to 1,000,000, 0 meaning no timeout. From releaseemr-7.11.0, aBATCHrun's timeout must be 1 to 10080 minutes (7 days), as the model and the 7.11.0 release notes say; a run outside that is refused withValidationException. - CancelJobRun moves a run that has not finished to
CANCELLED. A finished run answersValidationException"Job run is not in a cancellable state". - ListJobRuns returns the newest runs first, filtered by
states,createdAtAfter,createdAtBeforeandmode, at most 50 per page. With severalstates, the runs are grouped by state in the order the filter names them. A state or mode that does not exist, or more than 8 states, is refused withValidationException. An unknown application has no runs. AnextTokenresumes only the listing it came from, with the same application and filters. - Attempts: no run is retried, and in the
immediatemode none fails: every run is a single attempt, number 1.GetJobRun,ListJobRunsandListJobRunAttemptsreport it, with the run's own creation and update times.GetJobRunwithattempt1, or without one, returns the run; anattemptof 2 or more answersResourceNotFoundException, and one below 1 or not an integerValidationException. createdByis the caller's ARN, asGetCallerIdentityreports it.releaseLabelandnetworkConfigurationcome from the application.
A state machine drives applications and job runs through the emr-serverless: integrations, .sync
included; see Step Functions: Amazon EMR Serverless tasks.
Running Spark job runs
With runner-mode set to docker (FLOCI_SERVICES_EMRSERVERLESS_RUNNER_MODE=docker), a Spark job
run executes. Floci starts one container per run from the release's image, the base image AWS
publishes for customizing EMR Serverless images (public.ecr.aws/emr-serverless/spark/<release>, and
spark4/<release> for the AWS runtime for Apache Spark releases, emr-spark-8.0.0 on), and runs
spark-submit in local mode in it: the container holds the driver and its executors.
- States: a run is
SCHEDULEDwhile its image is pulled and its container starts (the first run of a release pulls its image, 3.5 to 4.3 GB unpacked; a pull still downloading afterfloci.docker.image-pull-timeout-seconds, five minutes by default, is tried again, three attempts in all, so on a slow connection raise it or rundocker pullon the image first, or the run isFAILEDwith "Could not pull image"), andRUNNINGonce the Spark driver runs, which setsstartedAt. It isSUCCESSwhen the driver exits with code 0 andFAILEDotherwise, with the exit code instateDetails.CancelJobRun, or aBATCHrun passing itsexecutionTimeoutMinutes, makes itCANCELLINGuntil the driver is stopped, thenCANCELLED, as the job run states describe. A run Floci was running when it stopped isFAILEDonce it is read again. Floci's shutdown and a state reset stop every driver, one whose image is still being pulled included, and a driver a hard stop left running is removed the first time job runs are used after Floci restarts, unlessreconcile-containers-on-startupis off. - What
spark-submitgets: theentryPoint, itsentryPointArgumentsand thesparkSubmitParameters, after thespark-defaultsproperties of the application'sruntimeConfiguration, or the run's own inconfigurationOverrides.applicationConfiguration, which replace the application's wholespark-defaultsclassification. Fromemr-spark-8.1.0the two are merged by property instead, the run's winning.s3://ands3a://paths reach Floci's S3 through S3A, on releases before 7.10.0 too, wheres3://would otherwise be EMRFS; the entry point and its files can be read from there. The Glue Data Catalog client is pointed at Floci's Glue. - Credentials: the container gets a session for the run's execution role when IAM knows the role,
otherwise the account's placeholder credentials, and
AWS_ENDPOINT_URLpointing at Floci, so the job's own SDK clients reach Floci too. - Output: the driver's output goes to Floci's log, prefixed
emr-serverless:<jobRunId>. With an S3 log destination, the run'ss3MonitoringConfigurationor else the application's, its standard output and error also go to Floci's S3 once the run ends, gzipped, asstdout.gzandstderr.gzunder<logUri>/applications/<applicationId>/jobs/<jobRunId>/SPARK_DRIVER/, withattempts/1/beforeSPARK_DRIVER/for aSTREAMINGrun, or aBATCHrun whoseretryPolicyallows more than one attempt, on release 7.1.0 or later. A cancelled run's logs hold what its driver wrote until it was stopped. Logs that cannot be written, to a bucket that does not exist say, leave the run as it ended. - CloudWatch Logs: with CloudWatch logging enabled, the run's
cloudWatchLoggingConfigurationor else the application's ("enabled": falseat the run turns it off), the driver's standard output and error go line by line, while it runs, to the log streams<prefix>/applications/<applicationId>/jobs/<jobRunId>/SPARK_DRIVER/stdoutand.../stderrin Floci's CloudWatch Logs, withattempts/1/beforeSPARK_DRIVERas for S3. The log group islogGroupName,/aws/emr-serverlessby default, and<prefix>thelogStreamNamePrefix, if any; both are created as needed, in the run's account and Region. AlogTypessends only the driver's types it names, and none of the driver's output when it names onlySPARK_EXECUTOR. - Image:
image-templatenames another image, such as a mirror or a dated tag of the release's image:registry.internal/emr-serverless/spark/%s:20250425.
Limitations and Differences from AWS
- Jobs do not execute by default: in the
immediaterunner modeStartJobRunrecords a job run and moves it through its states, but nothing runs. In thedockermode a Spark run executes in local mode in one container, so worker counts and sizes, initial and maximum capacity are not applied; a Hive run isFAILEDwithout running, in Floci's own wording, as is a driver's failure. Of the monitoring configuration, the S3 log destination gets the driver'sstdout.gzandstderr.gzonce a run ends (no event logs, rotated files or uploads while it runs), and CloudWatch Logs the driver's output while it runs (there are no executor logs, local mode having no executors, andencryptionKeyArnis ignored), and Floci's CA is not in the job's Java truststore, so a job reaches Floci over its HTTP endpoint only. - Unsourced errors: AWS documents neither the error for a run submitted to an application whose auto-start is off nor the one for a run on a terminated application; Floci answers both with
ValidationException. The same holds for aBATCHtimeout outside 1 to 10080 minutes onemr-7.11.0or later (ValidationException) and forGetJobRunnaming an attempt a run never made (ResourceNotFoundException), both in Floci's own wording. A run needs ajobDriver, as everyJobRuncarries one, andjobDriveris a union, so a run without a driver, or with one that sets bothsparkSubmitandhiveor neither, is refused withValidationExceptiontoo, in Floci's own wording. - StopApplication with running jobs: AWS requires running jobs to finish or be cancelled before an application stops. Floci does not enforce it; with the default duration of
0noBATCHrun is ever running. - Not emulated: retries (
retryPolicyis stored and returned, but no run is retried), queuing (QUEUED),totalResourceUtilizationandbilledResourceUtilization, andshutdownGracePeriodInSeconds(accepted and ignored: a cancelled run's driver is stopped at once). - Instant Start/Stop: Floci marks the application
STARTEDorSTOPPEDimmediately without provisioning actual compute capacity in the background. - Data Plane: Floci does not implement the data-plane API, such as interactive sessions.