Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(emr-serverless): send Spark driver output to CloudWatch Logs (#6703)

Priamo Germosen5c9eb3eb30f9
diff --git a/docs/services/emr-serverless.md b/docs/services/emr-serverless.md
index 871a097cd7..4f9029f06d 100644
--- a/docs/services/emr-serverless.md
+++ b/docs/services/emr-serverless.md
@@ -136,12 +136,19 @@ publishes for customizing EMR Serverless images (`public.ecr.aws/emr-serverless/
attempt, on release 7.1.0 or later. A cancelled run's logs hold what its driver wrote until it was
stopped. Logs that cannot be written, to a bucket that does not exist say, leave the run as it
ended.
+* **CloudWatch Logs**: with CloudWatch logging enabled, the run's `cloudWatchLoggingConfiguration` or
+ else the application's (`"enabled": false` at the run turns it off), the driver's standard output and
+ error go line by line, while it runs, to the log streams `<prefix>/applications/<applicationId>/jobs/<jobRunId>/SPARK_DRIVER/stdout`
+ and `.../stderr` in Floci's CloudWatch Logs, with `attempts/1/` before `SPARK_DRIVER` as for S3. The log
+ group is `logGroupName`, `/aws/emr-serverless` by default, and `<prefix>` the `logStreamNamePrefix`, if
+ any; both are created as needed, in the run's account and Region. A `logTypes` sends only the driver's
+ types it names, and none of the driver's output when it names only `SPARK_EXECUTOR`.
* **Image**: `image-template` names another image, such as a mirror or a dated tag of the release's
image: `registry.internal/emr-serverless/spark/%s:20250425`.
## Limitations and Differences from AWS
-* **Jobs do not execute by default**: in the `immediate` runner mode `StartJobRun` records a job run and moves it through its states, but nothing runs. In the `docker` mode a Spark run executes in local mode in one container, so worker counts and sizes, initial and maximum capacity are not applied; a Hive run is `FAILED` without running, in Floci's own wording, as is a driver's failure. Of the monitoring configuration only the S3 log destination is written to, with the driver's `stdout.gz` and `stderr.gz` once a run ends (no event logs, rotated files or uploads while it runs), and Floci's CA is not in the job's Java truststore, so a job reaches Floci over its HTTP endpoint only.
+* **Jobs do not execute by default**: in the `immediate` runner mode `StartJobRun` records a job run and moves it through its states, but nothing runs. In the `docker` mode a Spark run executes in local mode in one container, so worker counts and sizes, initial and maximum capacity are not applied; a Hive run is `FAILED` without running, in Floci's own wording, as is a driver's failure. Of the monitoring configuration, the S3 log destination gets the driver's `stdout.gz` and `stderr.gz` once a run ends (no event logs, rotated files or uploads while it runs), and CloudWatch Logs the driver's output while it runs (there are no executor logs, local mode having no executors, and `encryptionKeyArn` is ignored), and Floci's CA is not in the job's Java truststore, so a job reaches Floci over its HTTP endpoint only.
* **Unsourced errors**: AWS documents neither the error for a run submitted to an application whose auto-start is off nor the one for a run on a terminated application; Floci answers both with `ValidationException`. The same holds for a `BATCH` timeout outside 1 to 10080 minutes on `emr-7.11.0` or later (`ValidationException`) and for `GetJobRun` naming an attempt a run never made (`ResourceNotFoundException`), both in Floci's own wording. A run needs a `jobDriver`, as every `JobRun` carries one, and `jobDriver` is a union, so a run without a driver, or with one that sets both `sparkSubmit` and `hive` or neither, is refused with `ValidationException` too, in Floci's own wording.
* **StopApplication with running jobs**: AWS requires running jobs to finish or be cancelled before an application stops. Floci does not enforce it; with the default duration of `0` no `BATCH` run is ever running.
* **Not emulated**: retries (`retryPolicy` is stored and returned, but no run is retried), queuing (`QUEUED`), `totalResourceUtilization` and `billedResourceUtilization`, and `shutdownGracePeriodInSeconds` (accepted and ignored: a cancelled run's driver is stopped at once).
diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLogStreamer.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLogStreamer.java
index bc74678392..619cf23b33 100644
--- a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLogStreamer.java
+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLogStreamer.java
@@ -33,6 +33,7 @@ import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.ScheduledThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.BiConsumer;
import java.util.function.Consumer;
/**
@@ -120,6 +121,34 @@ public class ContainerLogStreamer {
return follow(containerId, since, emitter);
}
+ /**
+ * Like {@link #attachForAccount}, but forwards the container's standard output and standard error to
+ * log streams of their own in {@code logGroup}, {@code stdoutStream} and {@code stderrStream}, either
+ * of which may be null to forward that output nowhere. A container with a terminal has no standard
+ * error of its own: all of its output is standard output. Every line still goes to the console logger.
+ */
+ public Closeable attachByStreamForAccount(
+ String accountId, String containerId, String logGroup, String stdoutStream, String stderrStream,
+ String region, String logPrefix) {
+ for (String logStream : new String[] {stdoutStream, stderrStream}) {
+ if (logStream != null) {
+ ensureLogGroupAndStreamForAccount(accountId, logGroup, logStream, region);
+ }
+ }
+ BiConsumer<StreamType, String> emitter = (stream, line) -> {
+ String trimmed = line.stripTrailing();
+ if (!trimmed.isEmpty()) {
+ LOG.infov("[{0}] {1}", logPrefix, trimmed);
+ String logStream = stream == StreamType.STDERR ? stderrStream : stdoutStream;
+ if (logStream != null) {
+ forwardToCloudWatchLogs(accountId, logGroup, logStream, region, trimmed);
+ }
+ }
+ };
+ return follow(containerId, null,
+ new LogReassemblyCallback(LogReassemblyCallback.DEFAULT_CLOSE_GRACE_MILLIS, emitter));
+ }
+
/**
* Like {@link #attach}, but sends the output only to the console logger, not to CloudWatch Logs.
* Use it for a container whose output must stay visible locally but has no CloudWatch
@@ -136,8 +165,11 @@ public class ContainerLogStreamer {
}
private Closeable follow(String containerId, Instant since, Consumer<String> emitter) {
+ return follow(containerId, since, new LogReassemblyCallback(emitter));
+ }
+
+ private Closeable follow(String containerId, Instant since, LogReassemblyCallback callback) {
try {
- LogReassemblyCallback callback = new LogReassemblyCallback(emitter);
LogContainerCmd command = dockerClient.logContainerCmd(containerId)
.withStdOut(true)
.withStdErr(true)
@@ -279,9 +311,9 @@ public class ContainerLogStreamer {
/**
* Docker frame callback that reassembles multiplexed stdout/stderr frames into newline-delimited
- * lines and forwards each to {@code lineConsumer}. One logical line can span several frames and one
- * frame can carry several lines, so bytes are buffered per stream and emitted one line at a time
- * (matching real AWS Lambda).
+ * lines and forwards each, with the stream it came from, to {@code lineConsumer}. One logical line can
+ * span several frames and one frame can carry several lines, so bytes are buffered per stream and
+ * emitted one line at a time (matching real AWS Lambda).
*/
static final class LogReassemblyCallback extends ResultCallback.Adapter<Frame> {
@@ -304,12 +336,12 @@ public class ContainerLogStreamer {
private final Map<StreamType, LogLineBuffer> buffers = new EnumMap<>(StreamType.class);
private final Map<StreamType, Long> tailSequences = new EnumMap<>(StreamType.class);
- private final Deque<String> pendingLines = new ArrayDeque<>();
+ private final Deque<PendingLine> pendingLines = new ArrayDeque<>();
private final Object emissionLock = new Object();
// A fair write lock makes fallback finalization wait for callbacks already in progress, then blocks
// any later callback until the transport has been closed and the final tail has been flushed.
private final ReentrantReadWriteLock callbackGate = new ReentrantReadWriteLock(true);
- private final Consumer<String> lineConsumer;
+ private final BiConsumer<StreamType, String> lineConsumer;
private final long closeGraceMillis;
// Guarded by 'buffers'. Set only by the Docker transport terminal callback; afterwards onNext
// refuses to append, so nothing is ever buffered past the final flush.
@@ -327,6 +359,10 @@ public class ContainerLogStreamer {
}
LogReassemblyCallback(Consumer<String> lineConsumer, long closeGraceMillis) {
+ this(closeGraceMillis, (stream, line) -> lineConsumer.accept(line));
+ }
+
+ LogReassemblyCallback(long closeGraceMillis, BiConsumer<StreamType, String> lineConsumer) {
this.lineConsumer = lineConsumer;
this.closeGraceMillis = closeGraceMillis;
}
@@ -342,7 +378,9 @@ public class ContainerLogStreamer {
return; // the final drain has run; never append into a buffer that won't be flushed
}
LogLineBuffer buffer = buffers.computeIfAbsent(frame.getStreamType(), t -> new LogLineBuffer());
- pendingLines.addAll(buffer.append(frame.getPayload()));
+ for (String line : buffer.append(frame.getPayload())) {
+ pendingLines.addLast(new PendingLine(frame.getStreamType(), line));
+ }
if (buffer.hasPendingBytes()) {
tailSequences.put(frame.getStreamType(), nextTailSequence++);
} else {
@@ -411,7 +449,8 @@ public class ContainerLogStreamer {
buffers.entrySet().stream()
.sorted(Comparator.comparingLong(entry -> tailSequences.getOrDefault(
entry.getKey(), Long.MAX_VALUE)))
- .forEach(entry -> entry.getValue().flush().ifPresent(pendingLines::addLast));
+ .forEach(entry -> entry.getValue().flush()
+ .ifPresent(line -> pendingLines.addLast(new PendingLine(entry.getKey(), line))));
tailSequences.clear();
}
emit = startEmissionIfNeeded();
@@ -458,19 +497,23 @@ public class ContainerLogStreamer {
private void emitPendingLines() {
while (true) {
- String line;
+ PendingLine pending;
synchronized (emissionLock) {
synchronized (buffers) {
- line = pendingLines.pollFirst();
- if (line == null) {
+ pending = pendingLines.pollFirst();
+ if (pending == null) {
emitting = false;
return;
}
}
}
- lineConsumer.accept(line);
+ lineConsumer.accept(pending.stream(), pending.line());
}
}
+
+ /** A reassembled line waiting to be emitted, and the stream it came from. */
+ private record PendingLine(StreamType stream, String line) {
+ }
}
/**
diff --git a/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java b/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java
index 2df3636824..773cc021a6 100644
--- a/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java
@@ -1214,7 +1214,7 @@ public class EmrService implements Resettable {
accountId, region, cluster.getReleaseLabel(), imageTemplate, dockerNetwork,
executionRole(cluster, step, accountId),
EmrSparkSteps.sparkDefaults(cluster.getConfigurations(), SNAPSHOT_MAPPER), application,
- Duration.ZERO, EmrSparkSteps.logFolder(cluster, step));
+ Duration.ZERO, EmrSparkSteps.logFolder(cluster, step), null);
StepLaunch launch = new StepLaunch(accountId, region, cluster.getId(), step.getId(), generation);
executing.add(stepKey(accountId, step.getId()));
stepExecutor.execute(() -> runStep(job, launch));
diff --git a/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunService.java b/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunService.java
index a1203183d3..bc48304773 100644
--- a/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunService.java
@@ -85,6 +85,10 @@ public class EmrServerlessJobRunService implements Resettable {
private static final String SERVICE = "emr-serverless";
private static final String SPARK_DEFAULTS = "spark-defaults";
private static final String S3_MONITORING = "s3MonitoringConfiguration";
+ private static final String CLOUD_WATCH_LOGGING = "cloudWatchLoggingConfiguration";
+ // "If you don't specify the optional values, then CloudWatch publishes the logs to a default log group".
+ private static final String DEFAULT_LOG_GROUP = "/aws/emr-serverless";
+ private static final String SPARK_DRIVER = "SPARK_DRIVER";
// No source gives AWS's wording for these; they are Floci's.
static final String HIVE_NOT_RUN = "Floci runs Spark job runs only, and this is a Hive job run.";
static final String INTERRUPTED = "Floci stopped while the job run was in progress.";
@@ -460,7 +464,8 @@ public class EmrServerlessJobRunService implements Resettable {
? Duration.ZERO : Duration.ofMinutes(minutes);
SparkJobRunner.Job job = new SparkJobRunner.Job(SERVICE, run.getJobRunId(), arn.accountId(), arn.region(),
run.getReleaseLabel(), imageTemplate, dockerNetwork, run.getExecutionRole(), sparkDefaults(app, run),
- application(run.getJobDriver().getSparkSubmit()), timeout, driverLogFolder(app, run));
+ application(run.getJobDriver().getSparkSubmit()), timeout, driverLogFolder(app, run),
+ driverCloudWatchLogs(app, run));
executing.add(run.getJobRunId());
long launchedIn = generation;
runExecutor.execute(() -> execute(job, launchedIn));
@@ -613,28 +618,73 @@ public class EmrServerlessJobRunService implements Resettable {
}
/**
- * The S3 folder a run's driver logs go to, or null without an S3 log destination. The run's
- * {@code s3MonitoringConfiguration} supersedes the application's ("MonitoringConfiguration -
- * Configuration type"), and an empty one removes it. Under its {@code logUri} the user guide puts a
- * run's logs at {@code /applications/<applicationId>/jobs/<jobId>}, with
- * {@code /attempts/<attemptNumber>} after it for "all streaming jobs and any batch jobs that have retry
+ * The S3 folder a run's driver logs go to, or null without an S3 log destination: the run's
+ * {@code s3MonitoringConfiguration} or else the application's, an empty one at the run removing it. Under
+ * its {@code logUri} the user guide puts a run's logs at {@code /applications/<applicationId>/jobs/<jobId>},
+ * with {@code /attempts/<attemptNumber>} after it for "all streaming jobs and any batch jobs that have retry
* enabled" from release 7.1.0, and the driver's in {@code SPARK_DRIVER/}.
*/
static String driverLogFolder(Application app, JobRun run) {
+ String logUri = monitoring(app, run, S3_MONITORING) instanceof Map<?, ?> configuration
+ && configuration.get("logUri") instanceof String uri ? uri : null;
+ return EmrServerlessSparkRunner.logFolder(logUri, driverLogPath(run).substring(1) + "/");
+ }
+
+ /**
+ * Where in CloudWatch Logs a run's driver output goes, or null when CloudWatch logging is not enabled.
+ * The run's {@code cloudWatchLoggingConfiguration} supersedes the application's, and one with
+ * {@code enabled} false removes it. The user guide's defaults: the log group {@code /aws/emr-serverless}
+ * and the log stream {@code /applications/applicationId/jobs/jobId/worker-type}, after the
+ * {@code logStreamNamePrefix}, with the attempt as for S3, and "By default, EMR Serverless publishes only
+ * the driver stdout and stderr logs". Each goes to a stream of its own under the driver's, as the
+ * clients that read them (Dagster's EMR Serverless client) name them: {@code .../SPARK_DRIVER/stdout}
+ * and {@code .../SPARK_DRIVER/stderr}. A {@code logTypes} that names some types takes only those of the
+ * driver's it names, none when it names only other workers': "The types of logs that you want to publish
+ * to CloudWatch. If you don't specify any log types, driver STDOUT and STDERR logs will be published".
+ */
+ static SparkJobRunner.CloudWatchLogs driverCloudWatchLogs(Application app, JobRun run) {
+ if (!(monitoring(app, run, CLOUD_WATCH_LOGGING) instanceof Map<?, ?> configuration)
+ || !Boolean.TRUE.equals(configuration.get("enabled"))) {
+ return null;
+ }
+ String logGroup = configuration.get("logGroupName") instanceof String name ? name : DEFAULT_LOG_GROUP;
+ String prefix = configuration.get("logStreamNamePrefix") instanceof String named ? named : "";
+ String driver = prefix + driverLogPath(run) + "/";
+ List<?> driverTypes = null;
+ if (configuration.get("logTypes") instanceof Map<?, ?> logTypes && !logTypes.isEmpty()) {
+ driverTypes = logTypes.get(SPARK_DRIVER) instanceof List<?> types ? types : List.of();
+ }
+ String stdout = publishes(driverTypes, "stdout") ? driver + "stdout" : null;
+ String stderr = publishes(driverTypes, "stderr") ? driver + "stderr" : null;
+ return stdout == null && stderr == null ? null : new SparkJobRunner.CloudWatchLogs(logGroup, stdout, stderr);
+ }
+
+ // Whether the driver's log types, as logTypes lists them for SPARK_DRIVER, take this one; both without a list.
+ private static boolean publishes(List<?> driverTypes, String logType) {
+ return driverTypes == null
+ || driverTypes.stream().anyMatch(type -> logType.equalsIgnoreCase(String.valueOf(type)));
+ }
+
+ /**
+ * One of a run's monitoring configurations, {@code type}: the run's, which supersedes the application's
+ * ("The granularity level when the merge occurs ... MonitoringConfiguration - Configuration type"), or
+ * else the application's; null when neither has one.
+ */
+ private static Object monitoring(Application app, JobRun run, String type) {
Object overrides = run.getConfigurationOverrides() != null
? run.getConfigurationOverrides().get("monitoringConfiguration") : null;
- Object s3;
- if (overrides instanceof Map<?, ?> monitoring && monitoring.containsKey(S3_MONITORING)) {
- s3 = monitoring.get(S3_MONITORING);
- } else {
- s3 = app.getMonitoringConfiguration() != null
- ? app.getMonitoringConfiguration().getProperties().get(S3_MONITORING) : null;
- }
- String logUri = s3 instanceof Map<?, ?> configuration && configuration.get("logUri") instanceof String uri
- ? uri : null;
- String attempts = retried(run) && retriesAttempts(run.getReleaseLabel()) ? "attempts/" + ATTEMPT + "/" : "";
- return EmrServerlessSparkRunner.logFolder(logUri, "applications/" + run.getApplicationId() + "/jobs/"
- + run.getJobRunId() + "/" + attempts + "SPARK_DRIVER/");
+ if (overrides instanceof Map<?, ?> monitoring && monitoring.containsKey(type)) {
+ return monitoring.get(type);
+ }
+ return app.getMonitoringConfiguration() != null
+ ? app.getMonitoringConfiguration().getProperties().get(type) : null;
+ }
+
+ // Where under a log destination the user guide puts a run's driver logs, the attempt for a retried run.
+ private static String driverLogPath(JobRun run) {
+ String attempts = retried(run) && retriesAttempts(run.getReleaseLabel()) ? "/attempts/" + ATTEMPT : "";
+ return "/applications/" + run.getApplicationId() + "/jobs/" + run.getJobRunId() + attempts + "/"
+ + SPARK_DRIVER;
}
// A STREAMING run is retried, and a BATCH run whose retryPolicy allows more than one attempt.
diff --git a/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunner.java b/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunner.java
index 3326548c86..6948aa9cf0 100644
--- a/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunner.java
+++ b/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunner.java
@@ -45,7 +45,8 @@ import java.util.zip.GZIPOutputStream;
* supplies at run time, which the public image lacks, so the runner starts {@code spark-submit} itself,
* in local mode: the container holds the driver and its executors. The container reaches Floci as the
* other containers Floci launches do, with the run's execution role as its credentials, and S3A, which
- * takes {@code s3://} on every release, is pointed at Floci's S3. Once the driver ends, its standard output
+ * takes {@code s3://} on every release, is pointed at Floci's S3. The driver's output goes to Floci's log and,
+ * while it runs, to the run's CloudWatch log streams, when it has some; once it ends, its standard output
* and error go, gzipped, to the run's log folder in Floci's S3, when it has one.
*/
@ApplicationScoped
@@ -141,7 +142,7 @@ public class EmrServerlessSparkRunner implements SparkJobRunner, ContainerTeardo
if (launch.stopRequested || progress.cancelled()) {
return Outcome.halted();
}
- logHandle = logStreamer.attachConsoleOnly(containerId, job.service() + ":" + job.runId());
+ logHandle = attachLog(containerId, job);
progress.started();
Outcome outcome = awaitEnd(containerId, job, launch, progress);
writeLogs(job, launch);
@@ -228,6 +229,17 @@ public class EmrServerlessSparkRunner implements SparkJobRunner, ContainerTeardo
}
}
+ // Floci's log always gets the driver's output, and the job's CloudWatch log streams too when it has some.
+ private Closeable attachLog(String containerId, Job job) {
+ String prefix = job.service() + ":" + job.runId();
+ CloudWatchLogs cloudWatch = job.cloudWatchLogs();
+ if (cloudWatch == null) {
+ return logStreamer.attachConsoleOnly(containerId, prefix);
+ }
+ return logStreamer.attachByStreamForAccount(job.accountId(), containerId, cloudWatch.logGroup(),
+ cloudWatch.stdoutStream(), cloudWatch.stderrStream(), job.region(), prefix);
+ }
+
/**
* The folder {@code path} names under {@code logUri}, the S3 location a service writes its logs to, or
* null when there is none: {@code path} follows {@code logUri} after one slash, whether or not
diff --git a/src/main/java/io/github/hectorvent/floci/services/emrserverless/SparkJobRunner.java b/src/main/java/io/github/hectorvent/floci/services/emrserverless/SparkJobRunner.java
index c355169353..2677bea5a2 100644
--- a/src/main/java/io/github/hectorvent/floci/services/emrserverless/SparkJobRunner.java
+++ b/src/main/java/io/github/hectorvent/floci/services/emrserverless/SparkJobRunner.java
@@ -54,12 +54,20 @@ public interface SparkJobRunner {
* whose credentials it gets; the {@code spark-defaults} properties that apply to it;
* {@code application}, what follows Floci's own settings on {@code spark-submit}'s command line (the
* run's options, then its application and that application's arguments); how long it may run
- * ({@link Duration#ZERO} for no limit); and the S3 folder its driver's {@code stdout.gz} and
- * {@code stderr.gz} go to once it ends, ending in a slash (null for none).
+ * ({@link Duration#ZERO} for no limit); the S3 folder its driver's {@code stdout.gz} and
+ * {@code stderr.gz} go to once it ends, ending in a slash (null for none); and where in CloudWatch Logs
+ * its driver's output goes while it runs (null for nowhere but Floci's log).
*/
record Job(String service, String runId, String accountId, String region, String releaseLabel,
String imageTemplate, String dockerNetwork, String executionRole, Map<String, String> sparkDefaults,
- List<String> application, Duration timeout, String logFolder) {
+ List<String> application, Duration timeout, String logFolder, CloudWatchLogs cloudWatchLogs) {
+ }
+
+ /**
+ * A log group in the run's account and Region, and the log streams the driver's standard output and
+ * standard error go to in it, either null for that output to go nowhere but Floci's log.
+ */
+ record CloudWatchLogs(String logGroup, String stdoutStream, String stderrStream) {
}
/**
diff --git a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLogStreamerTest.java b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLogStreamerTest.java
index 9a6fda50ec..ae01cf28a1 100644
--- a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLogStreamerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLogStreamerTest.java
@@ -130,6 +130,61 @@ class ContainerLogStreamerTest {
verify(cloudWatchLogsService, never()).putLogEventsForAccount(any(), any(), any(), anyList(), any());
}
+ @Test
+ void aByStreamAttachSendsEachStreamToItsOwnLogStream() {
+ DockerClient dockerClient = mock(DockerClient.class);
+ LogContainerCmd command = mock(LogContainerCmd.class, RETURNS_SELF);
+ when(dockerClient.logContainerCmd("container-1")).thenReturn(command);
+ CloudWatchLogsService cloudWatchLogsService = mock(CloudWatchLogsService.class);
+ ContainerLogStreamer streamer = new ContainerLogStreamer(dockerClient, cloudWatchLogsService);
+
+ streamer.attachByStreamForAccount("555555555555", "container-1", "/aws/emr-serverless",
+ "/jobs/1/SPARK_DRIVER/stdout", "/jobs/1/SPARK_DRIVER/stderr", "us-east-1", "emr-serverless:1");
+ ArgumentCaptor<ContainerLogStreamer.LogReassemblyCallback> callback =
+ ArgumentCaptor.forClass(ContainerLogStreamer.LogReassemblyCallback.class);
+ verify(command).exec(callback.capture());
+
+ List<LogRecord> records = LogCapture.capture(ContainerLogStreamer.class, () -> {
+ callback.getValue().onNext(new Frame(StreamType.STDOUT, utf8("Pi is roughly 3.14\n")));
+ callback.getValue().onNext(new Frame(StreamType.STDERR, utf8("INFO SparkContext: stopped\n")));
+ });
+
+ assertEquals(2, records.size());
+ verify(cloudWatchLogsService).createLogStreamForAccount(
+ "555555555555", "/aws/emr-serverless", "/jobs/1/SPARK_DRIVER/stdout", "us-east-1");
+ verify(cloudWatchLogsService).createLogStreamForAccount(
+ "555555555555", "/aws/emr-serverless", "/jobs/1/SPARK_DRIVER/stderr", "us-east-1");
+ verify(cloudWatchLogsService).putLogEventsForAccount(eq("555555555555"), eq("/aws/emr-serverless"),
+ eq("/jobs/1/SPARK_DRIVER/stdout"),
+ argThat(events -> "Pi is roughly 3.14".equals(events.getFirst().get("message"))), eq("us-east-1"));
+ verify(cloudWatchLogsService).putLogEventsForAccount(eq("555555555555"), eq("/aws/emr-serverless"),
+ eq("/jobs/1/SPARK_DRIVER/stderr"),
+ argThat(events -> "INFO SparkContext: stopped".equals(events.getFirst().get("message"))),
+ eq("us-east-1"));
+ }
+
+ @Test
+ void aByStreamAttachForwardsAStreamWithoutALogStreamOnlyToTheConsole() {
+ DockerClient dockerClient = mock(DockerClient.class);
+ LogContainerCmd command = mock(LogContainerCmd.class, RETURNS_SELF);
+ when(dockerClient.logContainerCmd("container-1")).thenReturn(command);
+ CloudWatchLogsService cloudWatchLogsService = mock(CloudWatchLogsService.class);
+ ContainerLogStreamer streamer = new ContainerLogStreamer(dockerClient, cloudWatchLogsService);
+
+ streamer.attachByStreamForAccount("555555555555", "container-1", "/aws/emr-serverless",
+ "/jobs/1/SPARK_DRIVER/stdout", null, "us-east-1", "emr-serverless:1");
+ ArgumentCaptor<ContainerLogStreamer.LogReassemblyCallback> callback =
+ ArgumentCaptor.forClass(ContainerLogStreamer.LogReassemblyCallback.class);
+ verify(command).exec(callback.capture());
+
+ List<LogRecord> records = LogCapture.capture(ContainerLogStreamer.class, () ->
+ callback.getValue().onNext(new Frame(StreamType.STDERR, utf8("INFO SparkContext: stopped\n"))));
+
+ assertEquals(1, records.size());
+ verify(cloudWatchLogsService, never()).createLogStreamForAccount(any(), any(), eq(null), any());
+ verify(cloudWatchLogsService, never()).putLogEventsForAccount(any(), any(), any(), anyList(), any());
+ }
+
@ParameterizedTest
@ValueSource(booleans = {true, false})
void execStreamAlwaysReachesCloudWatchAndConsoleIsOptional(boolean logToConsole) {
@@ -511,6 +566,21 @@ class ContainerLogStreamerTest {
assertEquals(List.of("no-newline-tail"), emitted);
}
+ @Test
+ void reassembledLinesKeepTheStreamTheyCameFrom() {
+ List<String> emitted = new ArrayList<>();
+ ContainerLogStreamer.LogReassemblyCallback cb = new ContainerLogStreamer.LogReassemblyCallback(
+ 1_000, (stream, line) -> emitted.add(stream + ": " + line));
+
+ cb.onNext(new Frame(StreamType.STDOUT, utf8("Pi is ")));
+ cb.onNext(new Frame(StreamType.STDERR, utf8("INFO starting\n")));
+ cb.onNext(new Frame(StreamType.STDOUT, utf8("roughly 3.14\n")));
+ cb.onNext(new Frame(StreamType.STDERR, utf8("INFO stopp")));
+ cb.onComplete();
+
+ assertEquals(List.of("STDERR: INFO starting", "STDOUT: Pi is roughly 3.14", "STDERR: INFO stopp"), emitted);
+ }
+
@Test
void flushesStreamTailsInTheirLastFrameArrivalOrder() {
List<String> emitted = new ArrayList<>();
diff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceContainerStepsTest.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceContainerStepsTest.java
index f9d4f5684d..552a0581f0 100644
--- a/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceContainerStepsTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceContainerStepsTest.java
@@ -82,6 +82,7 @@ class EmrServiceContainerStepsTest {
assertEquals(Map.of("spark.executor.cores", "2"), job.sparkDefaults());
assertEquals(Duration.ZERO, job.timeout());
assertNull(job.logFolder());
+ assertNull(job.cloudWatchLogs());
}
@Test
diff --git a/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunServiceTest.java
index bd5ffd05a1..c7afd6332e 100644
--- a/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunServiceTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunServiceTest.java
@@ -264,6 +264,79 @@ class EmrServerlessJobRunServiceTest {
assertEquals(Duration.ofMinutes(30), job.timeout());
assertEquals(Duration.ZERO, runner.jobs.get(1).timeout());
assertNull(job.logFolder());
+ assertNull(job.cloudWatchLogs());
+ }
+
+ @Test
+ void aRunsDriverOutputGoesToCloudWatchWhenItsOwnOrTheApplicationsLoggingIsEnabled() {
+ FakeRunner runner = new FakeRunner();
+ EmrServerlessJobRunService service = dockerService(runner);
+ CreateApplicationRequest create = new CreateApplicationRequest();
+ create.setName("watched");
+ create.setReleaseLabel("emr-7.5.0");
+ create.setType("SPARK");
+ create.setClientToken(UUID.randomUUID().toString());
+ create.setMonitoringConfiguration(cloudWatchLogging(Map.of("enabled", true)));
+ String watched = applications.createApplication(create).getApplicationId();
+ StartJobRunRequest own = request(null, null);
+ own.setConfigurationOverrides(Map.of("monitoringConfiguration", Map.of("cloudWatchLoggingConfiguration",
+ Map.of("enabled", true, "logGroupName", "spark-jobs", "logStreamNamePrefix", "nightly",
+ "logTypes", Map.of("SPARK_DRIVER", List.of("STDERR"))))));
+ // "To remove cloudWatchLogging, pass the enabled flag as false."
+ StartJobRunRequest removed = request(null, null);
+ removed.setConfigurationOverrides(Map.of("monitoringConfiguration",
+ Map.of("cloudWatchLoggingConfiguration", Map.of("enabled", false))));
+
+ String inherited = service.startJobRun(watched, request(null, null)).getJobRunId();
+ String overridden = service.startJobRun(watched, own).getJobRunId();
+ service.startJobRun(watched, removed);
+ for (int i = 0; i < 3; i++) {
+ runNext();
+ }
+
+ String driver = "/applications/" + watched + "/jobs/" + inherited + "/SPARK_DRIVER/";
+ assertEquals(new SparkJobRunner.CloudWatchLogs("/aws/emr-serverless", driver + "stdout", driver + "stderr"),
+ runner.jobs.get(0).cloudWatchLogs());
+ assertEquals(new SparkJobRunner.CloudWatchLogs("spark-jobs", null,
+ "nightly/applications/" + watched + "/jobs/" + overridden + "/SPARK_DRIVER/stderr"),
+ runner.jobs.get(1).cloudWatchLogs());
+ assertNull(runner.jobs.get(2).cloudWatchLogs());
+ }
+
+ @Test
+ void aRetriedRunsCloudWatchLogStreamsAreUnderItsAttempt() {
+ Application app = new Application();
+ app.setMonitoringConfiguration(cloudWatchLogging(Map.of("enabled", true)));
+
+ assertEquals("/applications/app/jobs/run/attempts/1/SPARK_DRIVER/stdout",
+ EmrServerlessJobRunService.driverCloudWatchLogs(app, run("emr-7.1.0", "STREAMING", null))
+ .stdoutStream());
+ assertEquals("/applications/app/jobs/run/SPARK_DRIVER/stdout",
+ EmrServerlessJobRunService.driverCloudWatchLogs(app, run("emr-7.0.0", "STREAMING", null))
+ .stdoutStream());
+ }
+
+ @Test
+ void logTypesThatNameOnlyTheExecutorsSendNoneOfTheDriversOutput() {
+ Application app = new Application();
+ // "If you don't specify any log types, driver STDOUT and STDERR logs will be published to CloudWatch Logs
+ // by default."
+ app.setMonitoringConfiguration(cloudWatchLogging(Map.of("enabled", true,
+ "logTypes", Map.of("SPARK_EXECUTOR", List.of("STDOUT", "STDERR")))));
+
+ assertNull(EmrServerlessJobRunService.driverCloudWatchLogs(app, run("emr-7.5.0", "BATCH", null)));
+
+ app.setMonitoringConfiguration(cloudWatchLogging(Map.of("enabled", true, "logTypes", Map.of())));
+
+ assertEquals(new SparkJobRunner.CloudWatchLogs("/aws/emr-serverless",
+ "/applications/app/jobs/run/SPARK_DRIVER/stdout", "/applications/app/jobs/run/SPARK_DRIVER/stderr"),
+ EmrServerlessJobRunService.driverCloudWatchLogs(app, run("emr-7.5.0", "BATCH", null)));
+ }
+
+ private static MonitoringConfiguration cloudWatchLogging(Map<String, Object> configuration) {
+ MonitoringConfiguration monitoring = new MonitoringConfiguration();
+ monitoring.setProperty("cloudWatchLoggingConfiguration", configuration);
+ return monitoring;
}
@Test
diff --git a/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunnerTest.java b/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunnerTest.java
index 6549a28722..50ef0de074 100644
--- a/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunnerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunnerTest.java
@@ -60,6 +60,7 @@ class EmrServerlessSparkRunnerTest {
private final DockerClient docker = mock(DockerClient.class, RETURNS_DEEP_STUBS);
private final SparkJobRunner.Progress progress = mock(SparkJobRunner.Progress.class);
private final S3Service s3 = mock(S3Service.class);
+ private final ContainerLogStreamer logStreamer = mock(ContainerLogStreamer.class);
@Test
void sparkSubmitRunsLocallyOnFlocisS3ThenTheRunsOwnSettings() {
@@ -354,6 +355,27 @@ class EmrServerlessSparkRunnerTest {
verify(s3, never()).putObject(anyString(), anyString(), any(), anyString(), anyMap());
}
+ @Test
+ void theDriversOutputGoesToTheJobsCloudWatchLogStreamsWhileItRuns() {
+ EmrServerlessSparkRunner runner = runner();
+ String driver = "/applications/00f0000000000000/jobs/00f0000000000001/SPARK_DRIVER/";
+
+ runner.run(job(new SparkJobRunner.CloudWatchLogs("/aws/emr-serverless", driver + "stdout", driver + "stderr")),
+ progress);
+
+ verify(logStreamer).attachByStreamForAccount("123456789012", "driver", "/aws/emr-serverless",
+ driver + "stdout", driver + "stderr", "eu-west-1", "emr-serverless:00f0000000000001");
+ verify(logStreamer, never()).attachConsoleOnly(anyString(), anyString());
+ }
+
+ @Test
+ void withoutCloudWatchLogStreamsTheDriversOutputGoesOnlyToFlocisLog() {
+ runner().run(job(Duration.ZERO), progress);
+
+ verify(logStreamer).attachConsoleOnly("driver", "emr-serverless:00f0000000000001");
+ verify(logStreamer, never()).attachByStreamForAccount(any(), any(), any(), any(), any(), any(), any());
+ }
+
@Test
void aRunWithoutALogFolderReadsNoOutput() {
runner().run(job(Duration.ZERO), progress);
@@ -396,7 +418,7 @@ class EmrServerlessSparkRunnerTest {
when(lifecycle.createAndStart(any())).thenReturn(new ContainerLifecycleManager.ContainerInfo("driver", Map.of()));
when(docker.inspectContainerCmd("driver").exec().getState().getRunning()).thenReturn(false);
when(docker.inspectContainerCmd("driver").exec().getState().getExitCodeLong()).thenReturn(0L);
- return new EmrServerlessSparkRunner(builder, lifecycle, mock(ContainerLogStreamer.class),
+ return new EmrServerlessSparkRunner(builder, lifecycle, logStreamer,
mock(LaunchedContainerAwsEnv.class), mock(LambdaExecutionRoleCredentials.class), s3, config,
new MutableClock(), Duration.ZERO);
}
@@ -440,7 +462,11 @@ class EmrServerlessSparkRunnerTest {
}
private static SparkJobRunner.Job job(String logFolder) {
- return job(null, List.of("s3://bucket/jobs/app.jar"), Map.of(), Duration.ZERO, logFolder);
+ return job(null, List.of("s3://bucket/jobs/app.jar"), Map.of(), Duration.ZERO, logFolder, null);
+ }
+
+ private static SparkJobRunner.Job job(SparkJobRunner.CloudWatchLogs cloudWatchLogs) {
+ return job(null, List.of("s3://bucket/jobs/app.jar"), Map.of(), Duration.ZERO, null, cloudWatchLogs);
}
private static SparkJobRunner.Job job(List<String> application, Map<String, String> sparkDefaults) {
@@ -449,18 +475,19 @@ class EmrServerlessSparkRunnerTest {
private static SparkJobRunner.Job job(List<String> application, Map<String, String> sparkDefaults,
Duration timeout) {
- return job(null, application, sparkDefaults, timeout, null);
+ return job(null, application, sparkDefaults, timeout, null, null);
}
private static SparkJobRunner.Job job(String imageTemplate, List<String> application,
Map<String, String> sparkDefaults, Duration timeout) {
- return job(imageTemplate, application, sparkDefaults, timeout, null);
+ return job(imageTemplate, application, sparkDefaults, timeout, null, null);
}
private static SparkJobRunner.Job job(String imageTemplate, List<String> application,
- Map<String, String> sparkDefaults, Duration timeout, String logFolder) {
+ Map<String, String> sparkDefaults, Duration timeout, String logFolder,
+ SparkJobRunner.CloudWatchLogs cloudWatchLogs) {
return new SparkJobRunner.Job("emr-serverless", "00f0000000000001", "123456789012", "eu-west-1", "emr-7.5.0",
imageTemplate, "floci-network", "arn:aws:iam::123456789012:role/job", sparkDefaults, application,
- timeout, logFolder);
+ timeout, logFolder, cloudWatchLogs);
}
}