Light, fluffy, and always free - The AWS Local Emulator alternative
feat(lambdaweb): write the platform events of Web Function environments (#5936)
TATSUNO “Taz” Yasuhiro301ac79c6ef1
diff --git a/docs/services/lambda-web-functions.md b/docs/services/lambda-web-functions.mdindex f253d16e6b..6f0f721f65 100644--- a/docs/services/lambda-web-functions.md+++ b/docs/services/lambda-web-functions.md@@ -41,7 +41,7 @@ The `Lambda Web` service model (apiVersion 2025-03-07, `aws lambda-web` in the A- **ARNs** given as identifiers must name the request's region and the caller's account, as on AWS. A revision or endpoint ARN resolves by its id or name only.- **Page tokens** are base64 cursors that carry the request and the position of the page, not the UUIDs AWS issues. A token reused with other parameters gets the AWS "does not match the request" error, and any other token the "invalid or has expired" error.- **Domain names** take the form `<id>.001.lambda-web.<region>.<floci host>` and `<id>.lambda-web.global.<floci host>`, with the host and port of `floci.base-url`. A `PerRegion` endpoint gets one domain per region in `regionalEndpoints`.-- **Not implemented yet:** SigV4a on `MultiRegion` endpoints, the check that a policy's principal exists, the automatic deletion of revisions past 50, the metadata API, the platform log events, the metrics, and serving a `MultiRegion` or `PerRegion` endpoint in its other regions.+- **Not implemented yet:** SigV4a on `MultiRegion` endpoints, the check that a policy's principal exists, the automatic deletion of revisions past 50, the metadata API, the metrics, and serving a `MultiRegion` or `PerRegion` endpoint in its other regions.### `aws lambda-web deploy`@@ -64,7 +64,7 @@ curl -H "Host: <domainName>" http://localhost:4566/The `Host` must match the domain exactly, as on AWS: another case or another port is `404` with `lambda-web-error-type: Platform.InvalidEndpoint`.-An execution environment is a `public.ecr.aws/lambda/nodejs:24` container that a request starts when every environment of the endpoint is full. The code is the revision's S3 object unpacked into `/var/task`. Node starts `/var/runtime/index.js` with the flags AWS uses, as on AWS, and that runtime imports `index.js` or the file `AWS_LAMBDA_NODEJS_ENTRYPOINT` names. So `require.main` is not the app's module, as on AWS. The app sees `AWS_LAMBDA_HTTP_ENDPOINT=0.0.0.0:3000`, `AWS_LAMBDA_INITIALIZATION_TYPE=native-http`, `AWS_EXECUTION_ENV`, `AWS_LAMBDA_FUNCTION_MEMORY_SIZE=2048`, `AWS_LAMBDA_LOG_LEVEL`, `AWS_ACCOUNT_ID`, `AWS_REGION`, credentials and `AWS_ENDPOINT_URL` for Floci, `NODE_PATH`, and its own variables. The server has 10 seconds to listen. An environment that sees no request for `floci.services.lambda.container-idle-timeout-seconds` is stopped with `SIGTERM` and one second, as on scale to zero. A process that exits between requests is replaced by a fresh environment on the next request.+An execution environment is a `public.ecr.aws/lambda/nodejs:24` container that a request starts when every environment of the endpoint is full. The code is the revision's S3 object unpacked into `/var/task`. Node starts `/var/runtime/index.js` with the flags AWS uses, as on AWS, and that runtime imports `index.js` or the file `AWS_LAMBDA_NODEJS_ENTRYPOINT` names. So `require.main` is not the app's module, as on AWS. The app sees `AWS_LAMBDA_HTTP_ENDPOINT=0.0.0.0:3000`, `AWS_LAMBDA_INITIALIZATION_TYPE=native-http`, `AWS_EXECUTION_ENV`, `AWS_LAMBDA_FUNCTION_MEMORY_SIZE=2048`, `AWS_LAMBDA_LOG_LEVEL`, `AWS_ACCOUNT_ID`, `AWS_REGION`, credentials and `AWS_ENDPOINT_URL` for Floci, `NODE_PATH`, and its own variables. The server has 9 seconds to listen. An environment that sees no request for `floci.services.lambda.container-idle-timeout-seconds` is stopped with `SIGTERM` and one second, as on scale to zero. Node runs under Docker's init, so a server without a `SIGTERM` handler stops at once, as on AWS. A process that exits between requests is replaced by a fresh environment on the next request.Each endpoint scales on its own, as on AWS. An environment takes up to the revision's `maxConcurrencyPerEnvironment` requests at once, the least busy one first, so requests rotate over the environments. A request that finds every environment full starts another one, or waits for one that is starting and has room. When the endpoint already runs `maxEnvironments`, an idle environment of another revision makes room, and without one the request is refused with `429` and `Platform.EndpointMaxEnvThrottle`. A `rateLimit` admits a burst of up to `rateLimit` requests that refills at `rateLimit` per second, and refuses the rest with `429` and `Platform.EndpointRequestThrottle`. AWS adjusts the concurrency of each environment under `maxConcurrencyPerEnvironment` from the measured load, so a burst there can spread over more environments than in Floci, and the account quotas are not applied.@@ -85,7 +85,7 @@ The platform answers these cases itself, with the AWS HTML pages and `lambda-web| 404 | `Platform.InvalidEndpoint` | The `Host` is a `lambda-web` domain Floci did not mint || 429 | `Platform.EndpointMaxEnvThrottle` | Every environment is full and the endpoint runs `maxEnvironments` || 429 | `Platform.EndpointRequestThrottle` | The requests exceed the endpoint's `rateLimit` |-| 502 | `Application.TimeoutListenError`, `Application.InitCodeError`, `Runtime.Exit` | The server did not listen within 10 seconds, threw at import, or exited at start |+| 502 | `Application.TimeoutListenError`, `Application.InitCodeError`, `Runtime.Exit` | The server did not listen within 9 seconds, threw at import, or exited at start || 503 | `Platform.EndpointDisabled` | The endpoint's `rateLimit` is 0 || 503 | `Runtime.OriginRequestError` | The process died while handling the request, or tens of milliseconds before it || 504 | `Runtime.RequestTimeout` | No response within the revision's `timeoutSeconds` |@@ -99,7 +99,22 @@ What an environment writes goes to the revision's `logGroup`, by default `/aws/lConsole output becomes the JSON events AWS writes, with `timestamp`, `level`, `message`, `region`, and `requestId` and `endpointName` for output made while a request runs. `console.log` and `console.info` are `INFO`, `console.debug` is `DEBUG`, `console.trace` is `TRACE` with the stack, `console.warn` is `WARN` and `console.error` is `ERROR`. One argument that is not a string goes into `message` as JSON, and an `Error` as `errorType`, `errorMessage` and `stackTrace`. Several arguments make one string, formatted as Node prints them. Output written to stdout or stderr directly, and `console.dir`, stays as it is, one event per line. Events below the revision's `applicationLogLevel` are dropped, and direct output counts as `INFO`. `FATAL` drops all console output, as on AWS. Bytes that are not UTF-8 become U+FFFD. A line longer than 262,118 bytes in UTF-8 is cut before the character that would pass it, and a line of only spaces makes no event.-The `platform.*` events that `systemLogLevel` filters are not written yet. The output of an environment that fails before it listens stays in Floci's own log. A stack trace shows the frames of Floci's runtime where AWS shows its own.+The output of an environment that fails before it listens is written too, as on AWS. A stack trace shows the frames of Floci's runtime where AWS shows its own.++The platform writes the `platform.*` events of the [system log schema](https://docs.aws.amazon.com/lambda/latest/dg/web-functions-logging-system-log-schema.html) to the same stream, and drops those below the revision's `systemLogLevel`:++| Event | When |+| --- | --- |+| `platform.initStart` | The environment starts |+| `platform.initReport` | The server listens, or fails to, with the error type of the `502` page. A server that does not listen also gets an `ERROR` event that says so, unless `applicationLogLevel` is `FATAL` |+| `platform.invokeStart` | Right after a successful `initReport` |+| `platform.requestComplete` | At `DEBUG`, for each request the environment took, with the status, the body bytes and the error type of a `503` or `504` page |+| `platform.invokeSummary` | Every 60 seconds from `invokeStart`, and once more when the environment ends |+| `platform.invokeReport` | When the environment ends, as a failure with `Runtime.Exit` or `Runtime.ExitError` when the process exited on its own |+| `platform.spinDownStart`, `platform.spinDownReport` | When the environment stops, after a failed init too. A process still running one second after `SIGTERM` is a `Runtime.SpinDownTimeout` failure |+| `platform.logsDropped` | Before the next event, after events were dropped because more than 10,000 were waiting |++CPU time and memory come from the container's cgroup, read at each report. Floci cannot read them once the container has stopped, so the CPU time since the last reading is lost when the process exits on its own, an init that fails because the process exited reports neither, and `spinDownReport` has no CPU time. AWS reports the memory of its whole environment, so its numbers are higher.## CloudFormationdiff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlane.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlane.javaindex 586b9b7f29..95e002343b 100644--- a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlane.java+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlane.java@@ -206,7 +206,7 @@ public class LambdaWebDataPlane {private void forward(HttpServerRequest request, String host, WebFunctionTarget target,WebFunctionRevision revision, Environment environment, String requestId, boolean firstTry) {long started = System.nanoTime();- Exchange exchange = new Exchange(request, environment);+ Exchange exchange = new Exchange(request, environment, requestId, target.endpoint().endpointName, started);boolean hasBody = hasRequestBody(request);AtomicBoolean bodyIn = new AtomicBoolean(!hasBody);AtomicBoolean expired = new AtomicBoolean();@@ -225,8 +225,10 @@ public class LambdaWebDataPlane {upstream.reset();if (request.response().headWritten()) {request.response().close();+ exchange.relayed(LambdaWebErrorPages.REQUEST_TIMEOUT.errorType());} else {page(request, LambdaWebErrorPages.REQUEST_TIMEOUT, requestId, false);+ exchange.replaced(LambdaWebErrorPages.REQUEST_TIMEOUT);}}};@@ -251,6 +253,7 @@ public class LambdaWebDataPlane {if (request.response().headWritten()) {if (exchange.finish()) {request.response().close();+ exchange.relayed(null);}return;}@@ -281,11 +284,13 @@ public class LambdaWebDataPlane {Future.fromCompletionStage(environment.awaitExit(EXIT_WAIT), vertx.getOrCreateContext()).onComplete(ignored -> {if (retry && environment.exitedBefore(started)) {+ environment.exchangeClosed();acquire(request, host, target, revision, requestId, environment);return;}request.resume();page(request, LambdaWebErrorPages.ORIGIN_REQUEST_ERROR, requestId, false);+ exchange.replaced(LambdaWebErrorPages.ORIGIN_REQUEST_ERROR);});}@@ -306,26 +311,53 @@ public class LambdaWebDataPlane {return;}if (piped.succeeded()) {+ exchange.relayed(null);return;}if (out.headWritten()) {out.close();+ exchange.relayed(null);} else {page(request, LambdaWebErrorPages.ORIGIN_REQUEST_ERROR, requestId, false);+ exchange.replaced(LambdaWebErrorPages.ORIGIN_REQUEST_ERROR);}});}- /** One request against one environment. The first finisher releases the reservation and the deadline. */+ /**+ * One request against one environment. The first finisher releases the reservation and the deadline, and reports+ * the request in {@code platform.requestComplete}.+ */private final class Exchange {private final HttpServerRequest request;private final Environment environment;+ private final String requestId;+ private final String endpointName;+ private final long started;private final AtomicBoolean finished = new AtomicBoolean();private long deadline = -1;- Exchange(HttpServerRequest request, Environment environment) {+ Exchange(HttpServerRequest request, Environment environment, String requestId, String endpointName,+ long started) {this.request = request;this.environment = environment;+ this.requestId = requestId;+ this.endpointName = endpointName;+ this.started = started;+ environment.exchangeOpened();+ }++ /** The answer of the app, cut when {@code errorType} is set. */+ void relayed(String errorType) {+ HttpServerResponse response = request.response();+ environment.requestComplete(requestId, endpointName, response.getStatusCode(), errorType,+ System.nanoTime() - started, response.bytesWritten());+ }++ /** A page in place of the app's answer. AWS counts none of its bytes. */+ void replaced(LambdaWebErrorPages.Page page) {+ environment.requestComplete(requestId, endpointName, page.status(), page.errorType(),+ System.nanoTime() - started, 0);}boolean finish() {diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.javaindex 3c278d3d20..7277d49310 100644--- a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java@@ -58,6 +58,7 @@ import java.util.concurrent.ConcurrentHashMap;import java.util.concurrent.ExecutionException;import java.util.concurrent.Executors;import java.util.concurrent.ScheduledExecutorService;+import java.util.concurrent.ScheduledFuture;import java.util.concurrent.TimeUnit;import java.util.concurrent.atomic.AtomicBoolean;import java.util.concurrent.atomic.AtomicInteger;@@ -70,7 +71,8 @@ import java.util.stream.Stream;* started by requests that find no room, reused while requests keep coming and removed after the* Lambda idle timeout. Node starts {@code /var/runtime/index.js} with the flags AWS uses, as on AWS. That runtime* turns console output into log events and imports {@code index.js}, or the file {@code AWS_LAMBDA_NODEJS_ENTRYPOINT}- * names, from {@code /var/task}. What the process writes goes to the revision's log group.+ * names, from {@code /var/task}. What the process writes goes to the revision's log group, with the+ * {@code platform.*} events of each phase.*/@ApplicationScopedpublic class LambdaWebEnvironmentManager implements ContainerTeardown {@@ -80,7 +82,8 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {static final String RUNTIME_DIR = "/var/runtime";private static final String RUNTIME_RESOURCE = "lambdaweb/index.mjs";static final int MEMORY_MB = 2048;- static final Duration LISTEN_TIMEOUT = Duration.ofSeconds(10);+ static final Duration LISTEN_TIMEOUT = Duration.ofSeconds(9);+ static final Duration SUMMARY_PERIOD = Duration.ofSeconds(60);static final List<String> NODE_FLAGS = List.of("--expose-gc", "--max-http-header-size", "135168","--no-experimental-require-module", "--max-semi-space-size=102", "--max-old-space-size=1024");private static final Logger LOG = Logger.getLogger(LambdaWebEnvironmentManager.class);@@ -94,6 +97,12 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {private static final Duration LOG_TAIL_TIMEOUT = Duration.ofSeconds(2);private static final Duration EXIT_WATCH_REARM = Duration.ofSeconds(1);private static final Duration EXIT_STATUS_TIMEOUT = Duration.ofSeconds(5);+ private static final int USAGE_TIMEOUT_SECONDS = 2;+ private static final String[] READ_USAGE = {"/bin/cat", "/sys/fs/cgroup/cpu.stat", "/sys/fs/cgroup/memory.peak"};+ private static final long SIGKILL_STATUS = 137;+ private static final long SIGTERM_STATUS = 143;+ /** Longer than the data plane waits to see an exit, so a request the exit failed is counted. */+ private static final Duration EXCHANGES_WAIT = Duration.ofSeconds(2);/*** One container of an endpoint's pool. It joins the pool before it starts, so requests that@@ -111,6 +120,12 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {private volatile int port;private volatile String logStream;private volatile LambdaWebLogs.Sink logs;+ private volatile LambdaWebPlatformLog platformLog;+ private volatile ScheduledFuture<?> summaries;+ /** Set once the app listens. An exit before that is the start's to report. */+ private volatile boolean serving;+ /** Requests sent to the environment and not reported yet. Guarded by the environment. */+ private int openExchanges;private volatile SessionCreds roleCredentials;private final AtomicInteger inFlight = new AtomicInteger();private final AtomicLong lastUsed = new AtomicLong(System.nanoTime());@@ -154,8 +169,44 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {}private void reserve() {- inFlight.incrementAndGet();+ int reserved = inFlight.incrementAndGet();lastUsed.set(System.nanoTime());+ LambdaWebPlatformLog platform = platformLog;+ if (platform != null) {+ platform.concurrency(reserved);+ }+ }++ public synchronized void exchangeOpened() {+ openExchanges++;+ }++ /** {@code errorType} names the page the platform answered in place of the app. */+ public void requestComplete(String requestId, String endpointName, int result, String errorType,+ long durationNanos, long responseBytes) {+ LambdaWebPlatformLog platform = platformLog;+ if (platform != null) {+ platform.requestComplete(requestId, endpointName, result, errorType, durationNanos, responseBytes);+ }+ exchangeClosed();+ }++ /** Also for a request that goes to another environment, because this one had exited before it was sent. */+ public synchronized void exchangeClosed() {+ openExchanges--;+ notifyAll();+ }++ private synchronized void awaitExchanges(Duration limit) {+ long deadline = System.nanoTime() + limit.toNanos();+ try {+ for (long left = limit.toMillis(); openExchanges > 0 && left > 0;+ left = TimeUnit.NANOSECONDS.toMillis(deadline - System.nanoTime())) {+ wait(left);+ }+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();+ }}/**@@ -360,6 +411,7 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {if (environment.containerId == null) {return;}+ stopSummaries(environment);try {exitCodeIfStopped(environment.containerId).ifPresent(exitCode ->LOG.infov("The Lambda Web environment {0} had exited with status {1}: {2}",@@ -367,7 +419,46 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {} catch (RuntimeException e) {LOG.debugv(e, "Could not inspect the Lambda Web environment {0}", environment.containerId);}- stopContainer(environment);+ LambdaWebPlatformLog platform = environment.platformLog;+ if (platform == null || !environment.serving) {+ stopContainer(environment);+ return;+ }+ platform.end(usage(environment.containerId), null);+ platform.spinDownStart();+ long stopping = System.nanoTime();+ Long status = stopForStatus(environment.containerId);+ try {+ stopContainer(environment);+ } finally {+ platform.spinDownReport(spinDownError(status), System.nanoTime() - stopping);+ }+ }++ /** Docker sends SIGKILL once the second after SIGTERM has passed. */+ private static String spinDownError(Long status) {+ if (status == null || status == 0 || status == SIGTERM_STATUS) {+ return null;+ }+ return status == SIGKILL_STATUS ? "Runtime.SpinDownTimeout" : "Runtime.ExitError";+ }++ private Long stopForStatus(String containerId) {+ try {+ lifecycleManager.getDockerClient().stopContainerCmd(containerId).withTimeout(1).exec();+ return lifecycleManager.getDockerClient().inspectContainerCmd(containerId).exec().getState()+ .getExitCodeLong();+ } catch (RuntimeException e) {+ LOG.debugv(e, "Could not stop the Lambda Web environment {0}", containerId);+ return null;+ }+ }++ private static void stopSummaries(Environment environment) {+ ScheduledFuture<?> summaries = environment.summaries;+ if (summaries != null) {+ summaries.cancel(false);+ }}private void stopContainer(Environment environment) {@@ -449,6 +540,7 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {.withWorkingDir(TASK_DIR).withMemoryMb(MEMORY_MB).withLoopbackPortBinding(APP_PORT, 0)+ .withInit().withDockerNetwork(config.services().lambda().dockerNetwork()).withHostDockerInternalOnLinux().withEmbeddedDns()@@ -462,30 +554,90 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {roleCredentials.ifPresent(creds -> executionRoleCredentials.unregister(accountId, creds.accessKeyId()));throw e;}+ LambdaWebPlatformLog platform = null;try {copyCode(containerId, accountId, region, fn, revision);RetryingTarCopier.copyBytes(lifecycleManager.getDockerClient(), containerId, RUNTIME_DIR, "index.js",runtime, 0644);+ environment.containerId = containerId;+ environment.logStream = LOG_STREAM_DATE.format(Instant.now()) + "/" + fn.functionName + "["+ + revision.revisionId + "]" + UUID.randomUUID().toString().replace("-", "");+ environment.logs = logs.open(accountId, region, revision.serviceConfig.logGroup, environment.logStream,+ revision.serviceConfig.systemLogLevel);+ platform = new LambdaWebPlatformLog(environment.logs, region, fn.functionName, revision.revisionId,+ revision.buildConfig.runtime, revision.serviceConfig.applicationLogLevel);+ environment.platformLog = platform;ContainerInfo info = lifecycleManager.startCreated(containerId, spec);+ platform.initStart();+ watchExit(environment, true);EndpointInfo endpoint = info.getEndpoint(APP_PORT);- awaitListening(containerId);- environment.containerId = containerId;+ awaitListening(containerId, platform);+ platform.initReport(null, usage(containerId));environment.host = endpoint.host();environment.port = endpoint.port();- environment.logStream = LOG_STREAM_DATE.format(Instant.now()) + "/" + fn.functionName + "["- + revision.revisionId + "]" + UUID.randomUUID().toString().replace("-", "");environment.roleCredentials = roleCredentials.orElse(null);- environment.logs = logs.open(accountId, region, revision.serviceConfig.logGroup, environment.logStream);- watchExit(environment, true);+ platform.invokeStart(environment.inFlight.get());+ environment.summaries = reaper.scheduleAtFixedRate(() -> summarize(environment),+ SUMMARY_PERIOD.toMillis(), SUMMARY_PERIOD.toMillis(), TimeUnit.MILLISECONDS);+ Long exitedEarly;+ synchronized (environment) {+ environment.serving = true;+ exitedEarly = environment.exitSeen.getNow(null);+ }+ if (exitedEarly != null) {+ exited(environment, exitedEarly);+ }LOG.infov("Started the Lambda Web environment {0} of {1} revision {2} at {3}:{4}",containerId, fn.functionName, revision.revisionId, endpoint.host(), endpoint.port());} catch (RuntimeException e) {+ if (platform != null && e instanceof StartFailure failure) {+ platform.initReport(failure.page().errorType(), usage(containerId));+ platform.spinDownStart();+ }+ long stopping = System.nanoTime();lifecycleManager.stopAndRemove(containerId, null, 1);+ if (platform != null && e instanceof StartFailure) {+ platform.spinDownReport(null, System.nanoTime() - stopping);+ }roleCredentials.ifPresent(creds -> executionRoleCredentials.unregister(accountId, creds.accessKeyId()));throw e;}}+ private void summarize(Environment environment) {+ Thread.ofVirtual().name("lambda-web-summary").start(() -> {+ if (environment.isAlive()) {+ environment.platformLog.invokeSummary(usage(environment.containerId), environment.inFlight.get());+ }+ });+ }++ /** The CPU time and the peak memory of the container, or null when they cannot be read. */+ private LambdaWebPlatformLog.Usage usage(String containerId) {+ try {+ ContainerExec.Result result = ContainerExec.runMerged(lifecycleManager.getDockerClient(), containerId,+ READ_USAGE, USAGE_TIMEOUT_SECONDS);+ if (result.timedOut()) {+ return null;+ }+ // A kernel before 5.19 has no memory.peak. The CPU time still counts.+ long cpuMicros = 0;+ long memoryBytes = 0;+ for (String line : result.stdout().split("\n")) {+ String[] fields = line.trim().split(" ");+ if (fields.length == 2 && "usage_usec".equals(fields[0])) {+ cpuMicros = Long.parseLong(fields[1]);+ } else if (fields.length == 1 && !fields[0].isEmpty()) {+ memoryBytes = Long.parseLong(fields[0]);+ }+ }+ return cpuMicros == 0 && memoryBytes == 0 ? null : new LambdaWebPlatformLog.Usage(cpuMicros, memoryBytes);+ } catch (RuntimeException e) {+ LOG.debugv(e, "Could not read the usage of the Lambda Web environment {0}", containerId);+ return null;+ }+ }+private List<String> environment(String accountId, String region, WebFunctionRevision revision,Map<String, String> userEnv, Optional<SessionCreds> roleCredentials) {List<String> env = new ArrayList<>();@@ -562,7 +714,7 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {}}- private void awaitListening(String containerId) {+ private void awaitListening(String containerId, LambdaWebPlatformLog platform) {long deadline = System.nanoTime() + LISTEN_TIMEOUT.toNanos();while (true) {Optional<Long> exitCode = exitCodeIfStopped(containerId);@@ -576,6 +728,10 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {return;}if (System.nanoTime() > deadline) {+ platform.applicationError(LambdaWebErrorPages.TIMEOUT_LISTEN_ERROR.errorType()+ + ": nothing was listening on 0.0.0.0:" + APP_PORT + " within " + LISTEN_TIMEOUT.toMillis()+ + "ms. Your server must listen on the host and port from the AWS_LAMBDA_HTTP_ENDPOINT"+ + " environment variable");throw new StartFailure(LambdaWebErrorPages.TIMEOUT_LISTEN_ERROR);}try {@@ -738,6 +894,12 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {/** The environment leaves the pool before waiters hear of the exit, so the request after a failed one starts anew. */private void exited(Environment environment, long seenAt) {+ synchronized (environment) {+ if (!environment.serving) {+ environment.sawExit(seenAt);+ return;+ }+ }boolean running = environment.alive.compareAndSet(true, false);if (running) {removeFromPool(environment);@@ -747,12 +909,22 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {return;}// Docker can take seconds to report the status under load. The reaper stays free to rearm dropped watches.+ stopSummaries(environment);Thread.ofVirtual().name("lambda-web-exit").start(() -> {+ LambdaWebPlatformLog platform = environment.platformLog;try {+ Integer status = exitStatus(environment.containerId);LOG.infov("The Lambda Web environment {0} exited with status {1}: {2}",- environment.containerId, exitStatus(environment.containerId),- lastLogLines(environment.containerId));- lifecycleManager.stopAndRemove(environment.containerId, null, 0);+ environment.containerId, status, lastLogLines(environment.containerId));+ environment.awaitExchanges(EXCHANGES_WAIT);+ platform.end(null, Integer.valueOf(0).equals(status) ? "Runtime.Exit" : "Runtime.ExitError");+ platform.spinDownStart();+ long stopping = System.nanoTime();+ try {+ lifecycleManager.stopAndRemove(environment.containerId, null, 0);+ } finally {+ platform.spinDownReport(null, System.nanoTime() - stopping);+ }} finally {forgetCredentials(environment);}diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogs.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogs.javaindex ef3daab8e3..caee8e6a74 100644--- a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogs.java+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogs.java@@ -1,5 +1,7 @@package io.github.hectorvent.floci.services.lambdaweb;+import com.fasterxml.jackson.databind.ObjectMapper;+import com.fasterxml.jackson.databind.node.ObjectNode;import io.github.hectorvent.floci.core.common.AwsException;import io.github.hectorvent.floci.services.cloudwatch.logs.CloudWatchLogsService;import jakarta.annotation.PreDestroy;@@ -7,6 +9,10 @@ import jakarta.enterprise.context.ApplicationScoped;import jakarta.inject.Inject;import org.jboss.logging.Logger;+import java.nio.charset.StandardCharsets;+import java.time.Instant;+import java.time.ZoneOffset;+import java.time.format.DateTimeFormatter;import java.util.ArrayList;import java.util.List;import java.util.Map;@@ -20,14 +26,21 @@ import java.util.concurrent.atomic.AtomicInteger;* Delivers the log events of Web Function environments. Each event goes to the stream of its* environment in the {@code .local} companion of the revision's log group, and to the same stream* with an {@code -<account>-<region>} suffix in the log group itself, where AWS centralizes the- * regions an endpoint serves. A group or stream is created when its first event arrives.+ * regions an endpoint serves. A group or stream is created when its first event arrives. The+ * {@code platform.*} events below the revision's {@code systemLogLevel} are dropped.*/@ApplicationScopedpublic class LambdaWebLogs {private static final Logger LOG = Logger.getLogger(LambdaWebLogs.class);- private static final int MAX_QUEUED_EVENTS = 10_000;+ static final int MAX_QUEUED_EVENTS = 10_000;private static final int MAX_BATCH_EVENTS = 1_000;+ private static final List<String> SYSTEM_LEVELS = List.of("DEBUG", "INFO", "WARN");+ private static final DateTimeFormatter SECONDS =+ DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss").withZone(ZoneOffset.UTC);+ private static final String LOGS_DROPPED_REASON =+ "Some logs were dropped because the downstream consumer is slower than the logs production rate";+ private static final ObjectMapper JSON = new ObjectMapper();private final CloudWatchLogsService logs;private final ExecutorService writers = Executors.newVirtualThreadPerTaskExecutor();@@ -37,12 +50,19 @@ public class LambdaWebLogs {this.logs = logs;}- public Sink open(String accountId, String region, String logGroup, String logStream) {- return new Sink(accountId, region, List.of(+ public Sink open(String accountId, String region, String logGroup, String logStream, String systemLogLevel) {+ return new Sink(accountId, region, SYSTEM_LEVELS.indexOf(systemLogLevel), List.of(new Destination(logGroup + ".local", logStream),new Destination(logGroup, logStream + "-" + accountId + "-" + region)));}+ /** A time as AWS writes it in a platform event, without the trailing zeros of the fraction. */+ static String timestamp(Instant at, int fractionDigits) {+ long fraction = at.getNano() / (long) Math.pow(10, 9 - fractionDigits);+ String digits = String.format("%0" + fractionDigits + "d", fraction).replaceFirst("0+$", "");+ return SECONDS.format(at) + (digits.isEmpty() ? "" : "." + digits) + "Z";+ }+@PreDestroyvoid shutdown() {writers.shutdownNow();@@ -61,27 +81,75 @@ public class LambdaWebLogs {public final class Sink {private final String accountId;private final String region;+ private final int systemLevel;private final List<Destination> destinations;private final ConcurrentLinkedQueue<Event> queue = new ConcurrentLinkedQueue<>();private final AtomicInteger queued = new AtomicInteger();private final AtomicBoolean writing = new AtomicBoolean();+ private int droppedRecords;+ private long droppedBytes;- private Sink(String accountId, String region, List<Destination> destinations) {+ private Sink(String accountId, String region, int systemLevel, List<Destination> destinations) {this.accountId = accountId;this.region = region;+ this.systemLevel = systemLevel;this.destinations = destinations;}public void accept(String message) {- if (queued.incrementAndGet() > MAX_QUEUED_EVENTS) {- queued.decrementAndGet();+ accept(System.currentTimeMillis(), message);+ }++ void accept(Instant at, String message) {+ accept(at.toEpochMilli(), message);+ }++ /** A {@code platform.*} event, unless its level is below {@code systemLogLevel}. */+ public void platform(Instant at, String level, String type, ObjectNode record) {+ if (SYSTEM_LEVELS.indexOf(level) < systemLevel) {+ return;+ }+ accept(at.toEpochMilli(), envelope(at, level, type, record));+ }++ private String envelope(Instant at, String level, String type, ObjectNode record) {+ ObjectNode event = JSON.createObjectNode();+ event.put("timestamp", timestamp(at, 3));+ event.put("level", level);+ event.put("type", type);+ event.put("region", region);+ event.set("record", record);+ return event.toString();+ }++ /** After a drop, the next event that fits is preceded by a {@code platform.logsDropped} event, as on AWS. */+ private synchronized void accept(long timestamp, String message) {+ boolean reportDrop = droppedRecords > 0 && SYSTEM_LEVELS.indexOf("WARN") >= systemLevel;+ if (queued.get() + (reportDrop ? 2 : 1) > MAX_QUEUED_EVENTS) {+ droppedRecords++;+ droppedBytes += message.getBytes(StandardCharsets.UTF_8).length;LOG.debugv("Dropped a log event of {0}: too many are waiting", destinations.getFirst().logStream());return;}- queue.add(new Event(System.currentTimeMillis(), message));+ if (reportDrop) {+ ObjectNode record = JSON.createObjectNode();+ record.put("reason", LOGS_DROPPED_REASON);+ record.put("droppedRecords", droppedRecords);+ record.put("droppedBytes", droppedBytes);+ Instant at = Instant.ofEpochMilli(timestamp);+ enqueue(timestamp, envelope(at, "WARN", "platform.logsDropped", record));+ }+ droppedRecords = 0;+ droppedBytes = 0;+ enqueue(timestamp, message);schedule();}+ private void enqueue(long timestamp, String message) {+ queued.incrementAndGet();+ queue.add(new Event(timestamp, message));+ }+private void schedule() {if (!writing.compareAndSet(false, true)) {return;diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebPlatformLog.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebPlatformLog.javanew file mode 100644index 0000000000..ec452e6fc7--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebPlatformLog.java@@ -0,0 +1,225 @@+package io.github.hectorvent.floci.services.lambdaweb;++import com.fasterxml.jackson.databind.ObjectMapper;+import com.fasterxml.jackson.databind.node.ObjectNode;+import io.github.hectorvent.floci.core.common.AwsArnUtils;++import java.math.BigDecimal;+import java.math.RoundingMode;+import java.time.Instant;+import java.time.ZoneOffset;+import java.time.format.DateTimeFormatter;+import java.util.UUID;++/**+ * The {@code platform.*} events of one environment. AWS reports the CPU time and the peak memory of+ * the whole environment, which Floci reads from the container's cgroup. A reading that fails keeps+ * the last one, so CPU time after the last reading is lost when the process exits on its own.+ */+final class LambdaWebPlatformLog {++ static final String INIT_TYPE = "native-http";+ private static final String RUNTIME_VERSION = "nodejs.web24.v30";+ private static final ObjectMapper JSON = new ObjectMapper();+ private static final long MIB = 1024 * 1024;+ private static final DateTimeFormatter CONSOLE_TIME =+ DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'").withZone(ZoneOffset.UTC);++ /** The CPU time and the peak memory of the environment so far. */+ record Usage(long cpuMicros, long memoryBytes) {+ }++ private final LambdaWebLogs.Sink sink;+ private final String region;+ private final String functionName;+ private final String revisionId;+ private final String runtime;+ private final String applicationLogLevel;+ private final String instanceId = UUID.randomUUID().toString();+ private long initStarted;+ private long invokeStarted;+ private Instant windowStart;+ private int windowRequests;+ private int windowConcurrency;+ private int totalRequests;+ private int totalConcurrency;+ private long cpuMicros;+ private long cpuAtInvokeStart;+ private long cpuAtWindowStart;+ private long peakMemoryMb;+ private boolean serving;+ private boolean ended;++ LambdaWebPlatformLog(LambdaWebLogs.Sink sink, String region, String functionName, String revisionId,+ String runtime, String applicationLogLevel) {+ this.sink = sink;+ this.region = region;+ this.functionName = functionName;+ this.revisionId = revisionId;+ this.runtime = runtime;+ this.applicationLogLevel = applicationLogLevel;+ }++ synchronized void initStart() {+ initStarted = System.nanoTime();+ ObjectNode record = JSON.createObjectNode();+ record.put("runtimeVersion", runtime + ":" + RUNTIME_VERSION);+ record.put("runtimeVersionArn", AwsArnUtils.Arn.of("lambda", region, "",+ "runtime/" + runtime + "/version/" + RUNTIME_VERSION).toString());+ record.put("functionName", functionName);+ record.put("revisionId", revisionId);+ record.put("instanceId", instanceId);+ record.put("initializationType", INIT_TYPE);+ sink.platform(Instant.now(), "INFO", "platform.initStart", record);+ }++ /** {@code errorType} is null when the app listens. */+ synchronized void initReport(String errorType, Usage usage) {+ read(usage);+ ObjectNode record = status(errorType);+ record.put("instanceId", instanceId);+ record.put("initializationType", INIT_TYPE);+ ObjectNode metrics = record.putObject("metrics");+ metrics.put("totalDurationMs", wholeMillis(System.nanoTime() - initStarted));+ metrics.put("totalCpuTimeMs", wholeMillis(cpuMicros * 1000));+ metrics.put("maxMemoryUsageMb", peakMemoryMb);+ sink.platform(Instant.now(), errorType == null ? "INFO" : "WARN", "platform.initReport", record);+ }++ /** A failure AWS writes in the shape of console output, with the time as Node prints it, and drops at FATAL. */+ void applicationError(String message) {+ if ("FATAL".equals(applicationLogLevel)) {+ return;+ }+ ObjectNode event = JSON.createObjectNode();+ Instant now = Instant.now();+ event.put("timestamp", CONSOLE_TIME.format(now));+ event.put("level", "ERROR");+ event.put("message", message);+ event.put("region", region);+ sink.accept(now, event.toString());+ }++ synchronized void invokeStart(int inFlight) {+ serving = true;+ invokeStarted = System.nanoTime();+ windowStart = Instant.now();+ cpuAtInvokeStart = cpuMicros;+ cpuAtWindowStart = cpuMicros;+ windowConcurrency = inFlight;+ totalConcurrency = inFlight;+ ObjectNode record = JSON.createObjectNode();+ record.put("functionName", functionName);+ record.put("revisionId", revisionId);+ record.put("instanceId", instanceId);+ sink.platform(windowStart, "INFO", "platform.invokeStart", record);+ }++ synchronized boolean serving() {+ return serving;+ }++ synchronized void concurrency(int inFlight) {+ windowConcurrency = Math.max(windowConcurrency, inFlight);+ totalConcurrency = Math.max(totalConcurrency, inFlight);+ }++ /** {@code errorType} names the page the platform answered in place of the app. */+ synchronized void requestComplete(String requestId, String endpointName, int result, String errorType,+ long durationNanos, long responseBytes) {+ windowRequests++;+ totalRequests++;+ ObjectNode record = JSON.createObjectNode();+ record.put("requestId", requestId);+ record.put("endpointName", endpointName);+ record.put("instanceId", instanceId);+ record.put("result", result);+ if (errorType != null) {+ record.put("errorType", errorType);+ }+ ObjectNode metrics = record.putObject("metrics");+ metrics.put("durationMs", BigDecimal.valueOf(durationNanos).movePointLeft(6).setScale(3, RoundingMode.HALF_UP));+ metrics.put("responseBytes", responseBytes);+ sink.platform(Instant.now(), "DEBUG", "platform.requestComplete", record);+ }++ /** Closes the window that started at the last summary, or at {@code invokeStart}, unless the environment ended. */+ synchronized void invokeSummary(Usage usage, int inFlight) {+ if (ended) {+ return;+ }+ read(usage);+ Instant now = Instant.now();+ ObjectNode record = JSON.createObjectNode();+ record.put("startTime", LambdaWebLogs.timestamp(windowStart, 9));+ record.put("instanceId", instanceId);+ ObjectNode metrics = record.putObject("metrics");+ metrics.put("requests", windowRequests);+ metrics.put("concurrentRequests", windowConcurrency);+ metrics.put("cpuTimeMs", wholeMillis((cpuMicros - cpuAtWindowStart) * 1000));+ metrics.put("maxMemoryUsageMb", peakMemoryMb);+ sink.platform(now, "INFO", "platform.invokeSummary", record);+ windowStart = now;+ windowRequests = 0;+ windowConcurrency = inFlight;+ cpuAtWindowStart = cpuMicros;+ }++ /**+ * The last summary and {@code invokeReport}. {@code errorType} is null when Floci stops the environment, and names+ * the exit when the process ends.+ */+ synchronized void end(Usage usage, String errorType) {+ invokeSummary(usage, 0);+ ended = true;+ ObjectNode record = status(errorType);+ record.put("instanceId", instanceId);+ ObjectNode metrics = record.putObject("metrics");+ metrics.put("totalCpuTimeMs", wholeMillis((cpuMicros - cpuAtInvokeStart) * 1000));+ metrics.put("totalDurationMs", wholeMillis(System.nanoTime() - invokeStarted));+ metrics.put("totalRequests", totalRequests);+ metrics.put("concurrentRequests", totalConcurrency);+ metrics.put("maxMemoryUsageMb", peakMemoryMb);+ sink.platform(Instant.now(), errorType == null ? "INFO" : "WARN", "platform.invokeReport", record);+ }++ synchronized void spinDownStart() {+ ObjectNode record = JSON.createObjectNode();+ record.put("functionName", functionName);+ record.put("revisionId", revisionId);+ record.put("instanceId", instanceId);+ sink.platform(Instant.now(), "INFO", "platform.spinDownStart", record);+ }++ /** The cgroup is gone once the process has stopped, so the CPU time of the shutdown is not known. */+ synchronized void spinDownReport(String errorType, long durationNanos) {+ ObjectNode record = status(errorType);+ record.put("instanceId", instanceId);+ ObjectNode metrics = record.putObject("metrics");+ metrics.put("totalDurationMs", wholeMillis(durationNanos));+ metrics.put("totalCpuTimeMs", wholeMillis(0));+ metrics.put("maxMemoryUsageMb", peakMemoryMb);+ sink.platform(Instant.now(), errorType == null ? "INFO" : "WARN", "platform.spinDownReport", record);+ }++ private void read(Usage usage) {+ if (usage != null) {+ cpuMicros = Math.max(cpuMicros, usage.cpuMicros());+ peakMemoryMb = Math.max(peakMemoryMb, usage.memoryBytes() / MIB);+ }+ }++ private static ObjectNode status(String errorType) {+ ObjectNode record = JSON.createObjectNode();+ record.put("status", errorType == null ? "success" : "failure");+ if (errorType != null) {+ record.put("errorType", errorType);+ }+ return record;+ }++ /** AWS writes these durations in whole milliseconds with three decimals. */+ private static BigDecimal wholeMillis(long nanos) {+ return BigDecimal.valueOf(Math.round(nanos / 1_000_000.0)).setScale(3);+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.javaindex 70c7c1592a..4680d3a881 100644--- a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java@@ -34,6 +34,7 @@ import java.util.concurrent.ExecutorService;import java.util.concurrent.Executors;import java.util.concurrent.Future;import java.util.concurrent.TimeUnit;+import java.util.function.Predicate;import java.util.zip.ZipEntry;import java.util.zip.ZipOutputStream;@@ -64,6 +65,7 @@ class LambdaWebDataPlaneDockerIntegrationTest {const http = require('node:http');const [host, port] = process.env.AWS_LAMBDA_HTTP_ENDPOINT.split(':');console.log('starting');+ if (process.env.MODE === 'throw') { throw new Error('init boom'); }http.createServer((req, res) => {if (req.url.startsWith('/log')) {console.info({route: 'log', main: require.main === module, argv: process.argv.slice(1)});@@ -209,7 +211,10 @@ class LambdaWebDataPlaneDockerIntegrationTest {String logStream = logged.header("lambda-web-logstream");String group = "/aws/lambda/web/" + functionName;- List<String> local = awaitLogEvents(group + ".local", logStream);+ List<String> all = awaitLogEvents(group + ".local", logStream);+ assertEquals(List.of("platform.initStart", "platform.initReport", "platform.invokeStart"),+ types(all), all.toString());+ List<String> local = all.stream().filter(event -> !isPlatformEvent(event)).toList();assertEquals(5, local.size(), local.toString());JsonPath starting = JsonPath.from(local.getFirst());assertEquals("INFO", starting.getString("level"));@@ -229,7 +234,7 @@ class LambdaWebDataPlaneDockerIntegrationTest {assertTrue(local.stream().allMatch(event -> event.endsWith("\n")), local.toString());List<String> centralized = awaitLogEvents(group, logStream + "-000000000000-us-east-1");- assertEquals(new HashSet<>(local), new HashSet<>(centralized));+ assertEquals(new HashSet<>(all), new HashSet<>(centralized));}@Test@@ -398,6 +403,11 @@ class LambdaWebDataPlaneDockerIntegrationTest {}private String createFunction(int timeoutSeconds, int maxConcurrencyPerEnvironment) throws IOException {+ return createFunction(timeoutSeconds, maxConcurrencyPerEnvironment, "INFO", "serve");+ }++ private String createFunction(int timeoutSeconds, int maxConcurrencyPerEnvironment, String systemLogLevel,+ String mode) throws IOException {functionName = "dp-docker-" + Long.toString(System.nanoTime(), 36);s3Service.createBucket(BUCKET, "us-east-1");s3Service.putObject(BUCKET, functionName + ".zip", zip(APP), "application/zip", Map.of());@@ -409,16 +419,136 @@ class LambdaWebDataPlaneDockerIntegrationTest {"runtimeConfig": {"runtime": "nodejs24.x"}},"serviceConfig": {"executionRoleArn": "%s", "timeoutSeconds": %d,"maxConcurrencyPerEnvironment": %d,- "environmentVariables": {"CUSTOM": "yes"}}},+ "environmentVariables": {"CUSTOM": "yes", "MODE": "%s"},+ "telemetryConfig": {"loggingConfig": {"systemLogLevel": "%s"}}}},"endpointConfig": {"endpointName": "dev", "endpointType": "HomeRegion","authType": "ApplicationManaged", "autoDeploymentMode": "LatestRevision"}}""".formatted(functionName, BUCKET, functionName, ROLE, timeoutSeconds,- maxConcurrencyPerEnvironment))+ maxConcurrencyPerEnvironment, mode, systemLogLevel)).when().put(BASE + "/web-functions").then().statusCode(202).extract().path("endpoint.domainName");}+ @Test+ @Timeout(180)+ void platformEventsFollowAnEnvironmentFromItsStartToItsEnd() throws Exception {+ String domain = createFunction(30, 64, "DEBUG", "serve");+ String group = "/aws/lambda/web/" + functionName + ".local";+ Response served = given().header("Host", domain).when().get("/status/201");+ served.then().statusCode(201);+ String crashed = served.header("lambda-web-logstream");+ given().header("Host", domain).when().get("/crash").then().statusCode(503);++ List<String> crash = awaitLogEvents(group, crashed, events -> types(events).contains("platform.spinDownReport")+ && types(events).stream().filter("platform.requestComplete"::equals).count() == 2);+ JsonPath initStart = platformEvent(crash, "platform.initStart");+ String instanceId = initStart.getString("record.instanceId");+ assertEquals("INFO", initStart.getString("level"));+ assertEquals("us-east-1", initStart.getString("region"));+ assertEquals(functionName, initStart.getString("record.functionName"));+ assertEquals("nodejs24.x:nodejs.web24.v30", initStart.getString("record.runtimeVersion"));+ assertEquals("arn:aws:lambda:us-east-1::runtime/nodejs24.x/version/nodejs.web24.v30",+ initStart.getString("record.runtimeVersionArn"));+ assertEquals("native-http", initStart.getString("record.initializationType"));+ JsonPath initReport = platformEvent(crash, "platform.initReport");+ assertEquals("success", initReport.getString("record.status"));+ assertEquals(instanceId, initReport.getString("record.instanceId"));+ assertTrue(initReport.getInt("record.metrics.maxMemoryUsageMb") > 0, initReport.prettify());+ List<JsonPath> requests = crash.stream().filter(event -> event.contains("platform.requestComplete"))+ .map(JsonPath::from).toList();+ assertEquals(2, requests.size(), crash.toString());+ JsonPath ok = requests.stream().filter(event -> event.getInt("record.result") == 201).findFirst().orElseThrow();+ assertEquals("DEBUG", ok.getString("level"));+ assertEquals(served.header("lambda-web-request-id"), ok.getString("record.requestId"));+ assertEquals("dev", ok.getString("record.endpointName"));+ assertEquals(11, ok.getInt("record.metrics.responseBytes"));+ assertFalse(ok.getMap("record").containsKey("errorType"), ok.prettify());+ JsonPath failed = requests.stream().filter(event -> event.getInt("record.result") == 503).findFirst().orElseThrow();+ assertEquals("Runtime.OriginRequestError", failed.getString("record.errorType"));+ assertEquals(0, failed.getInt("record.metrics.responseBytes"));+ JsonPath crashReport = platformEvent(crash, "platform.invokeReport");+ assertEquals("WARN", crashReport.getString("level"));+ assertEquals("failure", crashReport.getString("record.status"));+ assertEquals("Runtime.ExitError", crashReport.getString("record.errorType"));+ assertEquals(2, crashReport.getInt("record.metrics.totalRequests"));+ assertEquals("success", platformEvent(crash, "platform.spinDownReport").getString("record.status"));++ String stopped = given().header("Host", domain).when().get("/").then().statusCode(200)+ .extract().header("lambda-web-logstream");+ environments.discard(environments.environmentOf(stopped));+ List<String> stop = awaitLogEvents(group, stopped, events -> types(events).contains("platform.spinDownReport"));+ assertEquals(List.of("platform.initStart", "platform.initReport", "platform.invokeStart",+ "platform.requestComplete", "platform.invokeSummary", "platform.invokeReport",+ "platform.spinDownStart", "platform.spinDownReport"), types(stop), stop.toString());+ JsonPath stopReport = platformEvent(stop, "platform.invokeReport");+ assertEquals("INFO", stopReport.getString("level"));+ assertEquals("success", stopReport.getString("record.status"));+ assertEquals(1, stopReport.getInt("record.metrics.totalRequests"));+ assertEquals(1, platformEvent(stop, "platform.invokeSummary").getInt("record.metrics.requests"));+ assertEquals("success", platformEvent(stop, "platform.spinDownReport").getString("record.status"));+ }++ @Test+ @Timeout(180)+ void anAppThatFailsAtImportWritesItsOutputAndAFailedInitReport() throws Exception {+ String domain = createFunction(30, 64, "WARN", "throw");+ String group = "/aws/lambda/web/" + functionName + ".local";+ given().header("Host", domain).when().get("/").then().statusCode(502)+ .header("lambda-web-error-type", "Application.InitCodeError");++ List<String> events = awaitLogEvents(group, awaitLogStream(group), each -> types(each).contains(+ "platform.initReport") && each.stream().anyMatch(event -> event.contains("init boom")));+ assertEquals(List.of("platform.initReport"), types(events), events.toString());+ JsonPath initReport = platformEvent(events, "platform.initReport");+ assertEquals("WARN", initReport.getString("level"));+ assertEquals("failure", initReport.getString("record.status"));+ assertEquals("Application.InitCodeError", initReport.getString("record.errorType"));+ assertTrue(events.stream().anyMatch(event -> event.contains("\"message\":\"starting\"")), events.toString());+ assertTrue(events.stream().anyMatch(event -> event.contains("init boom")), events.toString());+ }++ private static boolean isPlatformEvent(String event) {+ return event.startsWith("{") && event.contains("\"type\":\"platform.");+ }++ private static List<String> types(List<String> events) {+ return events.stream().filter(LambdaWebDataPlaneDockerIntegrationTest::isPlatformEvent)+ .map(event -> JsonPath.from(event).getString("type")).toList();+ }++ private static JsonPath platformEvent(List<String> events, String type) {+ return events.stream().filter(LambdaWebDataPlaneDockerIntegrationTest::isPlatformEvent)+ .map(JsonPath::from).filter(event -> type.equals(event.getString("type")))+ .findFirst().orElseThrow(() -> new AssertionError("No " + type + " in " + events));+ }++ /** The messages of a log stream once they are {@code complete}, or what it holds after 15 seconds. */+ private static List<String> awaitLogEvents(String logGroup, String logStream, Predicate<List<String>> complete)+ throws InterruptedException {+ long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(15);+ List<String> messages = logEvents(logGroup, logStream);+ while (!complete.test(messages) && System.nanoTime() < deadline) {+ Thread.sleep(200);+ messages = logEvents(logGroup, logStream);+ }+ return messages;+ }++ private static String awaitLogStream(String logGroup) throws InterruptedException {+ long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(15);+ while (true) {+ List<String> streams = RestAssuredJsonUtils.awsAction("Logs_20140328", "DescribeLogStreams",+ "{\"logGroupName\": \"%s\"}".formatted(logGroup))+ .then().extract().jsonPath().getList("logStreams.logStreamName", String.class);+ if (streams != null && !streams.isEmpty()) {+ return streams.getFirst();+ }+ assertTrue(System.nanoTime() < deadline, "No log stream in " + logGroup);+ Thread.sleep(200);+ }+ }+/*** The messages of a log stream once the last output of {@code /log} on stdout and on stderr is in it, or what it* holds after 15 seconds. The output of one stream arrives in order, so nothing written before comes later.diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogsTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogsTest.javaindex 361b3a1fae..b3950365d0 100644--- a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogsTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogsTest.java@@ -16,6 +16,7 @@ import java.util.concurrent.CountDownLatch;import java.util.concurrent.TimeUnit;import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertTrue;import static org.mockito.ArgumentMatchers.anyList;import static org.mockito.ArgumentMatchers.argThat;import static org.mockito.ArgumentMatchers.eq;@@ -57,7 +58,7 @@ class LambdaWebLogsTest {doThrow(notFound()).doReturn("token").when(cloudWatchLogs).putLogEventsForAccount(eq(ACCOUNT), eq(GROUP), eq(CENTRAL_STREAM), anyList(), eq(REGION));- logs.open(ACCOUNT, REGION, GROUP, STREAM).accept("first\n");+ logs.open(ACCOUNT, REGION, GROUP, STREAM, "INFO").accept("first\n");ArgumentCaptor<List<Map<String, Object>>> local = ArgumentCaptor.forClass(List.class);verify(cloudWatchLogs, timeout(5_000).times(2)).putLogEventsForAccount(eq(ACCOUNT), eq(GROUP + ".local"),@@ -78,7 +79,7 @@ class LambdaWebLogsTest {doThrow(new AwsException("InvalidParameterException", "Invalid log group name.", 400)).when(cloudWatchLogs).createLogGroupForAccount(ACCOUNT, GROUP + ".local", null, null, REGION);- logs.open(ACCOUNT, REGION, GROUP, STREAM).accept("first\n");+ logs.open(ACCOUNT, REGION, GROUP, STREAM, "INFO").accept("first\n");verify(cloudWatchLogs, timeout(5_000)).putLogEventsForAccount(eq(ACCOUNT), eq(GROUP), eq(CENTRAL_STREAM),anyList(), eq(REGION));@@ -95,7 +96,7 @@ class LambdaWebLogsTest {throw new IllegalStateException("the store failed");}).doReturn("token").when(cloudWatchLogs).putLogEventsForAccount(eq(ACCOUNT), eq(GROUP + ".local"), eq(STREAM), anyList(), eq(REGION));- LambdaWebLogs.Sink sink = logs.open(ACCOUNT, REGION, GROUP, STREAM);+ LambdaWebLogs.Sink sink = logs.open(ACCOUNT, REGION, GROUP, STREAM, "INFO");sink.accept("first\n");writing.await(5, TimeUnit.SECONDS);@@ -106,6 +107,41 @@ class LambdaWebLogsTest {argThat(batch -> batch.size() == 1 && "second\n".equals(batch.getFirst().get("message"))), eq(REGION));}+ @Test+ @SuppressWarnings("unchecked")+ void eventsDroppedWhileTheQueueIsFullAreReportedBeforeTheNextOne() throws Exception {+ CountDownLatch writing = new CountDownLatch(1);+ CountDownLatch release = new CountDownLatch(1);+ doAnswer(invocation -> {+ writing.countDown();+ release.await(5, TimeUnit.SECONDS);+ return "token";+ }).doReturn("token").when(cloudWatchLogs)+ .putLogEventsForAccount(eq(ACCOUNT), eq(GROUP + ".local"), eq(STREAM), anyList(), eq(REGION));+ LambdaWebLogs.Sink sink = logs.open(ACCOUNT, REGION, GROUP, STREAM, "INFO");+ sink.accept("first\n");+ writing.await(5, TimeUnit.SECONDS);++ for (int i = 0; i < LambdaWebLogs.MAX_QUEUED_EVENTS + 3; i++) {+ sink.accept("queued\n");+ }+ release.countDown();+ int drained = 1 + LambdaWebLogs.MAX_QUEUED_EVENTS / 1_000;+ verify(cloudWatchLogs, timeout(5_000).times(drained)).putLogEventsForAccount(eq(ACCOUNT), eq(GROUP + ".local"),+ eq(STREAM), anyList(), eq(REGION));+ sink.accept("after\n");++ ArgumentCaptor<List<Map<String, Object>>> batches = ArgumentCaptor.forClass(List.class);+ verify(cloudWatchLogs, timeout(5_000).times(drained + 1)).putLogEventsForAccount(eq(ACCOUNT),+ eq(GROUP + ".local"), eq(STREAM), batches.capture(), eq(REGION));+ List<String> messages = batches.getAllValues().stream().flatMap(List::stream)+ .map(event -> (String) event.get("message")).toList();+ String dropped = messages.get(messages.size() - 2);+ assertTrue(dropped.contains("\"level\":\"WARN\",\"type\":\"platform.logsDropped\""), dropped);+ assertTrue(dropped.endsWith("\"droppedRecords\":3,\"droppedBytes\":21}}"), dropped);+ assertEquals("after\n", messages.getLast());+ }+private static AwsException notFound() {return new AwsException("ResourceNotFoundException", "The specified log stream does not exist.", 400);}diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebPlatformLogTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebPlatformLogTest.javanew file mode 100644index 0000000000..4add17135a--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebPlatformLogTest.java@@ -0,0 +1,109 @@+package io.github.hectorvent.floci.services.lambdaweb;++import io.github.hectorvent.floci.services.cloudwatch.logs.CloudWatchLogsService;+import io.restassured.path.json.JsonPath;+import org.junit.jupiter.api.AfterEach;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Test;+import org.junit.jupiter.api.extension.ExtendWith;+import org.mockito.ArgumentCaptor;+import org.mockito.Mock;+import org.mockito.junit.jupiter.MockitoExtension;++import java.time.Instant;+import java.util.List;+import java.util.Map;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.mockito.ArgumentMatchers.eq;+import static org.mockito.Mockito.timeout;+import static org.mockito.Mockito.verify;++@ExtendWith(MockitoExtension.class)+class LambdaWebPlatformLogTest {++ private static final String GROUP = "/aws/lambda/web/web";+ private static final String STREAM = "2026/10/18/web[rev-abcde]0123456789abcdef0123456789abcdef";++ @Mock+ CloudWatchLogsService cloudWatchLogs;++ private LambdaWebLogs logs;++ @BeforeEach+ void start() {+ logs = new LambdaWebLogs(cloudWatchLogs);+ }++ @AfterEach+ void stop() {+ logs.shutdown();+ }++ @Test+ void timesDropTheTrailingZerosOfTheFraction() {+ assertEquals("2026-10-18T13:49:46.67Z", LambdaWebLogs.timestamp(Instant.parse("2026-10-18T13:49:46.670Z"), 3));+ assertEquals("2026-10-18T13:49:33.4Z", LambdaWebLogs.timestamp(Instant.parse("2026-10-18T13:49:33.400999Z"), 3));+ assertEquals("2026-10-18T13:49:33Z", LambdaWebLogs.timestamp(Instant.parse("2026-10-18T13:49:33.000Z"), 3));+ assertEquals("2026-10-18T13:50:09.0405147Z",+ LambdaWebLogs.timestamp(Instant.parse("2026-10-18T13:50:09.040514700Z"), 9));+ }++ @Test+ void aRequestIsReportedWithItsDurationInMillisecondsToThreeDecimals() {+ LambdaWebPlatformLog platform = new LambdaWebPlatformLog(+ logs.open("111122223333", "eu-west-1", GROUP, STREAM, "DEBUG"), "eu-west-1", "web", "rev-abcde",+ "nodejs24.x", "INFO");++ platform.requestComplete("KXH25FBDZ33GP4TQ4ARAUYSTSA", "dev", 504, "Runtime.RequestTimeout", 3_000_716_400L, 0);++ JsonPath event = JsonPath.from(firstMessage());+ assertEquals("DEBUG", event.getString("level"));+ assertEquals("platform.requestComplete", event.getString("type"));+ assertEquals("eu-west-1", event.getString("region"));+ assertTrue(firstMessage().endsWith("\"result\":504,\"errorType\":\"Runtime.RequestTimeout\","+ + "\"metrics\":{\"durationMs\":3000.716,\"responseBytes\":0}}}"), firstMessage());+ }++ @Test+ void eventsBelowTheSystemLogLevelAreDroppedAndAFailedReportIsWarn() {+ LambdaWebPlatformLog platform = new LambdaWebPlatformLog(+ logs.open("111122223333", "eu-west-1", GROUP, STREAM, "WARN"), "eu-west-1", "web", "rev-abcde",+ "nodejs24.x", "INFO");+ platform.initStart();+ platform.initReport(null, new LambdaWebPlatformLog.Usage(178_400, 142L * 1024 * 1024 + 5));+ platform.invokeStart(1);+ platform.concurrency(3);++ platform.end(null, "Runtime.ExitError");++ JsonPath event = JsonPath.from(firstMessage());+ assertEquals("WARN", event.getString("level"));+ assertEquals("platform.invokeReport", event.getString("type"));+ assertEquals("failure", event.getString("record.status"));+ assertEquals(3, event.getInt("record.metrics.concurrentRequests"));+ assertEquals(142, event.getInt("record.metrics.maxMemoryUsageMb"));+ assertTrue(firstMessage().contains("\"totalCpuTimeMs\":0.000,"), firstMessage());+ }++ @Test+ void theListenFailureIsDroppedAtFatal() {+ LambdaWebPlatformLog platform = new LambdaWebPlatformLog(+ logs.open("111122223333", "eu-west-1", GROUP, STREAM, "DEBUG"), "eu-west-1", "web", "rev-abcde",+ "nodejs24.x", "FATAL");++ platform.applicationError("Application.TimeoutListenError: nothing was listening");+ platform.spinDownStart();++ assertEquals("platform.spinDownStart", JsonPath.from(firstMessage()).getString("type"));+ }++ @SuppressWarnings("unchecked")+ private String firstMessage() {+ ArgumentCaptor<List<Map<String, Object>>> batch = ArgumentCaptor.forClass(List.class);+ verify(cloudWatchLogs, timeout(5_000).atLeastOnce()).putLogEventsForAccount(eq("111122223333"),+ eq(GROUP + ".local"), eq(STREAM), batch.capture(), eq("eu-west-1"));+ return (String) batch.getAllValues().getFirst().getFirst().get("message");+ }+}