Light, fluffy, and always free - The AWS Local Emulator alternative
feat(lambdaweb): send what a Web Function writes to its log groups as AWS does (#5919)
TATSUNO “Taz” Yasuhiro8067ecf1de79
diff --git a/docs/services/lambda-web-functions.md b/docs/services/lambda-web-functions.mdindex 64b02f7eca..f253d16e6b 100644--- a/docs/services/lambda-web-functions.md+++ b/docs/services/lambda-web-functions.md@@ -41,7 +41,7 @@ The `Lambda Web` service model (apiVersion 2025-03-07, `aws lambda-web` in the A- **ARNs** given as identifiers must name the request's region and the caller's account, as on AWS. A revision or endpoint ARN resolves by its id or name only.- **Page tokens** are base64 cursors that carry the request and the position of the page, not the UUIDs AWS issues. A token reused with other parameters gets the AWS "does not match the request" error, and any other token the "invalid or has expired" error.- **Domain names** take the form `<id>.001.lambda-web.<region>.<floci host>` and `<id>.lambda-web.global.<floci host>`, with the host and port of `floci.base-url`. A `PerRegion` endpoint gets one domain per region in `regionalEndpoints`.-- **Not implemented yet:** SigV4a on `MultiRegion` endpoints, the check that a policy's principal exists, the automatic deletion of revisions past 50, the metadata API, the logs and metrics, and serving a `MultiRegion` or `PerRegion` endpoint in its other regions.+- **Not implemented yet:** SigV4a on `MultiRegion` endpoints, the check that a policy's principal exists, the automatic deletion of revisions past 50, the metadata API, the platform log events, the metrics, and serving a `MultiRegion` or `PerRegion` endpoint in its other regions.### `aws lambda-web deploy`@@ -64,7 +64,7 @@ curl -H "Host: <domainName>" http://localhost:4566/The `Host` must match the domain exactly, as on AWS: another case or another port is `404` with `lambda-web-error-type: Platform.InvalidEndpoint`.-An execution environment is a `public.ecr.aws/lambda/nodejs:24` container that a request starts when every environment of the endpoint is full. The code is the revision's S3 object unpacked into `/var/task`, the entry file is `index.js` or the file `AWS_LAMBDA_NODEJS_ENTRYPOINT` names, and node runs with the flags AWS uses. The app sees `AWS_LAMBDA_HTTP_ENDPOINT=0.0.0.0:3000`, `AWS_LAMBDA_INITIALIZATION_TYPE=native-http`, `AWS_EXECUTION_ENV`, `AWS_LAMBDA_FUNCTION_MEMORY_SIZE=2048`, `AWS_LAMBDA_LOG_LEVEL`, `AWS_ACCOUNT_ID`, `AWS_REGION`, credentials and `AWS_ENDPOINT_URL` for Floci, `NODE_PATH`, and its own variables. The server has 10 seconds to listen. An environment that sees no request for `floci.services.lambda.container-idle-timeout-seconds` is stopped with `SIGTERM` and one second, as on scale to zero. A process that exits between requests is replaced by a fresh environment on the next request.+An execution environment is a `public.ecr.aws/lambda/nodejs:24` container that a request starts when every environment of the endpoint is full. The code is the revision's S3 object unpacked into `/var/task`. Node starts `/var/runtime/index.js` with the flags AWS uses, as on AWS, and that runtime imports `index.js` or the file `AWS_LAMBDA_NODEJS_ENTRYPOINT` names. So `require.main` is not the app's module, as on AWS. The app sees `AWS_LAMBDA_HTTP_ENDPOINT=0.0.0.0:3000`, `AWS_LAMBDA_INITIALIZATION_TYPE=native-http`, `AWS_EXECUTION_ENV`, `AWS_LAMBDA_FUNCTION_MEMORY_SIZE=2048`, `AWS_LAMBDA_LOG_LEVEL`, `AWS_ACCOUNT_ID`, `AWS_REGION`, credentials and `AWS_ENDPOINT_URL` for Floci, `NODE_PATH`, and its own variables. The server has 10 seconds to listen. An environment that sees no request for `floci.services.lambda.container-idle-timeout-seconds` is stopped with `SIGTERM` and one second, as on scale to zero. A process that exits between requests is replaced by a fresh environment on the next request.Each endpoint scales on its own, as on AWS. An environment takes up to the revision's `maxConcurrencyPerEnvironment` requests at once, the least busy one first, so requests rotate over the environments. A request that finds every environment full starts another one, or waits for one that is starting and has room. When the endpoint already runs `maxEnvironments`, an idle environment of another revision makes room, and without one the request is refused with `429` and `Platform.EndpointMaxEnvThrottle`. A `rateLimit` admits a burst of up to `rateLimit` requests that refills at `rateLimit` per second, and refuses the rest with `429` and `Platform.EndpointRequestThrottle`. AWS adjusts the concurrency of each environment under `maxConcurrencyPerEnvironment` from the measured load, so a burst there can spread over more environments than in Floci, and the account quotas are not applied.@@ -93,6 +93,14 @@ The platform answers these cases itself, with the AWS HTML pages and `lambda-webA request head over 128 KiB is refused with `431`, without the AWS page. A request without a `Host` header cannot be told apart from one for another Floci service, so it does not get `Platform.MissingHostHeader`.+## Logs++What an environment writes goes to the revision's `logGroup`, by default `/aws/lambda/web/<function>`. Each event goes to the stream that `lambda-web-logstream` names in the `<logGroup>.local` group, and to the same stream with a `-<account>-<region>` suffix in `<logGroup>`, where AWS centralizes the regions an endpoint serves. A group and a stream are created when their first event arrives, with no retention.++Console output becomes the JSON events AWS writes, with `timestamp`, `level`, `message`, `region`, and `requestId` and `endpointName` for output made while a request runs. `console.log` and `console.info` are `INFO`, `console.debug` is `DEBUG`, `console.trace` is `TRACE` with the stack, `console.warn` is `WARN` and `console.error` is `ERROR`. One argument that is not a string goes into `message` as JSON, and an `Error` as `errorType`, `errorMessage` and `stackTrace`. Several arguments make one string, formatted as Node prints them. Output written to stdout or stderr directly, and `console.dir`, stays as it is, one event per line. Events below the revision's `applicationLogLevel` are dropped, and direct output counts as `INFO`. `FATAL` drops all console output, as on AWS. Bytes that are not UTF-8 become U+FFFD. A line longer than 262,118 bytes in UTF-8 is cut before the character that would pass it, and a line of only spaces makes no event.++The `platform.*` events that `systemLogLevel` filters are not written yet. The output of an environment that fails before it listens stays in Floci's own log. A stack trace shows the frames of Floci's runtime where AWS shows its own.+## CloudFormation`AWS::Lambda::WebFunction`, `AWS::Lambda::WebFunctionRevision` and `AWS::Lambda::WebFunctionEndpoint` provision through the resource registry. `Ref` of each is its ARN. Every property of a revision is createOnly, so a changed revision mints a new one and the displaced revision is deleted once the update commits. An endpoint keeps its identity while its function, name, type and regions stay the same. A changed type or region list under the same name is refused the way CloudFormation refuses to replace a custom-named resource. `Fn::GetAtt` on `RegionalEndpoints` is not supported yet.diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.javaindex 73514be00a..3c278d3d20 100644--- a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java@@ -6,6 +6,7 @@ import com.github.dockerjava.api.command.InspectContainerResponse;import com.github.dockerjava.api.command.WaitContainerResultCallback;import com.github.dockerjava.api.exception.NotFoundException;import com.github.dockerjava.api.model.Frame;+import com.github.dockerjava.api.model.StreamType;import io.github.hectorvent.floci.config.EmulatorConfig;import io.github.hectorvent.floci.core.common.ContainerTeardown;import io.github.hectorvent.floci.core.common.RequestScopes;@@ -35,6 +36,7 @@ import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;import org.jboss.logging.Logger;import java.io.IOException;+import java.io.InputStream;import java.io.OutputStream;import java.io.UncheckedIOException;import java.nio.charset.StandardCharsets;@@ -46,6 +48,7 @@ import java.time.ZoneOffset;import java.time.format.DateTimeFormatter;import java.util.ArrayList;import java.util.Comparator;+import java.util.EnumMap;import java.util.List;import java.util.Map;import java.util.Optional;@@ -65,14 +68,17 @@ import java.util.stream.Stream;/*** The execution environments of Web Function endpoints: Node.js server containers of a revision,* started by requests that find no room, reused while requests keep coming and removed after the- * Lambda idle timeout. The runtime loads {@code index.js}, or the file {@code AWS_LAMBDA_NODEJS_ENTRYPOINT} names,- * from {@code /var/task} with the node flags AWS uses.+ * Lambda idle timeout. Node starts {@code /var/runtime/index.js} with the flags AWS uses, as on AWS. That runtime+ * turns console output into log events and imports {@code index.js}, or the file {@code AWS_LAMBDA_NODEJS_ENTRYPOINT}+ * names, from {@code /var/task}. What the process writes goes to the revision's log group.*/@ApplicationScopedpublic class LambdaWebEnvironmentManager implements ContainerTeardown {static final int APP_PORT = 3000;static final String TASK_DIR = "/var/task";+ static final String RUNTIME_DIR = "/var/runtime";+ private static final String RUNTIME_RESOURCE = "lambdaweb/index.mjs";static final int MEMORY_MB = 2048;static final Duration LISTEN_TIMEOUT = Duration.ofSeconds(10);static final List<String> NODE_FLAGS = List.of("--expose-gc", "--max-http-header-size", "135168",@@ -104,6 +110,7 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {private volatile String host;private volatile int port;private volatile String logStream;+ private volatile LambdaWebLogs.Sink logs;private volatile SessionCreds roleCredentials;private final AtomicInteger inFlight = new AtomicInteger();private final AtomicLong lastUsed = new AtomicLong(System.nanoTime());@@ -202,6 +209,8 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {private final ZipExtractor zipExtractor;private final LaunchedContainerAwsEnv awsEnv;private final LambdaExecutionRoleCredentials executionRoleCredentials;+ private final LambdaWebLogs logs;+ private final byte[] runtime;/** The environments of each endpoint in each region. A list is only touched inside a map operation on its key. */private final ConcurrentHashMap<String, List<Environment>> pools = new ConcurrentHashMap<>();private final ScheduledExecutorService reaper = Executors.newSingleThreadScheduledExecutor(task -> {@@ -216,7 +225,7 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {@StreamingDocker DockerClient streamingDocker, ImageResolver imageResolver,EcrRegistryManager ecrRegistryManager, S3Service s3Service,ZipExtractor zipExtractor, LaunchedContainerAwsEnv awsEnv,- LambdaExecutionRoleCredentials executionRoleCredentials) {+ LambdaExecutionRoleCredentials executionRoleCredentials, LambdaWebLogs logs) {this.config = config;this.containerBuilder = containerBuilder;this.lifecycleManager = lifecycleManager;@@ -227,6 +236,8 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {this.zipExtractor = zipExtractor;this.awsEnv = awsEnv;this.executionRoleCredentials = executionRoleCredentials;+ this.logs = logs;+ this.runtime = runtimeScript();long period = Math.max(1, Math.min(30, config.services().lambda().containerIdleTimeoutSeconds() / 2 + 1));reaper.scheduleWithFixedDelay(this::reapIdle, period, period, TimeUnit.SECONDS);}@@ -427,7 +438,7 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {Map<String, String> userEnv = revision.serviceConfig.environmentVariables == null? Map.of() : revision.serviceConfig.environmentVariables;List<String> cmd = new ArrayList<>(NODE_FLAGS);- cmd.add(userEnv.getOrDefault("AWS_LAMBDA_NODEJS_ENTRYPOINT", "index.js"));+ cmd.add(RUNTIME_DIR + "/index.js");Optional<SessionCreds> roleCredentials = executionRoleCredentials.forRole(revision.serviceConfig.executionRoleArn, accountId, fn.functionName);ContainerSpec spec = containerBuilder.newContainer(image)@@ -453,6 +464,8 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {}try {copyCode(containerId, accountId, region, fn, revision);+ RetryingTarCopier.copyBytes(lifecycleManager.getDockerClient(), containerId, RUNTIME_DIR, "index.js",+ runtime, 0644);ContainerInfo info = lifecycleManager.startCreated(containerId, spec);EndpointInfo endpoint = info.getEndpoint(APP_PORT);awaitListening(containerId);@@ -462,7 +475,8 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {environment.logStream = LOG_STREAM_DATE.format(Instant.now()) + "/" + fn.functionName + "["+ revision.revisionId + "]" + UUID.randomUUID().toString().replace("-", "");environment.roleCredentials = roleCredentials.orElse(null);- watchExit(environment);+ environment.logs = logs.open(accountId, region, revision.serviceConfig.logGroup, environment.logStream);+ watchExit(environment, true);LOG.infov("Started the Lambda Web environment {0} of {1} revision {2} at {3}:{4}",containerId, fn.functionName, revision.revisionId, endpoint.host(), endpoint.port());} catch (RuntimeException e) {@@ -643,23 +657,34 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {/*** A process that exits between requests is invisible to clients: the next request starts anew.* The attach stream ends when the process exits, and Floci times that on its own clock. The exit- * time Docker records comes from the daemon's clock, which can be seconds off. The output is- * dropped here. A reader that falls behind would block the app's writes.+ * time Docker records comes from the daemon's clock, which can be seconds off. The stream also+ * carries the output, which only goes into a queue here. A reader that falls behind would block+ * the app's writes. {@code fromStart} replays what the app wrote while it started.*/- private void watchExit(Environment environment) {+ private void watchExit(Environment environment, boolean fromStart) {+ Map<StreamType, LambdaWebOutputLines> streams = new EnumMap<>(StreamType.class);try {streamingDocker.attachContainerCmd(environment.containerId)- .withStdOut(true).withStdErr(true).withFollowStream(true)+ .withStdOut(true).withStdErr(true).withFollowStream(true).withLogs(fromStart).exec(new ResultCallback.Adapter<Frame>() {+ @Override+ public void onNext(Frame frame) {+ LambdaWebOutputLines lines = streams.computeIfAbsent(frame.getStreamType(),+ type -> new LambdaWebOutputLines());+ lines.append(frame.getPayload()).forEach(event -> deliver(environment, event));+ }+@Overridepublic void onComplete() {super.onComplete();+ streams.values().forEach(lines -> lines.flush().forEach(event -> deliver(environment, event)));exited(environment, System.nanoTime());}@Overridepublic void onError(Throwable throwable) {super.onError(throwable);+ streams.values().forEach(lines -> lines.flush().forEach(event -> deliver(environment, event)));LOG.debugv(throwable, "The exit watch of the Lambda Web environment {0} dropped",environment.containerId);rearmExitWatch(environment);@@ -689,10 +714,28 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {exited(environment, System.nanoTime());return;}- watchExit(environment);+ watchExit(environment, false);}, EXIT_WATCH_REARM.toMillis(), TimeUnit.MILLISECONDS);}+ /** AWS writes no event for a line that is empty or only spaces. */+ private static void deliver(Environment environment, String event) {+ if (!event.isBlank()) {+ environment.logs.accept(event);+ }+ }++ private static byte[] runtimeScript() {+ try (InputStream in = Thread.currentThread().getContextClassLoader().getResourceAsStream(RUNTIME_RESOURCE)) {+ if (in == null) {+ throw new IllegalStateException(RUNTIME_RESOURCE + " is missing from the classpath");+ }+ return in.readAllBytes();+ } catch (IOException e) {+ throw new UncheckedIOException("Could not read " + RUNTIME_RESOURCE, e);+ }+ }+/** The environment leaves the pool before waiters hear of the exit, so the request after a failed one starts anew. */private void exited(Environment environment, long seenAt) {boolean running = environment.alive.compareAndSet(true, false);diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogs.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogs.javanew file mode 100644index 0000000000..ef3daab8e3--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogs.java@@ -0,0 +1,166 @@+package io.github.hectorvent.floci.services.lambdaweb;++import io.github.hectorvent.floci.core.common.AwsException;+import io.github.hectorvent.floci.services.cloudwatch.logs.CloudWatchLogsService;+import jakarta.annotation.PreDestroy;+import jakarta.enterprise.context.ApplicationScoped;+import jakarta.inject.Inject;+import org.jboss.logging.Logger;++import java.util.ArrayList;+import java.util.List;+import java.util.Map;+import java.util.concurrent.ConcurrentLinkedQueue;+import java.util.concurrent.ExecutorService;+import java.util.concurrent.Executors;+import java.util.concurrent.atomic.AtomicBoolean;+import java.util.concurrent.atomic.AtomicInteger;++/**+ * Delivers the log events of Web Function environments. Each event goes to the stream of its+ * environment in the {@code .local} companion of the revision's log group, and to the same stream+ * with an {@code -<account>-<region>} suffix in the log group itself, where AWS centralizes the+ * regions an endpoint serves. A group or stream is created when its first event arrives.+ */+@ApplicationScoped+public class LambdaWebLogs {++ private static final Logger LOG = Logger.getLogger(LambdaWebLogs.class);+ private static final int MAX_QUEUED_EVENTS = 10_000;+ private static final int MAX_BATCH_EVENTS = 1_000;++ private final CloudWatchLogsService logs;+ private final ExecutorService writers = Executors.newVirtualThreadPerTaskExecutor();++ @Inject+ public LambdaWebLogs(CloudWatchLogsService logs) {+ this.logs = logs;+ }++ public Sink open(String accountId, String region, String logGroup, String logStream) {+ return new Sink(accountId, region, List.of(+ new Destination(logGroup + ".local", logStream),+ new Destination(logGroup, logStream + "-" + accountId + "-" + region)));+ }++ @PreDestroy+ void shutdown() {+ writers.shutdownNow();+ }++ private record Destination(String logGroup, String logStream) {+ }++ private record Event(long timestamp, String message) {+ }++ /**+ * The events of one environment. {@link #accept} only queues, because the app blocks on its+ * output while the Docker stream that carries it waits. Past {@code MAX_QUEUED_EVENTS} events are dropped.+ */+ public final class Sink {+ private final String accountId;+ private final String region;+ private final List<Destination> destinations;+ private final ConcurrentLinkedQueue<Event> queue = new ConcurrentLinkedQueue<>();+ private final AtomicInteger queued = new AtomicInteger();+ private final AtomicBoolean writing = new AtomicBoolean();++ private Sink(String accountId, String region, List<Destination> destinations) {+ this.accountId = accountId;+ this.region = region;+ this.destinations = destinations;+ }++ public void accept(String message) {+ if (queued.incrementAndGet() > MAX_QUEUED_EVENTS) {+ queued.decrementAndGet();+ LOG.debugv("Dropped a log event of {0}: too many are waiting", destinations.getFirst().logStream());+ return;+ }+ queue.add(new Event(System.currentTimeMillis(), message));+ schedule();+ }++ private void schedule() {+ if (!writing.compareAndSet(false, true)) {+ return;+ }+ try {+ writers.execute(this::write);+ } catch (RuntimeException e) {+ writing.set(false);+ LOG.debugv(e, "Could not deliver the log events of {0}", destinations.getFirst().logStream());+ }+ }++ private void write() {+ try {+ List<Map<String, Object>> batch = new ArrayList<>();+ Event event;+ while ((event = queue.poll()) != null) {+ queued.decrementAndGet();+ batch.add(Map.of("timestamp", event.timestamp(), "message", event.message()));+ if (batch.size() == MAX_BATCH_EVENTS) {+ put(batch);+ batch = new ArrayList<>();+ }+ }+ if (!batch.isEmpty()) {+ put(batch);+ }+ } finally {+ writing.set(false);+ if (!queue.isEmpty()) {+ schedule();+ }+ }+ }++ private void put(List<Map<String, Object>> batch) {+ for (Destination destination : destinations) {+ try {+ putTo(destination, batch);+ } catch (RuntimeException e) {+ LOG.warnv(e, "Could not deliver {0} log events to {1}", batch.size(), destination.logGroup());+ }+ }+ }++ /** The group and the stream are created when an event finds one of them missing, also after a delete. */+ private void putTo(Destination destination, List<Map<String, Object>> batch) {+ try {+ logs.putLogEventsForAccount(accountId, destination.logGroup(), destination.logStream(), batch, region);+ } catch (AwsException e) {+ if (!"ResourceNotFoundException".equals(e.getErrorCode())) {+ throw e;+ }+ createLogGroup(destination);+ createLogStream(destination);+ logs.putLogEventsForAccount(accountId, destination.logGroup(), destination.logStream(), batch, region);+ }+ }++ private void createLogGroup(Destination destination) {+ try {+ logs.createLogGroupForAccount(accountId, destination.logGroup(), null, null, region);+ } catch (AwsException e) {+ ignoreAlreadyExists(e);+ }+ }++ private void createLogStream(Destination destination) {+ try {+ logs.createLogStreamForAccount(accountId, destination.logGroup(), destination.logStream(), region);+ } catch (AwsException e) {+ ignoreAlreadyExists(e);+ }+ }++ private static void ignoreAlreadyExists(AwsException e) {+ if (!"ResourceAlreadyExistsException".equals(e.getErrorCode())) {+ throw e;+ }+ }+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebOutputLines.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebOutputLines.javanew file mode 100644index 0000000000..71faa656a1--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebOutputLines.java@@ -0,0 +1,84 @@+package io.github.hectorvent.floci.services.lambdaweb;++import java.nio.ByteBuffer;+import java.nio.CharBuffer;+import java.nio.charset.CharsetDecoder;+import java.nio.charset.CodingErrorAction;+import java.nio.charset.StandardCharsets;+import java.util.ArrayList;+import java.util.List;++/**+ * Cuts what one output stream of an environment writes into log events. An event keeps its+ * newline. Bytes that are not UTF-8 become U+FFFD, and a line whose text is longer than+ * {@link #MAX_EVENT_BYTES} in UTF-8 is cut before the character that would pass it, without its+ * newline, as on AWS.+ */+final class LambdaWebOutputLines {++ /** The CloudWatch Logs event limit of 256 KiB less the 26 bytes it charges for each event. */+ static final int MAX_EVENT_BYTES = 262_118;++ private final CharsetDecoder decoder = StandardCharsets.UTF_8.newDecoder()+ .onMalformedInput(CodingErrorAction.REPLACE)+ .onUnmappableCharacter(CodingErrorAction.REPLACE);+ /** The start of a character whose other bytes come in the next payload. */+ private ByteBuffer undecoded = ByteBuffer.allocate(0);+ private final StringBuilder pending = new StringBuilder();+ private int pendingBytes;++ List<String> append(byte[] payload) {+ ByteBuffer in = ByteBuffer.allocate(undecoded.remaining() + payload.length).put(undecoded).put(payload).flip();+ CharBuffer out = CharBuffer.allocate(in.remaining());+ decoder.decode(in, out, false);+ undecoded = ByteBuffer.allocate(in.remaining()).put(in).flip();+ return cut(out.flip());+ }++ /** The output after the last newline, when the stream ends without one. */+ List<String> flush() {+ CharBuffer out = CharBuffer.allocate(undecoded.remaining() + 1);+ decoder.decode(undecoded, out, true);+ decoder.flush(out);+ decoder.reset();+ undecoded = ByteBuffer.allocate(0);+ List<String> events = cut(out.flip());+ if (!pending.isEmpty()) {+ events.add(take());+ }+ return events;+ }++ private List<String> cut(CharBuffer text) {+ List<String> events = new ArrayList<>();+ text.codePoints().forEach(codePoint -> {+ int length = utf8Length(codePoint);+ if (MAX_EVENT_BYTES < pendingBytes + length) {+ events.add(take());+ }+ pending.appendCodePoint(codePoint);+ pendingBytes += length;+ if (codePoint == '\n') {+ events.add(take());+ }+ });+ return events;+ }++ private static int utf8Length(int codePoint) {+ if (codePoint < 0x80) {+ return 1;+ }+ if (codePoint < 0x800) {+ return 2;+ }+ return codePoint < 0x10000 ? 3 : 4;+ }++ private String take() {+ String event = pending.toString();+ pending.setLength(0);+ pendingBytes = 0;+ return event;+ }+}diff --git a/src/main/resources/application.yml b/src/main/resources/application.ymlindex 93256dd9d7..7467be0b5c 100644--- a/src/main/resources/application.yml+++ b/src/main/resources/application.yml@@ -53,7 +53,8 @@ quarkus:# catalog (ec2/image-catalog.yaml), SageMaker reads its instance type# catalog (sagemaker/instance-type-catalog.yaml), KMS reads its AWS managed# key catalog (kms/aws-managed-keys.yaml), AppSync's GraphQL- # parser reads its message bundles, EC2 security group enforcement builds+ # parser reads its message bundles, Lambda Web Functions copy their+ # runtime (lambdaweb/index.mjs) into each environment, EC2 security group enforcement builds# its helper image from docker/*.Dockerfile, and API Gateway's Velocity# engine reads its default properties at runtime via# ClassLoader#getResourceAsStream. Quarkus's static resource analysis@@ -68,6 +69,7 @@ quarkus:- pricing-snapshots/**- ui/*.html- appsync/*.mjs+ - lambdaweb/*.mjs- ec2/*.yaml- docker/*.Dockerfile- sagemaker/*.yamldiff --git a/src/main/resources/lambdaweb/index.mjs b/src/main/resources/lambdaweb/index.mjsnew file mode 100644index 0000000000..654ed40b22--- /dev/null+++ b/src/main/resources/lambdaweb/index.mjs@@ -0,0 +1,96 @@+// The runtime entry Floci starts as /var/runtime/index.js, where AWS starts its own. It turns+// console output into the JSON log events AWS writes, with the request each one belongs to,+// filters the output by applicationLogLevel, then imports the app.+import { AsyncLocalStorage } from 'node:async_hooks';+import http from 'node:http';+import path from 'node:path';+import { pathToFileURL } from 'node:url';+import util from 'node:util';++// FATAL is above ERROR. Node has no console method for it, so it drops every console call.+const LEVELS = { TRACE: 0, DEBUG: 1, INFO: 2, WARN: 3, ERROR: 4, FATAL: 5 };+const minimum = LEVELS[process.env.AWS_LAMBDA_LOG_LEVEL] ?? LEVELS.INFO;+const region = process.env.AWS_REGION;+const requests = new AsyncLocalStorage();+const writeOut = process.stdout.write.bind(process.stdout);+const writeErr = process.stderr.write.bind(process.stderr);++const emit = http.Server.prototype.emit;+http.Server.prototype.emit = function (event, request) {+ if (event !== 'request') {+ return emit.apply(this, arguments);+ }+ const context = {+ requestId: request.headers['lambda-web-request-id'],+ endpointName: request.headers['lambda-web-endpoint-name'],+ };+ return requests.run(context, () => emit.apply(this, arguments));+};++function withErrors(key, value) {+ if (!(value instanceof Error)) {+ return value;+ }+ return { errorType: value.name, errorMessage: value.message, stackTrace: String(value.stack).split('\n'), ...value };+}++function line(level, message) {+ const context = requests.getStore();+ const event = {+ timestamp: new Date().toISOString(),+ level,+ message,+ requestId: context?.requestId,+ endpointName: context?.endpointName,+ region,+ };+ return JSON.stringify(event, withErrors) + '\n';+}++function render(level, args) {+ if (args.length !== 1 || typeof args[0] === 'string') {+ return line(level, args.length === 0 ? undefined : util.format(...args));+ }+ try {+ return line(level, args[0]);+ } catch (unserializable) {+ // A BigInt or a cycle is not JSON. AWS writes the text Node prints for it.+ return line(level, util.format(...args));+ }+}++function patch(name, level, write) {+ console[name] = function (...args) {+ if (LEVELS[level] >= minimum) {+ write(render(level, args));+ }+ };+}++patch('log', 'INFO', writeOut);+patch('info', 'INFO', writeOut);+patch('debug', 'DEBUG', writeOut);+patch('warn', 'WARN', writeErr);+patch('error', 'ERROR', writeErr);+console.trace = function (...args) {+ if (LEVELS.TRACE >= minimum) {+ const frames = String(new Error().stack).split('\n').slice(1).filter((frame) => !frame.includes(import.meta.url));+ writeOut(line('TRACE', { traceMessage: util.format(...args), stackTrace: frames }));+ }+};++// Output that does not go through console counts as INFO.+if (LEVELS.INFO < minimum) {+ const drop = (chunk, encoding, callback) => {+ const done = typeof encoding === 'function' ? encoding : callback;+ if (typeof done === 'function') {+ process.nextTick(done);+ }+ return true;+ };+ process.stdout.write = drop;+ process.stderr.write = drop;+}++const entry = process.env.AWS_LAMBDA_NODEJS_ENTRYPOINT || 'index.js';+await import(pathToFileURL(path.resolve('/var/task', entry)).href);diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.javaindex 7a0b041221..70c7c1592a 100644--- a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java@@ -3,12 +3,15 @@ package io.github.hectorvent.floci.services.lambdaweb;import com.github.dockerjava.api.DockerClient;import io.github.hectorvent.floci.services.lambdaweb.LambdaWebEnvironmentManager.Environment;import io.github.hectorvent.floci.services.s3.S3Service;+import io.github.hectorvent.floci.testing.RestAssuredJsonUtils;import io.quarkus.test.junit.QuarkusTest;import io.restassured.RestAssured;+import io.restassured.path.json.JsonPath;import io.restassured.response.Response;import jakarta.inject.Inject;import org.junit.jupiter.api.AfterEach;import org.junit.jupiter.api.Assumptions;+import org.junit.jupiter.api.BeforeAll;import org.junit.jupiter.api.BeforeEach;import org.junit.jupiter.api.Test;import org.junit.jupiter.api.Timeout;@@ -60,7 +63,18 @@ class LambdaWebDataPlaneDockerIntegrationTest {private static final String APP = """const http = require('node:http');const [host, port] = process.env.AWS_LAMBDA_HTTP_ENDPOINT.split(':');+ console.log('starting');http.createServer((req, res) => {+ if (req.url.startsWith('/log')) {+ console.info({route: 'log', main: require.main === module, argv: process.argv.slice(1)});+ console.debug('below the level');+ console.error(new Error('logged error'));+ process.stdout.write('raw line\\n');+ process.stdout.write(' \\n');+ process.stdout.write('last line\\n');+ res.end('logged');+ return;+ }if (req.url.startsWith('/slow')) { setTimeout(() => res.end('late'), 8000); return; }if (req.url.startsWith('/wait')) { setTimeout(() => res.end('waited'), 3000); return; }if (req.url.startsWith('/stream')) {@@ -100,6 +114,11 @@ class LambdaWebDataPlaneDockerIntegrationTest {private String functionName;+ @BeforeAll+ static void configureRestAssured() {+ RestAssuredJsonUtils.configureAwsContentTypes();+ }+@BeforeEachvoid requireDocker() {Assumptions.assumeTrue(isDockerAvailable(), "Docker must be available for the Lambda Web data plane");@@ -181,6 +200,38 @@ class LambdaWebDataPlaneDockerIntegrationTest {.body("headers.x-amz-date", equalTo("20260101T000000Z"));}+ @Test+ @Timeout(180)+ void theOutputGoesToTheLocalAndTheCentralizedLogGroupAsJsonEvents() throws Exception {+ String domain = createFunction(30);+ Response logged = given().header("Host", domain).when().get("/log");+ logged.then().statusCode(200);+ String logStream = logged.header("lambda-web-logstream");+ String group = "/aws/lambda/web/" + functionName;++ List<String> local = awaitLogEvents(group + ".local", logStream);+ assertEquals(5, local.size(), local.toString());+ JsonPath starting = JsonPath.from(local.getFirst());+ assertEquals("INFO", starting.getString("level"));+ assertEquals("starting", starting.getString("message"));+ assertEquals("us-east-1", starting.getString("region"));+ assertFalse(local.getFirst().contains("requestId"), local.getFirst());+ JsonPath info = JsonPath.from(local.stream().filter(event -> event.contains("\"route\"")).findFirst().orElseThrow());+ assertEquals(logged.header("lambda-web-request-id"), info.getString("requestId"));+ assertEquals("dev", info.getString("endpointName"));+ assertEquals(false, info.getBoolean("message.main"));+ assertEquals(List.of("/var/runtime/index.js"), info.getList("message.argv"));+ JsonPath error = JsonPath.from(local.stream().filter(event -> event.contains("ERROR")).findFirst().orElseThrow());+ assertEquals("Error", error.getString("message.errorType"));+ assertEquals("logged error", error.getString("message.errorMessage"));+ assertEquals("Error: logged error", error.getList("message.stackTrace").getFirst());+ assertTrue(local.contains("raw line\n"), local.toString());+ assertTrue(local.stream().allMatch(event -> event.endsWith("\n")), local.toString());++ List<String> centralized = awaitLogEvents(group, logStream + "-000000000000-us-east-1");+ assertEquals(new HashSet<>(local), new HashSet<>(centralized));+ }+@Test@Timeout(180)void anIamAuthEndpointServesSignedRequestsWithoutTheirSignature() throws Exception {@@ -368,6 +419,29 @@ class LambdaWebDataPlaneDockerIntegrationTest {.extract().path("endpoint.domainName");}+ /**+ * The messages of a log stream once the last output of {@code /log} on stdout and on stderr is in it, or what it+ * holds after 15 seconds. The output of one stream arrives in order, so nothing written before comes later.+ */+ private static List<String> awaitLogEvents(String logGroup, String logStream) throws InterruptedException {+ long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(15);+ List<String> messages = logEvents(logGroup, logStream);+ while (!(messages.contains("last line\n") && messages.stream().anyMatch(event -> event.contains("\"ERROR\"")))+ && System.nanoTime() < deadline) {+ Thread.sleep(200);+ messages = logEvents(logGroup, logStream);+ }+ return messages;+ }++ private static List<String> logEvents(String logGroup, String logStream) {+ List<String> messages = RestAssuredJsonUtils.awsAction("Logs_20140328", "GetLogEvents",+ "{\"logGroupName\": \"%s\", \"logStreamName\": \"%s\", \"startFromHead\": true}"+ .formatted(logGroup, logStream))+ .then().extract().jsonPath().getList("events.message", String.class);+ return messages == null ? List.of() : messages;+ }+/** Sends one raw HTTP/1.1 request with {@code Connection: close} and reads until the server closes. */private static String raw(String request) throws Exception {return raw(request, null);diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogsTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogsTest.javanew file mode 100644index 0000000000..361b3a1fae--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogsTest.java@@ -0,0 +1,112 @@+package io.github.hectorvent.floci.services.lambdaweb;++import io.github.hectorvent.floci.core.common.AwsException;+import io.github.hectorvent.floci.services.cloudwatch.logs.CloudWatchLogsService;+import org.junit.jupiter.api.AfterEach;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Test;+import org.junit.jupiter.api.extension.ExtendWith;+import org.mockito.ArgumentCaptor;+import org.mockito.Mock;+import org.mockito.junit.jupiter.MockitoExtension;++import java.util.List;+import java.util.Map;+import java.util.concurrent.CountDownLatch;+import java.util.concurrent.TimeUnit;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.mockito.ArgumentMatchers.anyList;+import static org.mockito.ArgumentMatchers.argThat;+import static org.mockito.ArgumentMatchers.eq;+import static org.mockito.Mockito.doAnswer;+import static org.mockito.Mockito.doThrow;+import static org.mockito.Mockito.never;+import static org.mockito.Mockito.timeout;+import static org.mockito.Mockito.verify;++@ExtendWith(MockitoExtension.class)+class LambdaWebLogsTest {++ private static final String ACCOUNT = "111122223333";+ private static final String REGION = "eu-west-1";+ private static final String GROUP = "/aws/lambda/web/web";+ private static final String STREAM = "2026/10/18/web[rev-abcde]0123456789abcdef0123456789abcdef";+ private static final String CENTRAL_STREAM = STREAM + "-" + ACCOUNT + "-" + REGION;++ @Mock+ CloudWatchLogsService cloudWatchLogs;++ private LambdaWebLogs logs;++ @BeforeEach+ void start() {+ logs = new LambdaWebLogs(cloudWatchLogs);+ }++ @AfterEach+ void stop() {+ logs.shutdown();+ }++ @Test+ @SuppressWarnings("unchecked")+ void eventsGoToTheLocalAndTheCentralizedGroupCreatedWhenMissing() {+ doThrow(notFound()).doReturn("token").when(cloudWatchLogs)+ .putLogEventsForAccount(eq(ACCOUNT), eq(GROUP + ".local"), eq(STREAM), anyList(), eq(REGION));+ doThrow(notFound()).doReturn("token").when(cloudWatchLogs)+ .putLogEventsForAccount(eq(ACCOUNT), eq(GROUP), eq(CENTRAL_STREAM), anyList(), eq(REGION));++ logs.open(ACCOUNT, REGION, GROUP, STREAM).accept("first\n");++ ArgumentCaptor<List<Map<String, Object>>> local = ArgumentCaptor.forClass(List.class);+ verify(cloudWatchLogs, timeout(5_000).times(2)).putLogEventsForAccount(eq(ACCOUNT), eq(GROUP + ".local"),+ eq(STREAM), local.capture(), eq(REGION));+ verify(cloudWatchLogs, timeout(5_000).times(2)).putLogEventsForAccount(eq(ACCOUNT), eq(GROUP),+ eq(CENTRAL_STREAM), anyList(), eq(REGION));+ verify(cloudWatchLogs).createLogGroupForAccount(ACCOUNT, GROUP + ".local", null, null, REGION);+ verify(cloudWatchLogs).createLogStreamForAccount(ACCOUNT, GROUP + ".local", STREAM, REGION);+ verify(cloudWatchLogs).createLogGroupForAccount(ACCOUNT, GROUP, null, null, REGION);+ verify(cloudWatchLogs).createLogStreamForAccount(ACCOUNT, GROUP, CENTRAL_STREAM, REGION);+ assertEquals("first\n", local.getValue().getFirst().get("message"));+ }++ @Test+ void aGroupThatCannotBeCreatedTakesNoEventsAndTheOtherStillDoes() {+ doThrow(notFound()).when(cloudWatchLogs)+ .putLogEventsForAccount(eq(ACCOUNT), eq(GROUP + ".local"), eq(STREAM), anyList(), eq(REGION));+ doThrow(new AwsException("InvalidParameterException", "Invalid log group name.", 400)).when(cloudWatchLogs)+ .createLogGroupForAccount(ACCOUNT, GROUP + ".local", null, null, REGION);++ logs.open(ACCOUNT, REGION, GROUP, STREAM).accept("first\n");++ verify(cloudWatchLogs, timeout(5_000)).putLogEventsForAccount(eq(ACCOUNT), eq(GROUP), eq(CENTRAL_STREAM),+ anyList(), eq(REGION));+ verify(cloudWatchLogs, never()).createLogStreamForAccount(ACCOUNT, GROUP + ".local", STREAM, REGION);+ }++ @Test+ void eventsQueuedWhileAWriteFailsAreStillDelivered() throws Exception {+ CountDownLatch writing = new CountDownLatch(1);+ CountDownLatch queued = new CountDownLatch(1);+ doAnswer(invocation -> {+ writing.countDown();+ queued.await(5, TimeUnit.SECONDS);+ throw new IllegalStateException("the store failed");+ }).doReturn("token").when(cloudWatchLogs)+ .putLogEventsForAccount(eq(ACCOUNT), eq(GROUP + ".local"), eq(STREAM), anyList(), eq(REGION));+ LambdaWebLogs.Sink sink = logs.open(ACCOUNT, REGION, GROUP, STREAM);++ sink.accept("first\n");+ writing.await(5, TimeUnit.SECONDS);+ sink.accept("second\n");+ queued.countDown();++ verify(cloudWatchLogs, timeout(5_000)).putLogEventsForAccount(eq(ACCOUNT), eq(GROUP + ".local"), eq(STREAM),+ argThat(batch -> batch.size() == 1 && "second\n".equals(batch.getFirst().get("message"))), eq(REGION));+ }++ private static AwsException notFound() {+ return new AwsException("ResourceNotFoundException", "The specified log stream does not exist.", 400);+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebOutputLinesTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebOutputLinesTest.javanew file mode 100644index 0000000000..51b33831f0--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebOutputLinesTest.java@@ -0,0 +1,83 @@+package io.github.hectorvent.floci.services.lambdaweb;++import org.junit.jupiter.api.Test;++import java.io.ByteArrayOutputStream;+import java.nio.charset.StandardCharsets;+import java.util.Arrays;+import java.util.List;++import static org.junit.jupiter.api.Assertions.assertEquals;++class LambdaWebOutputLinesTest {++ private static final int MAX = LambdaWebOutputLines.MAX_EVENT_BYTES;++ @Test+ void eachLineIsAnEventWithItsNewline() {+ LambdaWebOutputLines lines = new LambdaWebOutputLines();++ assertEquals(List.of("one\n", "two\n"), lines.append(bytes("one\ntwo\nthr")));+ assertEquals(List.of("three\n"), lines.append(bytes("ee\n")));+ assertEquals(List.of(), lines.flush());+ }++ @Test+ void theOutputAfterTheLastNewlineComesOutWhenTheStreamEnds() {+ LambdaWebOutputLines lines = new LambdaWebOutputLines();++ assertEquals(List.of(), lines.append(bytes("no newline")));+ assertEquals(List.of("no newline"), lines.flush());+ }++ @Test+ void aLongLineIsCutBeforeTheCharacterThatWouldPassTheLimit() {+ LambdaWebOutputLines lines = new LambdaWebOutputLines();++ assertEquals(List.of("y".repeat(MAX - 2), "日tail\n"), lines.append(bytes("y".repeat(MAX - 2) + "日tail\n")));+ assertEquals(List.of("z".repeat(MAX), "z\n"), lines.append(bytes("z".repeat(MAX + 1) + "\n")));+ }++ @Test+ void bytesThatAreNotUtf8BecomeReplacementCharactersCountedAtTheirEncodedSize() {+ LambdaWebOutputLines lines = new LambdaWebOutputLines();++ assertEquals(List.of("a�b��c\n"),+ lines.append(concat(bytes("a"), new byte[] {(byte) 0x80}, bytes("b"),+ new byte[] {(byte) 0xFF, (byte) 0xFE}, bytes("c\n"))));+ assertEquals(List.of("cut � end\n"),+ lines.append(concat(bytes("cut "), new byte[] {(byte) 0xE6, (byte) 0x97}, bytes(" end\n"))));+ assertEquals(List.of("x".repeat(MAX - 1), "����tail\n"),+ lines.append(concat(bytes("x".repeat(MAX - 1)), new byte[] {(byte) 0x80, (byte) 0x80, (byte) 0x80,+ (byte) 0x80}, bytes("tail\n"))));+ }++ @Test+ void aCharacterSplitAcrossPayloadsStaysWhole() {+ LambdaWebOutputLines lines = new LambdaWebOutputLines();+ byte[] word = bytes("日本\n");++ assertEquals(List.of(), lines.append(new byte[] {word[0], word[1]}));+ assertEquals(List.of("日本\n"), lines.append(Arrays.copyOfRange(word, 2, word.length)));+ }++ @Test+ void anUnfinishedCharacterAtTheEndBecomesAReplacementCharacter() {+ LambdaWebOutputLines lines = new LambdaWebOutputLines();++ assertEquals(List.of(), lines.append(new byte[] {'a', (byte) 0xE6, (byte) 0x97}));+ assertEquals(List.of("a�"), lines.flush());+ }++ private static byte[] bytes(String text) {+ return text.getBytes(StandardCharsets.UTF_8);+ }++ private static byte[] concat(byte[]... parts) {+ ByteArrayOutputStream out = new ByteArrayOutputStream();+ for (byte[] part : parts) {+ out.writeBytes(part);+ }+ return out.toByteArray();+ }+}