Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(lambdaweb): send what a Web Function writes to its log groups as AWS does (#5919)

TATSUNO “Taz” Yasuhiro8067ecf1de79
diff --git a/docs/services/lambda-web-functions.md b/docs/services/lambda-web-functions.md
index 64b02f7eca..f253d16e6b 100644
--- a/docs/services/lambda-web-functions.md
+++ b/docs/services/lambda-web-functions.md
@@ -41,7 +41,7 @@ The `Lambda Web` service model (apiVersion 2025-03-07, `aws lambda-web` in the A
- **ARNs** given as identifiers must name the request's region and the caller's account, as on AWS. A revision or endpoint ARN resolves by its id or name only.
- **Page tokens** are base64 cursors that carry the request and the position of the page, not the UUIDs AWS issues. A token reused with other parameters gets the AWS "does not match the request" error, and any other token the "invalid or has expired" error.
- **Domain names** take the form `<id>.001.lambda-web.<region>.<floci host>` and `<id>.lambda-web.global.<floci host>`, with the host and port of `floci.base-url`. A `PerRegion` endpoint gets one domain per region in `regionalEndpoints`.
-- **Not implemented yet:** SigV4a on `MultiRegion` endpoints, the check that a policy's principal exists, the automatic deletion of revisions past 50, the metadata API, the logs and metrics, and serving a `MultiRegion` or `PerRegion` endpoint in its other regions.
+- **Not implemented yet:** SigV4a on `MultiRegion` endpoints, the check that a policy's principal exists, the automatic deletion of revisions past 50, the metadata API, the platform log events, the metrics, and serving a `MultiRegion` or `PerRegion` endpoint in its other regions.
### `aws lambda-web deploy`
@@ -64,7 +64,7 @@ curl -H "Host: <domainName>" http://localhost:4566/
The `Host` must match the domain exactly, as on AWS: another case or another port is `404` with `lambda-web-error-type: Platform.InvalidEndpoint`.
-An execution environment is a `public.ecr.aws/lambda/nodejs:24` container that a request starts when every environment of the endpoint is full. The code is the revision's S3 object unpacked into `/var/task`, the entry file is `index.js` or the file `AWS_LAMBDA_NODEJS_ENTRYPOINT` names, and node runs with the flags AWS uses. The app sees `AWS_LAMBDA_HTTP_ENDPOINT=0.0.0.0:3000`, `AWS_LAMBDA_INITIALIZATION_TYPE=native-http`, `AWS_EXECUTION_ENV`, `AWS_LAMBDA_FUNCTION_MEMORY_SIZE=2048`, `AWS_LAMBDA_LOG_LEVEL`, `AWS_ACCOUNT_ID`, `AWS_REGION`, credentials and `AWS_ENDPOINT_URL` for Floci, `NODE_PATH`, and its own variables. The server has 10 seconds to listen. An environment that sees no request for `floci.services.lambda.container-idle-timeout-seconds` is stopped with `SIGTERM` and one second, as on scale to zero. A process that exits between requests is replaced by a fresh environment on the next request.
+An execution environment is a `public.ecr.aws/lambda/nodejs:24` container that a request starts when every environment of the endpoint is full. The code is the revision's S3 object unpacked into `/var/task`. Node starts `/var/runtime/index.js` with the flags AWS uses, as on AWS, and that runtime imports `index.js` or the file `AWS_LAMBDA_NODEJS_ENTRYPOINT` names. So `require.main` is not the app's module, as on AWS. The app sees `AWS_LAMBDA_HTTP_ENDPOINT=0.0.0.0:3000`, `AWS_LAMBDA_INITIALIZATION_TYPE=native-http`, `AWS_EXECUTION_ENV`, `AWS_LAMBDA_FUNCTION_MEMORY_SIZE=2048`, `AWS_LAMBDA_LOG_LEVEL`, `AWS_ACCOUNT_ID`, `AWS_REGION`, credentials and `AWS_ENDPOINT_URL` for Floci, `NODE_PATH`, and its own variables. The server has 10 seconds to listen. An environment that sees no request for `floci.services.lambda.container-idle-timeout-seconds` is stopped with `SIGTERM` and one second, as on scale to zero. A process that exits between requests is replaced by a fresh environment on the next request.
Each endpoint scales on its own, as on AWS. An environment takes up to the revision's `maxConcurrencyPerEnvironment` requests at once, the least busy one first, so requests rotate over the environments. A request that finds every environment full starts another one, or waits for one that is starting and has room. When the endpoint already runs `maxEnvironments`, an idle environment of another revision makes room, and without one the request is refused with `429` and `Platform.EndpointMaxEnvThrottle`. A `rateLimit` admits a burst of up to `rateLimit` requests that refills at `rateLimit` per second, and refuses the rest with `429` and `Platform.EndpointRequestThrottle`. AWS adjusts the concurrency of each environment under `maxConcurrencyPerEnvironment` from the measured load, so a burst there can spread over more environments than in Floci, and the account quotas are not applied.
@@ -93,6 +93,14 @@ The platform answers these cases itself, with the AWS HTML pages and `lambda-web
A request head over 128 KiB is refused with `431`, without the AWS page. A request without a `Host` header cannot be told apart from one for another Floci service, so it does not get `Platform.MissingHostHeader`.
+## Logs
+
+What an environment writes goes to the revision's `logGroup`, by default `/aws/lambda/web/<function>`. Each event goes to the stream that `lambda-web-logstream` names in the `<logGroup>.local` group, and to the same stream with a `-<account>-<region>` suffix in `<logGroup>`, where AWS centralizes the regions an endpoint serves. A group and a stream are created when their first event arrives, with no retention.
+
+Console output becomes the JSON events AWS writes, with `timestamp`, `level`, `message`, `region`, and `requestId` and `endpointName` for output made while a request runs. `console.log` and `console.info` are `INFO`, `console.debug` is `DEBUG`, `console.trace` is `TRACE` with the stack, `console.warn` is `WARN` and `console.error` is `ERROR`. One argument that is not a string goes into `message` as JSON, and an `Error` as `errorType`, `errorMessage` and `stackTrace`. Several arguments make one string, formatted as Node prints them. Output written to stdout or stderr directly, and `console.dir`, stays as it is, one event per line. Events below the revision's `applicationLogLevel` are dropped, and direct output counts as `INFO`. `FATAL` drops all console output, as on AWS. Bytes that are not UTF-8 become U+FFFD. A line longer than 262,118 bytes in UTF-8 is cut before the character that would pass it, and a line of only spaces makes no event.
+
+The `platform.*` events that `systemLogLevel` filters are not written yet. The output of an environment that fails before it listens stays in Floci's own log. A stack trace shows the frames of Floci's runtime where AWS shows its own.
+
## CloudFormation
`AWS::Lambda::WebFunction`, `AWS::Lambda::WebFunctionRevision` and `AWS::Lambda::WebFunctionEndpoint` provision through the resource registry. `Ref` of each is its ARN. Every property of a revision is createOnly, so a changed revision mints a new one and the displaced revision is deleted once the update commits. An endpoint keeps its identity while its function, name, type and regions stay the same. A changed type or region list under the same name is refused the way CloudFormation refuses to replace a custom-named resource. `Fn::GetAtt` on `RegionalEndpoints` is not supported yet.
diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java
index 73514be00a..3c278d3d20 100644
--- a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java
@@ -6,6 +6,7 @@ import com.github.dockerjava.api.command.InspectContainerResponse;
import com.github.dockerjava.api.command.WaitContainerResultCallback;
import com.github.dockerjava.api.exception.NotFoundException;
import com.github.dockerjava.api.model.Frame;
+import com.github.dockerjava.api.model.StreamType;
import io.github.hectorvent.floci.config.EmulatorConfig;
import io.github.hectorvent.floci.core.common.ContainerTeardown;
import io.github.hectorvent.floci.core.common.RequestScopes;
@@ -35,6 +36,7 @@ import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
import org.jboss.logging.Logger;
import java.io.IOException;
+import java.io.InputStream;
import java.io.OutputStream;
import java.io.UncheckedIOException;
import java.nio.charset.StandardCharsets;
@@ -46,6 +48,7 @@ import java.time.ZoneOffset;
import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
import java.util.Comparator;
+import java.util.EnumMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
@@ -65,14 +68,17 @@ import java.util.stream.Stream;
/**
* The execution environments of Web Function endpoints: Node.js server containers of a revision,
* started by requests that find no room, reused while requests keep coming and removed after the
- * Lambda idle timeout. The runtime loads {@code index.js}, or the file {@code AWS_LAMBDA_NODEJS_ENTRYPOINT} names,
- * from {@code /var/task} with the node flags AWS uses.
+ * Lambda idle timeout. Node starts {@code /var/runtime/index.js} with the flags AWS uses, as on AWS. That runtime
+ * turns console output into log events and imports {@code index.js}, or the file {@code AWS_LAMBDA_NODEJS_ENTRYPOINT}
+ * names, from {@code /var/task}. What the process writes goes to the revision's log group.
*/
@ApplicationScoped
public class LambdaWebEnvironmentManager implements ContainerTeardown {
static final int APP_PORT = 3000;
static final String TASK_DIR = "/var/task";
+ static final String RUNTIME_DIR = "/var/runtime";
+ private static final String RUNTIME_RESOURCE = "lambdaweb/index.mjs";
static final int MEMORY_MB = 2048;
static final Duration LISTEN_TIMEOUT = Duration.ofSeconds(10);
static final List<String> NODE_FLAGS = List.of("--expose-gc", "--max-http-header-size", "135168",
@@ -104,6 +110,7 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
private volatile String host;
private volatile int port;
private volatile String logStream;
+ private volatile LambdaWebLogs.Sink logs;
private volatile SessionCreds roleCredentials;
private final AtomicInteger inFlight = new AtomicInteger();
private final AtomicLong lastUsed = new AtomicLong(System.nanoTime());
@@ -202,6 +209,8 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
private final ZipExtractor zipExtractor;
private final LaunchedContainerAwsEnv awsEnv;
private final LambdaExecutionRoleCredentials executionRoleCredentials;
+ private final LambdaWebLogs logs;
+ private final byte[] runtime;
/** The environments of each endpoint in each region. A list is only touched inside a map operation on its key. */
private final ConcurrentHashMap<String, List<Environment>> pools = new ConcurrentHashMap<>();
private final ScheduledExecutorService reaper = Executors.newSingleThreadScheduledExecutor(task -> {
@@ -216,7 +225,7 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
@StreamingDocker DockerClient streamingDocker, ImageResolver imageResolver,
EcrRegistryManager ecrRegistryManager, S3Service s3Service,
ZipExtractor zipExtractor, LaunchedContainerAwsEnv awsEnv,
- LambdaExecutionRoleCredentials executionRoleCredentials) {
+ LambdaExecutionRoleCredentials executionRoleCredentials, LambdaWebLogs logs) {
this.config = config;
this.containerBuilder = containerBuilder;
this.lifecycleManager = lifecycleManager;
@@ -227,6 +236,8 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
this.zipExtractor = zipExtractor;
this.awsEnv = awsEnv;
this.executionRoleCredentials = executionRoleCredentials;
+ this.logs = logs;
+ this.runtime = runtimeScript();
long period = Math.max(1, Math.min(30, config.services().lambda().containerIdleTimeoutSeconds() / 2 + 1));
reaper.scheduleWithFixedDelay(this::reapIdle, period, period, TimeUnit.SECONDS);
}
@@ -427,7 +438,7 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
Map<String, String> userEnv = revision.serviceConfig.environmentVariables == null
? Map.of() : revision.serviceConfig.environmentVariables;
List<String> cmd = new ArrayList<>(NODE_FLAGS);
- cmd.add(userEnv.getOrDefault("AWS_LAMBDA_NODEJS_ENTRYPOINT", "index.js"));
+ cmd.add(RUNTIME_DIR + "/index.js");
Optional<SessionCreds> roleCredentials = executionRoleCredentials.forRole(
revision.serviceConfig.executionRoleArn, accountId, fn.functionName);
ContainerSpec spec = containerBuilder.newContainer(image)
@@ -453,6 +464,8 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
}
try {
copyCode(containerId, accountId, region, fn, revision);
+ RetryingTarCopier.copyBytes(lifecycleManager.getDockerClient(), containerId, RUNTIME_DIR, "index.js",
+ runtime, 0644);
ContainerInfo info = lifecycleManager.startCreated(containerId, spec);
EndpointInfo endpoint = info.getEndpoint(APP_PORT);
awaitListening(containerId);
@@ -462,7 +475,8 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
environment.logStream = LOG_STREAM_DATE.format(Instant.now()) + "/" + fn.functionName + "["
+ revision.revisionId + "]" + UUID.randomUUID().toString().replace("-", "");
environment.roleCredentials = roleCredentials.orElse(null);
- watchExit(environment);
+ environment.logs = logs.open(accountId, region, revision.serviceConfig.logGroup, environment.logStream);
+ watchExit(environment, true);
LOG.infov("Started the Lambda Web environment {0} of {1} revision {2} at {3}:{4}",
containerId, fn.functionName, revision.revisionId, endpoint.host(), endpoint.port());
} catch (RuntimeException e) {
@@ -643,23 +657,34 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
/**
* A process that exits between requests is invisible to clients: the next request starts anew.
* The attach stream ends when the process exits, and Floci times that on its own clock. The exit
- * time Docker records comes from the daemon's clock, which can be seconds off. The output is
- * dropped here. A reader that falls behind would block the app's writes.
+ * time Docker records comes from the daemon's clock, which can be seconds off. The stream also
+ * carries the output, which only goes into a queue here. A reader that falls behind would block
+ * the app's writes. {@code fromStart} replays what the app wrote while it started.
*/
- private void watchExit(Environment environment) {
+ private void watchExit(Environment environment, boolean fromStart) {
+ Map<StreamType, LambdaWebOutputLines> streams = new EnumMap<>(StreamType.class);
try {
streamingDocker.attachContainerCmd(environment.containerId)
- .withStdOut(true).withStdErr(true).withFollowStream(true)
+ .withStdOut(true).withStdErr(true).withFollowStream(true).withLogs(fromStart)
.exec(new ResultCallback.Adapter<Frame>() {
+ @Override
+ public void onNext(Frame frame) {
+ LambdaWebOutputLines lines = streams.computeIfAbsent(frame.getStreamType(),
+ type -> new LambdaWebOutputLines());
+ lines.append(frame.getPayload()).forEach(event -> deliver(environment, event));
+ }
+
@Override
public void onComplete() {
super.onComplete();
+ streams.values().forEach(lines -> lines.flush().forEach(event -> deliver(environment, event)));
exited(environment, System.nanoTime());
}
@Override
public void onError(Throwable throwable) {
super.onError(throwable);
+ streams.values().forEach(lines -> lines.flush().forEach(event -> deliver(environment, event)));
LOG.debugv(throwable, "The exit watch of the Lambda Web environment {0} dropped",
environment.containerId);
rearmExitWatch(environment);
@@ -689,10 +714,28 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
exited(environment, System.nanoTime());
return;
}
- watchExit(environment);
+ watchExit(environment, false);
}, EXIT_WATCH_REARM.toMillis(), TimeUnit.MILLISECONDS);
}
+ /** AWS writes no event for a line that is empty or only spaces. */
+ private static void deliver(Environment environment, String event) {
+ if (!event.isBlank()) {
+ environment.logs.accept(event);
+ }
+ }
+
+ private static byte[] runtimeScript() {
+ try (InputStream in = Thread.currentThread().getContextClassLoader().getResourceAsStream(RUNTIME_RESOURCE)) {
+ if (in == null) {
+ throw new IllegalStateException(RUNTIME_RESOURCE + " is missing from the classpath");
+ }
+ return in.readAllBytes();
+ } catch (IOException e) {
+ throw new UncheckedIOException("Could not read " + RUNTIME_RESOURCE, e);
+ }
+ }
+
/** The environment leaves the pool before waiters hear of the exit, so the request after a failed one starts anew. */
private void exited(Environment environment, long seenAt) {
boolean running = environment.alive.compareAndSet(true, false);
diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogs.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogs.java
new file mode 100644
index 0000000000..ef3daab8e3
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogs.java
@@ -0,0 +1,166 @@
+package io.github.hectorvent.floci.services.lambdaweb;
+
+import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.services.cloudwatch.logs.CloudWatchLogsService;
+import jakarta.annotation.PreDestroy;
+import jakarta.enterprise.context.ApplicationScoped;
+import jakarta.inject.Inject;
+import org.jboss.logging.Logger;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Delivers the log events of Web Function environments. Each event goes to the stream of its
+ * environment in the {@code .local} companion of the revision's log group, and to the same stream
+ * with an {@code -<account>-<region>} suffix in the log group itself, where AWS centralizes the
+ * regions an endpoint serves. A group or stream is created when its first event arrives.
+ */
+@ApplicationScoped
+public class LambdaWebLogs {
+
+ private static final Logger LOG = Logger.getLogger(LambdaWebLogs.class);
+ private static final int MAX_QUEUED_EVENTS = 10_000;
+ private static final int MAX_BATCH_EVENTS = 1_000;
+
+ private final CloudWatchLogsService logs;
+ private final ExecutorService writers = Executors.newVirtualThreadPerTaskExecutor();
+
+ @Inject
+ public LambdaWebLogs(CloudWatchLogsService logs) {
+ this.logs = logs;
+ }
+
+ public Sink open(String accountId, String region, String logGroup, String logStream) {
+ return new Sink(accountId, region, List.of(
+ new Destination(logGroup + ".local", logStream),
+ new Destination(logGroup, logStream + "-" + accountId + "-" + region)));
+ }
+
+ @PreDestroy
+ void shutdown() {
+ writers.shutdownNow();
+ }
+
+ private record Destination(String logGroup, String logStream) {
+ }
+
+ private record Event(long timestamp, String message) {
+ }
+
+ /**
+ * The events of one environment. {@link #accept} only queues, because the app blocks on its
+ * output while the Docker stream that carries it waits. Past {@code MAX_QUEUED_EVENTS} events are dropped.
+ */
+ public final class Sink {
+ private final String accountId;
+ private final String region;
+ private final List<Destination> destinations;
+ private final ConcurrentLinkedQueue<Event> queue = new ConcurrentLinkedQueue<>();
+ private final AtomicInteger queued = new AtomicInteger();
+ private final AtomicBoolean writing = new AtomicBoolean();
+
+ private Sink(String accountId, String region, List<Destination> destinations) {
+ this.accountId = accountId;
+ this.region = region;
+ this.destinations = destinations;
+ }
+
+ public void accept(String message) {
+ if (queued.incrementAndGet() > MAX_QUEUED_EVENTS) {
+ queued.decrementAndGet();
+ LOG.debugv("Dropped a log event of {0}: too many are waiting", destinations.getFirst().logStream());
+ return;
+ }
+ queue.add(new Event(System.currentTimeMillis(), message));
+ schedule();
+ }
+
+ private void schedule() {
+ if (!writing.compareAndSet(false, true)) {
+ return;
+ }
+ try {
+ writers.execute(this::write);
+ } catch (RuntimeException e) {
+ writing.set(false);
+ LOG.debugv(e, "Could not deliver the log events of {0}", destinations.getFirst().logStream());
+ }
+ }
+
+ private void write() {
+ try {
+ List<Map<String, Object>> batch = new ArrayList<>();
+ Event event;
+ while ((event = queue.poll()) != null) {
+ queued.decrementAndGet();
+ batch.add(Map.of("timestamp", event.timestamp(), "message", event.message()));
+ if (batch.size() == MAX_BATCH_EVENTS) {
+ put(batch);
+ batch = new ArrayList<>();
+ }
+ }
+ if (!batch.isEmpty()) {
+ put(batch);
+ }
+ } finally {
+ writing.set(false);
+ if (!queue.isEmpty()) {
+ schedule();
+ }
+ }
+ }
+
+ private void put(List<Map<String, Object>> batch) {
+ for (Destination destination : destinations) {
+ try {
+ putTo(destination, batch);
+ } catch (RuntimeException e) {
+ LOG.warnv(e, "Could not deliver {0} log events to {1}", batch.size(), destination.logGroup());
+ }
+ }
+ }
+
+ /** The group and the stream are created when an event finds one of them missing, also after a delete. */
+ private void putTo(Destination destination, List<Map<String, Object>> batch) {
+ try {
+ logs.putLogEventsForAccount(accountId, destination.logGroup(), destination.logStream(), batch, region);
+ } catch (AwsException e) {
+ if (!"ResourceNotFoundException".equals(e.getErrorCode())) {
+ throw e;
+ }
+ createLogGroup(destination);
+ createLogStream(destination);
+ logs.putLogEventsForAccount(accountId, destination.logGroup(), destination.logStream(), batch, region);
+ }
+ }
+
+ private void createLogGroup(Destination destination) {
+ try {
+ logs.createLogGroupForAccount(accountId, destination.logGroup(), null, null, region);
+ } catch (AwsException e) {
+ ignoreAlreadyExists(e);
+ }
+ }
+
+ private void createLogStream(Destination destination) {
+ try {
+ logs.createLogStreamForAccount(accountId, destination.logGroup(), destination.logStream(), region);
+ } catch (AwsException e) {
+ ignoreAlreadyExists(e);
+ }
+ }
+
+ private static void ignoreAlreadyExists(AwsException e) {
+ if (!"ResourceAlreadyExistsException".equals(e.getErrorCode())) {
+ throw e;
+ }
+ }
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebOutputLines.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebOutputLines.java
new file mode 100644
index 0000000000..71faa656a1
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebOutputLines.java
@@ -0,0 +1,84 @@
+package io.github.hectorvent.floci.services.lambdaweb;
+
+import java.nio.ByteBuffer;
+import java.nio.CharBuffer;
+import java.nio.charset.CharsetDecoder;
+import java.nio.charset.CodingErrorAction;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Cuts what one output stream of an environment writes into log events. An event keeps its
+ * newline. Bytes that are not UTF-8 become U+FFFD, and a line whose text is longer than
+ * {@link #MAX_EVENT_BYTES} in UTF-8 is cut before the character that would pass it, without its
+ * newline, as on AWS.
+ */
+final class LambdaWebOutputLines {
+
+ /** The CloudWatch Logs event limit of 256 KiB less the 26 bytes it charges for each event. */
+ static final int MAX_EVENT_BYTES = 262_118;
+
+ private final CharsetDecoder decoder = StandardCharsets.UTF_8.newDecoder()
+ .onMalformedInput(CodingErrorAction.REPLACE)
+ .onUnmappableCharacter(CodingErrorAction.REPLACE);
+ /** The start of a character whose other bytes come in the next payload. */
+ private ByteBuffer undecoded = ByteBuffer.allocate(0);
+ private final StringBuilder pending = new StringBuilder();
+ private int pendingBytes;
+
+ List<String> append(byte[] payload) {
+ ByteBuffer in = ByteBuffer.allocate(undecoded.remaining() + payload.length).put(undecoded).put(payload).flip();
+ CharBuffer out = CharBuffer.allocate(in.remaining());
+ decoder.decode(in, out, false);
+ undecoded = ByteBuffer.allocate(in.remaining()).put(in).flip();
+ return cut(out.flip());
+ }
+
+ /** The output after the last newline, when the stream ends without one. */
+ List<String> flush() {
+ CharBuffer out = CharBuffer.allocate(undecoded.remaining() + 1);
+ decoder.decode(undecoded, out, true);
+ decoder.flush(out);
+ decoder.reset();
+ undecoded = ByteBuffer.allocate(0);
+ List<String> events = cut(out.flip());
+ if (!pending.isEmpty()) {
+ events.add(take());
+ }
+ return events;
+ }
+
+ private List<String> cut(CharBuffer text) {
+ List<String> events = new ArrayList<>();
+ text.codePoints().forEach(codePoint -> {
+ int length = utf8Length(codePoint);
+ if (MAX_EVENT_BYTES < pendingBytes + length) {
+ events.add(take());
+ }
+ pending.appendCodePoint(codePoint);
+ pendingBytes += length;
+ if (codePoint == '\n') {
+ events.add(take());
+ }
+ });
+ return events;
+ }
+
+ private static int utf8Length(int codePoint) {
+ if (codePoint < 0x80) {
+ return 1;
+ }
+ if (codePoint < 0x800) {
+ return 2;
+ }
+ return codePoint < 0x10000 ? 3 : 4;
+ }
+
+ private String take() {
+ String event = pending.toString();
+ pending.setLength(0);
+ pendingBytes = 0;
+ return event;
+ }
+}
diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml
index 93256dd9d7..7467be0b5c 100644
--- a/src/main/resources/application.yml
+++ b/src/main/resources/application.yml
@@ -53,7 +53,8 @@ quarkus:
# catalog (ec2/image-catalog.yaml), SageMaker reads its instance type
# catalog (sagemaker/instance-type-catalog.yaml), KMS reads its AWS managed
# key catalog (kms/aws-managed-keys.yaml), AppSync's GraphQL
- # parser reads its message bundles, EC2 security group enforcement builds
+ # parser reads its message bundles, Lambda Web Functions copy their
+ # runtime (lambdaweb/index.mjs) into each environment, EC2 security group enforcement builds
# its helper image from docker/*.Dockerfile, and API Gateway's Velocity
# engine reads its default properties at runtime via
# ClassLoader#getResourceAsStream. Quarkus's static resource analysis
@@ -68,6 +69,7 @@ quarkus:
- pricing-snapshots/**
- ui/*.html
- appsync/*.mjs
+ - lambdaweb/*.mjs
- ec2/*.yaml
- docker/*.Dockerfile
- sagemaker/*.yaml
diff --git a/src/main/resources/lambdaweb/index.mjs b/src/main/resources/lambdaweb/index.mjs
new file mode 100644
index 0000000000..654ed40b22
--- /dev/null
+++ b/src/main/resources/lambdaweb/index.mjs
@@ -0,0 +1,96 @@
+// The runtime entry Floci starts as /var/runtime/index.js, where AWS starts its own. It turns
+// console output into the JSON log events AWS writes, with the request each one belongs to,
+// filters the output by applicationLogLevel, then imports the app.
+import { AsyncLocalStorage } from 'node:async_hooks';
+import http from 'node:http';
+import path from 'node:path';
+import { pathToFileURL } from 'node:url';
+import util from 'node:util';
+
+// FATAL is above ERROR. Node has no console method for it, so it drops every console call.
+const LEVELS = { TRACE: 0, DEBUG: 1, INFO: 2, WARN: 3, ERROR: 4, FATAL: 5 };
+const minimum = LEVELS[process.env.AWS_LAMBDA_LOG_LEVEL] ?? LEVELS.INFO;
+const region = process.env.AWS_REGION;
+const requests = new AsyncLocalStorage();
+const writeOut = process.stdout.write.bind(process.stdout);
+const writeErr = process.stderr.write.bind(process.stderr);
+
+const emit = http.Server.prototype.emit;
+http.Server.prototype.emit = function (event, request) {
+ if (event !== 'request') {
+ return emit.apply(this, arguments);
+ }
+ const context = {
+ requestId: request.headers['lambda-web-request-id'],
+ endpointName: request.headers['lambda-web-endpoint-name'],
+ };
+ return requests.run(context, () => emit.apply(this, arguments));
+};
+
+function withErrors(key, value) {
+ if (!(value instanceof Error)) {
+ return value;
+ }
+ return { errorType: value.name, errorMessage: value.message, stackTrace: String(value.stack).split('\n'), ...value };
+}
+
+function line(level, message) {
+ const context = requests.getStore();
+ const event = {
+ timestamp: new Date().toISOString(),
+ level,
+ message,
+ requestId: context?.requestId,
+ endpointName: context?.endpointName,
+ region,
+ };
+ return JSON.stringify(event, withErrors) + '\n';
+}
+
+function render(level, args) {
+ if (args.length !== 1 || typeof args[0] === 'string') {
+ return line(level, args.length === 0 ? undefined : util.format(...args));
+ }
+ try {
+ return line(level, args[0]);
+ } catch (unserializable) {
+ // A BigInt or a cycle is not JSON. AWS writes the text Node prints for it.
+ return line(level, util.format(...args));
+ }
+}
+
+function patch(name, level, write) {
+ console[name] = function (...args) {
+ if (LEVELS[level] >= minimum) {
+ write(render(level, args));
+ }
+ };
+}
+
+patch('log', 'INFO', writeOut);
+patch('info', 'INFO', writeOut);
+patch('debug', 'DEBUG', writeOut);
+patch('warn', 'WARN', writeErr);
+patch('error', 'ERROR', writeErr);
+console.trace = function (...args) {
+ if (LEVELS.TRACE >= minimum) {
+ const frames = String(new Error().stack).split('\n').slice(1).filter((frame) => !frame.includes(import.meta.url));
+ writeOut(line('TRACE', { traceMessage: util.format(...args), stackTrace: frames }));
+ }
+};
+
+// Output that does not go through console counts as INFO.
+if (LEVELS.INFO < minimum) {
+ const drop = (chunk, encoding, callback) => {
+ const done = typeof encoding === 'function' ? encoding : callback;
+ if (typeof done === 'function') {
+ process.nextTick(done);
+ }
+ return true;
+ };
+ process.stdout.write = drop;
+ process.stderr.write = drop;
+}
+
+const entry = process.env.AWS_LAMBDA_NODEJS_ENTRYPOINT || 'index.js';
+await import(pathToFileURL(path.resolve('/var/task', entry)).href);
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java
index 7a0b041221..70c7c1592a 100644
--- a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java
@@ -3,12 +3,15 @@ package io.github.hectorvent.floci.services.lambdaweb;
import com.github.dockerjava.api.DockerClient;
import io.github.hectorvent.floci.services.lambdaweb.LambdaWebEnvironmentManager.Environment;
import io.github.hectorvent.floci.services.s3.S3Service;
+import io.github.hectorvent.floci.testing.RestAssuredJsonUtils;
import io.quarkus.test.junit.QuarkusTest;
import io.restassured.RestAssured;
+import io.restassured.path.json.JsonPath;
import io.restassured.response.Response;
import jakarta.inject.Inject;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Assumptions;
+import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.Timeout;
@@ -60,7 +63,18 @@ class LambdaWebDataPlaneDockerIntegrationTest {
private static final String APP = """
const http = require('node:http');
const [host, port] = process.env.AWS_LAMBDA_HTTP_ENDPOINT.split(':');
+ console.log('starting');
http.createServer((req, res) => {
+ if (req.url.startsWith('/log')) {
+ console.info({route: 'log', main: require.main === module, argv: process.argv.slice(1)});
+ console.debug('below the level');
+ console.error(new Error('logged error'));
+ process.stdout.write('raw line\\n');
+ process.stdout.write(' \\n');
+ process.stdout.write('last line\\n');
+ res.end('logged');
+ return;
+ }
if (req.url.startsWith('/slow')) { setTimeout(() => res.end('late'), 8000); return; }
if (req.url.startsWith('/wait')) { setTimeout(() => res.end('waited'), 3000); return; }
if (req.url.startsWith('/stream')) {
@@ -100,6 +114,11 @@ class LambdaWebDataPlaneDockerIntegrationTest {
private String functionName;
+ @BeforeAll
+ static void configureRestAssured() {
+ RestAssuredJsonUtils.configureAwsContentTypes();
+ }
+
@BeforeEach
void requireDocker() {
Assumptions.assumeTrue(isDockerAvailable(), "Docker must be available for the Lambda Web data plane");
@@ -181,6 +200,38 @@ class LambdaWebDataPlaneDockerIntegrationTest {
.body("headers.x-amz-date", equalTo("20260101T000000Z"));
}
+ @Test
+ @Timeout(180)
+ void theOutputGoesToTheLocalAndTheCentralizedLogGroupAsJsonEvents() throws Exception {
+ String domain = createFunction(30);
+ Response logged = given().header("Host", domain).when().get("/log");
+ logged.then().statusCode(200);
+ String logStream = logged.header("lambda-web-logstream");
+ String group = "/aws/lambda/web/" + functionName;
+
+ List<String> local = awaitLogEvents(group + ".local", logStream);
+ assertEquals(5, local.size(), local.toString());
+ JsonPath starting = JsonPath.from(local.getFirst());
+ assertEquals("INFO", starting.getString("level"));
+ assertEquals("starting", starting.getString("message"));
+ assertEquals("us-east-1", starting.getString("region"));
+ assertFalse(local.getFirst().contains("requestId"), local.getFirst());
+ JsonPath info = JsonPath.from(local.stream().filter(event -> event.contains("\"route\"")).findFirst().orElseThrow());
+ assertEquals(logged.header("lambda-web-request-id"), info.getString("requestId"));
+ assertEquals("dev", info.getString("endpointName"));
+ assertEquals(false, info.getBoolean("message.main"));
+ assertEquals(List.of("/var/runtime/index.js"), info.getList("message.argv"));
+ JsonPath error = JsonPath.from(local.stream().filter(event -> event.contains("ERROR")).findFirst().orElseThrow());
+ assertEquals("Error", error.getString("message.errorType"));
+ assertEquals("logged error", error.getString("message.errorMessage"));
+ assertEquals("Error: logged error", error.getList("message.stackTrace").getFirst());
+ assertTrue(local.contains("raw line\n"), local.toString());
+ assertTrue(local.stream().allMatch(event -> event.endsWith("\n")), local.toString());
+
+ List<String> centralized = awaitLogEvents(group, logStream + "-000000000000-us-east-1");
+ assertEquals(new HashSet<>(local), new HashSet<>(centralized));
+ }
+
@Test
@Timeout(180)
void anIamAuthEndpointServesSignedRequestsWithoutTheirSignature() throws Exception {
@@ -368,6 +419,29 @@ class LambdaWebDataPlaneDockerIntegrationTest {
.extract().path("endpoint.domainName");
}
+ /**
+ * The messages of a log stream once the last output of {@code /log} on stdout and on stderr is in it, or what it
+ * holds after 15 seconds. The output of one stream arrives in order, so nothing written before comes later.
+ */
+ private static List<String> awaitLogEvents(String logGroup, String logStream) throws InterruptedException {
+ long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(15);
+ List<String> messages = logEvents(logGroup, logStream);
+ while (!(messages.contains("last line\n") && messages.stream().anyMatch(event -> event.contains("\"ERROR\"")))
+ && System.nanoTime() < deadline) {
+ Thread.sleep(200);
+ messages = logEvents(logGroup, logStream);
+ }
+ return messages;
+ }
+
+ private static List<String> logEvents(String logGroup, String logStream) {
+ List<String> messages = RestAssuredJsonUtils.awsAction("Logs_20140328", "GetLogEvents",
+ "{\"logGroupName\": \"%s\", \"logStreamName\": \"%s\", \"startFromHead\": true}"
+ .formatted(logGroup, logStream))
+ .then().extract().jsonPath().getList("events.message", String.class);
+ return messages == null ? List.of() : messages;
+ }
+
/** Sends one raw HTTP/1.1 request with {@code Connection: close} and reads until the server closes. */
private static String raw(String request) throws Exception {
return raw(request, null);
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogsTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogsTest.java
new file mode 100644
index 0000000000..361b3a1fae
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogsTest.java
@@ -0,0 +1,112 @@
+package io.github.hectorvent.floci.services.lambdaweb;
+
+import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.services.cloudwatch.logs.CloudWatchLogsService;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.ArgumentMatchers.anyList;
+import static org.mockito.ArgumentMatchers.argThat;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.timeout;
+import static org.mockito.Mockito.verify;
+
+@ExtendWith(MockitoExtension.class)
+class LambdaWebLogsTest {
+
+ private static final String ACCOUNT = "111122223333";
+ private static final String REGION = "eu-west-1";
+ private static final String GROUP = "/aws/lambda/web/web";
+ private static final String STREAM = "2026/10/18/web[rev-abcde]0123456789abcdef0123456789abcdef";
+ private static final String CENTRAL_STREAM = STREAM + "-" + ACCOUNT + "-" + REGION;
+
+ @Mock
+ CloudWatchLogsService cloudWatchLogs;
+
+ private LambdaWebLogs logs;
+
+ @BeforeEach
+ void start() {
+ logs = new LambdaWebLogs(cloudWatchLogs);
+ }
+
+ @AfterEach
+ void stop() {
+ logs.shutdown();
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ void eventsGoToTheLocalAndTheCentralizedGroupCreatedWhenMissing() {
+ doThrow(notFound()).doReturn("token").when(cloudWatchLogs)
+ .putLogEventsForAccount(eq(ACCOUNT), eq(GROUP + ".local"), eq(STREAM), anyList(), eq(REGION));
+ doThrow(notFound()).doReturn("token").when(cloudWatchLogs)
+ .putLogEventsForAccount(eq(ACCOUNT), eq(GROUP), eq(CENTRAL_STREAM), anyList(), eq(REGION));
+
+ logs.open(ACCOUNT, REGION, GROUP, STREAM).accept("first\n");
+
+ ArgumentCaptor<List<Map<String, Object>>> local = ArgumentCaptor.forClass(List.class);
+ verify(cloudWatchLogs, timeout(5_000).times(2)).putLogEventsForAccount(eq(ACCOUNT), eq(GROUP + ".local"),
+ eq(STREAM), local.capture(), eq(REGION));
+ verify(cloudWatchLogs, timeout(5_000).times(2)).putLogEventsForAccount(eq(ACCOUNT), eq(GROUP),
+ eq(CENTRAL_STREAM), anyList(), eq(REGION));
+ verify(cloudWatchLogs).createLogGroupForAccount(ACCOUNT, GROUP + ".local", null, null, REGION);
+ verify(cloudWatchLogs).createLogStreamForAccount(ACCOUNT, GROUP + ".local", STREAM, REGION);
+ verify(cloudWatchLogs).createLogGroupForAccount(ACCOUNT, GROUP, null, null, REGION);
+ verify(cloudWatchLogs).createLogStreamForAccount(ACCOUNT, GROUP, CENTRAL_STREAM, REGION);
+ assertEquals("first\n", local.getValue().getFirst().get("message"));
+ }
+
+ @Test
+ void aGroupThatCannotBeCreatedTakesNoEventsAndTheOtherStillDoes() {
+ doThrow(notFound()).when(cloudWatchLogs)
+ .putLogEventsForAccount(eq(ACCOUNT), eq(GROUP + ".local"), eq(STREAM), anyList(), eq(REGION));
+ doThrow(new AwsException("InvalidParameterException", "Invalid log group name.", 400)).when(cloudWatchLogs)
+ .createLogGroupForAccount(ACCOUNT, GROUP + ".local", null, null, REGION);
+
+ logs.open(ACCOUNT, REGION, GROUP, STREAM).accept("first\n");
+
+ verify(cloudWatchLogs, timeout(5_000)).putLogEventsForAccount(eq(ACCOUNT), eq(GROUP), eq(CENTRAL_STREAM),
+ anyList(), eq(REGION));
+ verify(cloudWatchLogs, never()).createLogStreamForAccount(ACCOUNT, GROUP + ".local", STREAM, REGION);
+ }
+
+ @Test
+ void eventsQueuedWhileAWriteFailsAreStillDelivered() throws Exception {
+ CountDownLatch writing = new CountDownLatch(1);
+ CountDownLatch queued = new CountDownLatch(1);
+ doAnswer(invocation -> {
+ writing.countDown();
+ queued.await(5, TimeUnit.SECONDS);
+ throw new IllegalStateException("the store failed");
+ }).doReturn("token").when(cloudWatchLogs)
+ .putLogEventsForAccount(eq(ACCOUNT), eq(GROUP + ".local"), eq(STREAM), anyList(), eq(REGION));
+ LambdaWebLogs.Sink sink = logs.open(ACCOUNT, REGION, GROUP, STREAM);
+
+ sink.accept("first\n");
+ writing.await(5, TimeUnit.SECONDS);
+ sink.accept("second\n");
+ queued.countDown();
+
+ verify(cloudWatchLogs, timeout(5_000)).putLogEventsForAccount(eq(ACCOUNT), eq(GROUP + ".local"), eq(STREAM),
+ argThat(batch -> batch.size() == 1 && "second\n".equals(batch.getFirst().get("message"))), eq(REGION));
+ }
+
+ private static AwsException notFound() {
+ return new AwsException("ResourceNotFoundException", "The specified log stream does not exist.", 400);
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebOutputLinesTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebOutputLinesTest.java
new file mode 100644
index 0000000000..51b33831f0
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebOutputLinesTest.java
@@ -0,0 +1,83 @@
+package io.github.hectorvent.floci.services.lambdaweb;
+
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayOutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+class LambdaWebOutputLinesTest {
+
+ private static final int MAX = LambdaWebOutputLines.MAX_EVENT_BYTES;
+
+ @Test
+ void eachLineIsAnEventWithItsNewline() {
+ LambdaWebOutputLines lines = new LambdaWebOutputLines();
+
+ assertEquals(List.of("one\n", "two\n"), lines.append(bytes("one\ntwo\nthr")));
+ assertEquals(List.of("three\n"), lines.append(bytes("ee\n")));
+ assertEquals(List.of(), lines.flush());
+ }
+
+ @Test
+ void theOutputAfterTheLastNewlineComesOutWhenTheStreamEnds() {
+ LambdaWebOutputLines lines = new LambdaWebOutputLines();
+
+ assertEquals(List.of(), lines.append(bytes("no newline")));
+ assertEquals(List.of("no newline"), lines.flush());
+ }
+
+ @Test
+ void aLongLineIsCutBeforeTheCharacterThatWouldPassTheLimit() {
+ LambdaWebOutputLines lines = new LambdaWebOutputLines();
+
+ assertEquals(List.of("y".repeat(MAX - 2), "日tail\n"), lines.append(bytes("y".repeat(MAX - 2) + "日tail\n")));
+ assertEquals(List.of("z".repeat(MAX), "z\n"), lines.append(bytes("z".repeat(MAX + 1) + "\n")));
+ }
+
+ @Test
+ void bytesThatAreNotUtf8BecomeReplacementCharactersCountedAtTheirEncodedSize() {
+ LambdaWebOutputLines lines = new LambdaWebOutputLines();
+
+ assertEquals(List.of("a�b��c\n"),
+ lines.append(concat(bytes("a"), new byte[] {(byte) 0x80}, bytes("b"),
+ new byte[] {(byte) 0xFF, (byte) 0xFE}, bytes("c\n"))));
+ assertEquals(List.of("cut � end\n"),
+ lines.append(concat(bytes("cut "), new byte[] {(byte) 0xE6, (byte) 0x97}, bytes(" end\n"))));
+ assertEquals(List.of("x".repeat(MAX - 1), "����tail\n"),
+ lines.append(concat(bytes("x".repeat(MAX - 1)), new byte[] {(byte) 0x80, (byte) 0x80, (byte) 0x80,
+ (byte) 0x80}, bytes("tail\n"))));
+ }
+
+ @Test
+ void aCharacterSplitAcrossPayloadsStaysWhole() {
+ LambdaWebOutputLines lines = new LambdaWebOutputLines();
+ byte[] word = bytes("日本\n");
+
+ assertEquals(List.of(), lines.append(new byte[] {word[0], word[1]}));
+ assertEquals(List.of("日本\n"), lines.append(Arrays.copyOfRange(word, 2, word.length)));
+ }
+
+ @Test
+ void anUnfinishedCharacterAtTheEndBecomesAReplacementCharacter() {
+ LambdaWebOutputLines lines = new LambdaWebOutputLines();
+
+ assertEquals(List.of(), lines.append(new byte[] {'a', (byte) 0xE6, (byte) 0x97}));
+ assertEquals(List.of("a�"), lines.flush());
+ }
+
+ private static byte[] bytes(String text) {
+ return text.getBytes(StandardCharsets.UTF_8);
+ }
+
+ private static byte[] concat(byte[]... parts) {
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ for (byte[] part : parts) {
+ out.writeBytes(part);
+ }
+ return out.toByteArray();
+ }
+}