Light, fluffy, and always free - The AWS Local Emulator alternative
feat(lambdaweb): scale environments per endpoint and apply maxEnvironments and rateLimit (#5785)
TATSUNO “Taz” Yasuhiro832f1008bb6e
diff --git a/docs/services/lambda-web-functions.md b/docs/services/lambda-web-functions.mdindex f4e3b19aff..7ab7f4114d 100644--- a/docs/services/lambda-web-functions.md+++ b/docs/services/lambda-web-functions.md@@ -41,7 +41,7 @@ The `Lambda Web` service model (apiVersion 2025-03-07, `aws lambda-web` in the A- **ARNs** given as identifiers must name the request's region and the caller's account, as on AWS. A revision or endpoint ARN resolves by its id or name only.- **Page tokens** are base64 cursors that carry the request and the position of the page, not the UUIDs AWS issues. A token reused with other parameters gets the AWS "does not match the request" error, and any other token the "invalid or has expired" error.- **Domain names** take the form `<id>.001.lambda-web.<region>.<floci host>` and `<id>.lambda-web.global.<floci host>`, with the host and port of `floci.base-url`. A `PerRegion` endpoint gets one domain per region in `regionalEndpoints`.-- **Not implemented yet:** `IamAuth` on requests, the automatic deletion of revisions past 50, the scaling limits (`maxConcurrencyPerEnvironment`, `maxEnvironments`, a `rateLimit` other than 0), the metadata API, the logs and metrics, and serving a `MultiRegion` or `PerRegion` endpoint in its other regions.+- **Not implemented yet:** `IamAuth` on requests, the automatic deletion of revisions past 50, the metadata API, the logs and metrics, and serving a `MultiRegion` or `PerRegion` endpoint in its other regions.### `aws lambda-web deploy`@@ -64,7 +64,9 @@ curl -H "Host: <domainName>" http://localhost:4566/The `Host` must match the domain exactly, as on AWS: another case or another port is `404` with `lambda-web-error-type: Platform.InvalidEndpoint`.-Floci runs each revision in one execution environment, a `public.ecr.aws/lambda/nodejs:24` container started by the first request. The code is the revision's S3 object unpacked into `/var/task`, the entry file is `index.js` or the file `AWS_LAMBDA_NODEJS_ENTRYPOINT` names, and node runs with the flags AWS uses. The app sees `AWS_LAMBDA_HTTP_ENDPOINT=0.0.0.0:3000`, `AWS_LAMBDA_INITIALIZATION_TYPE=native-http`, `AWS_EXECUTION_ENV`, `AWS_LAMBDA_FUNCTION_MEMORY_SIZE=2048`, `AWS_LAMBDA_LOG_LEVEL`, `AWS_ACCOUNT_ID`, `AWS_REGION`, credentials and `AWS_ENDPOINT_URL` for Floci, `NODE_PATH`, and its own variables. The server has 10 seconds to listen. An environment that sees no request for `floci.services.lambda.container-idle-timeout-seconds` is stopped with `SIGTERM` and one second, as on scale to zero. A process that exits between requests is replaced by a fresh environment on the next request.+An execution environment is a `public.ecr.aws/lambda/nodejs:24` container that a request starts when every environment of the endpoint is full. The code is the revision's S3 object unpacked into `/var/task`, the entry file is `index.js` or the file `AWS_LAMBDA_NODEJS_ENTRYPOINT` names, and node runs with the flags AWS uses. The app sees `AWS_LAMBDA_HTTP_ENDPOINT=0.0.0.0:3000`, `AWS_LAMBDA_INITIALIZATION_TYPE=native-http`, `AWS_EXECUTION_ENV`, `AWS_LAMBDA_FUNCTION_MEMORY_SIZE=2048`, `AWS_LAMBDA_LOG_LEVEL`, `AWS_ACCOUNT_ID`, `AWS_REGION`, credentials and `AWS_ENDPOINT_URL` for Floci, `NODE_PATH`, and its own variables. The server has 10 seconds to listen. An environment that sees no request for `floci.services.lambda.container-idle-timeout-seconds` is stopped with `SIGTERM` and one second, as on scale to zero. A process that exits between requests is replaced by a fresh environment on the next request.++Each endpoint scales on its own, as on AWS. An environment takes up to the revision's `maxConcurrencyPerEnvironment` requests at once, the least busy one first, so requests rotate over the environments. A request that finds every environment full starts another one, or waits for one that is starting and has room. When the endpoint already runs `maxEnvironments`, an idle environment of another revision makes room, and without one the request is refused with `429` and `Platform.EndpointMaxEnvThrottle`. A `rateLimit` admits a burst of up to `rateLimit` requests that refills at `rateLimit` per second, and refuses the rest with `429` and `Platform.EndpointRequestThrottle`. AWS adjusts the concurrency of each environment under `maxConcurrencyPerEnvironment` from the measured load, so a burst there can spread over more environments than in Floci, and the account quotas are not applied.Requests are forwarded over HTTP/1.1 with the headers AWS adds and drops: `lambda-web-request-id`, `lambda-web-endpoint-name`, `x-forwarded-proto`, `x-forwarded-port` and `Connection: close` are set, the client address is appended as an `X-Forwarded-For` line, and `Connection`, `Keep-Alive`, `Upgrade`, `Expect`, `Trailer`, the `Proxy-*` headers, `X-Real-IP`, the method override headers and any `lambda-web-*` header the client sent are dropped. A body without a `Content-Length` reaches the app chunked. Responses carry `lambda-web-request-id` and `lambda-web-logstream`, lose their connection headers and trailers, and stream as the app writes them. Requests with two revisions in `revisionWeights` are split by weight.@@ -78,6 +80,8 @@ The platform answers these cases itself, with the AWS HTML pages and `lambda-web| --- | --- | --- || 400 | `Platform.MalformedRequest` | A method other than `GET`, `HEAD`, `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` || 404 | `Platform.InvalidEndpoint` | The `Host` is a `lambda-web` domain Floci did not mint |+| 429 | `Platform.EndpointMaxEnvThrottle` | Every environment is full and the endpoint runs `maxEnvironments` |+| 429 | `Platform.EndpointRequestThrottle` | The requests exceed the endpoint's `rateLimit` || 502 | `Application.TimeoutListenError`, `Application.InitCodeError`, `Runtime.Exit` | The server did not listen within 10 seconds, threw at import, or exited at start || 503 | `Platform.EndpointDisabled` | The endpoint's `rateLimit` is 0 || 503 | `Runtime.OriginRequestError` | The process died while handling the request |diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlane.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlane.javaindex ffbfdc754a..77b2b13f68 100644--- a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlane.java+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlane.java@@ -4,6 +4,7 @@ import io.github.hectorvent.floci.core.common.RequestHost;import io.github.hectorvent.floci.core.common.ServiceConfigAccess;import io.github.hectorvent.floci.services.lambdaweb.LambdaWebEnvironmentManager.Environment;import io.github.hectorvent.floci.services.lambdaweb.LambdaWebEnvironmentManager.StartFailure;+import io.github.hectorvent.floci.services.lambdaweb.LambdaWebEnvironmentManager.Throttled;import io.github.hectorvent.floci.services.lambdaweb.LambdaWebService.WebFunctionTarget;import io.github.hectorvent.floci.services.lambdaweb.model.WebFunctionRevision;import io.vertx.core.Future;@@ -69,6 +70,7 @@ public class LambdaWebDataPlane {private final ServiceConfigAccess serviceConfigAccess;private final Vertx vertx;private final HttpClient client;+ private final LambdaWebRateLimiter rateLimiter = new LambdaWebRateLimiter();@Injectpublic LambdaWebDataPlane(LambdaWebService service, LambdaWebEnvironmentManager environments,@@ -109,11 +111,15 @@ public class LambdaWebDataPlane {return;}WebFunctionTarget target = located.get();- if (target.endpoint().throttleConfig != null- && Integer.valueOf(0).equals(target.endpoint().throttleConfig.rateLimit)) {+ Integer rateLimit = target.endpoint().throttleConfig == null ? null : target.endpoint().throttleConfig.rateLimit;+ if (Integer.valueOf(0).equals(rateLimit)) {page(request, LambdaWebErrorPages.ENDPOINT_DISABLED, requestId, false);return;}+ if (rateLimit != null && !rateLimiter.tryAcquire(endpointKey(target), rateLimit)) {+ page(request, LambdaWebErrorPages.ENDPOINT_REQUEST_THROTTLE, requestId, false);+ return;+ }WebFunctionRevision revision = LambdaWebService.pickRevision(target);if (revision == null) {page(request, LambdaWebErrorPages.ORIGIN_REQUEST_ERROR, requestId, false);@@ -126,6 +132,11 @@ public class LambdaWebDataPlane {acquire(request, host, target, revision, requestId, null);}+ /** The domain is minted for each creation, so a recreated endpoint shares no state with the deleted one. */+ private static String endpointKey(WebFunctionTarget target) {+ return String.join("|", target.accountId(), target.region(), target.endpoint().domainName);+ }+/** Starts or reuses the revision's environment off the event loop, then forwards. */private void acquire(HttpServerRequest request, String host, WebFunctionTarget target,WebFunctionRevision revision, String requestId, Environment dead) {@@ -133,7 +144,10 @@ public class LambdaWebDataPlane {if (dead != null) {environments.discard(dead);}- return environments.acquire(target.accountId(), target.region(), target.function(), revision);+ Integer maxEnvironments = target.endpoint().scalingConfig == null+ ? null : target.endpoint().scalingConfig.maxEnvironments;+ return environments.acquire(target.accountId(), target.region(), target.function(),+ endpointKey(target), maxEnvironments, revision);}, false).onSuccess(environment -> forward(request, host, target, revision, environment, requestId, dead == null)).onFailure(error -> {@@ -142,6 +156,10 @@ public class LambdaWebDataPlane {page(request, failure.page(), requestId, false);return;}+ if (error instanceof Throttled) {+ page(request, LambdaWebErrorPages.ENDPOINT_MAX_ENV_THROTTLE, requestId, false);+ return;+ }LOG.errorv(error, "Could not start an environment for {0}", host);internalError(request, requestId);});diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.javaindex 2e2d639fd4..9c082423af 100644--- a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java@@ -49,7 +49,9 @@ import java.util.List;import java.util.Map;import java.util.Optional;import java.util.UUID;+import java.util.concurrent.CompletableFuture;import java.util.concurrent.ConcurrentHashMap;+import java.util.concurrent.ExecutionException;import java.util.concurrent.Executors;import java.util.concurrent.ScheduledExecutorService;import java.util.concurrent.TimeUnit;@@ -60,9 +62,9 @@ import java.util.concurrent.atomic.AtomicReference;import java.util.stream.Stream;/**- * The execution environments of Web Function revisions: one Node.js server container per revision,- * started by the first request, reused while requests keep coming and removed after the Lambda idle- * timeout. The runtime loads {@code index.js}, or the file {@code AWS_LAMBDA_NODEJS_ENTRYPOINT} names,+ * The execution environments of Web Function endpoints: Node.js server containers of a revision,+ * started by requests that find no room, reused while requests keep coming and removed after the+ * Lambda idle timeout. The runtime loads {@code index.js}, or the file {@code AWS_LAMBDA_NODEJS_ENTRYPOINT} names,* from {@code /var/task} with the node flags AWS uses.*/@ApplicationScoped@@ -86,31 +88,34 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {private static final Duration EXIT_WATCH_REARM = Duration.ofSeconds(1);/**- * A started container. {@code acquire} reserves it for one request and {@code release} gives- * the reservation back; the idle reaper only stops an environment with no reservation.+ * One container of an endpoint's pool. It joins the pool before it starts, so requests that+ * arrive while it initializes wait for it instead of starting more. {@code acquire} reserves it+ * for one request and {@code release} gives the reservation back; the idle reaper only stops an+ * environment with no reservation.*/public static final class Environment {- private final String key;+ private final String poolKey;+ private final String revisionId;private final String accountId;- private final String containerId;- private final String host;- private final int port;- private final String logStream;- private final Optional<SessionCreds> roleCredentials;+ private final CompletableFuture<Void> ready = new CompletableFuture<>();+ private volatile String containerId;+ private volatile String host;+ private volatile int port;+ private volatile String logStream;+ private volatile SessionCreds roleCredentials;private final AtomicInteger inFlight = new AtomicInteger();private final AtomicLong lastUsed = new AtomicLong(System.nanoTime());private final AtomicBoolean alive = new AtomicBoolean(true);private final AtomicReference<Instant> exitedAt = new AtomicReference<>();- Environment(String key, String accountId, String containerId, String host, int port, String logStream,- Optional<SessionCreds> roleCredentials) {- this.key = key;+ Environment(String poolKey, String revisionId, String accountId) {+ this.poolKey = poolKey;+ this.revisionId = revisionId;this.accountId = accountId;- this.containerId = containerId;- this.host = host;- this.port = port;- this.logStream = logStream;- this.roleCredentials = roleCredentials;+ }++ private boolean isReady() {+ return ready.isDone() && !ready.isCompletedExceptionally();}public String containerId() {@@ -158,6 +163,13 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {}}+ /** Every environment of the endpoint is full and {@code maxEnvironments} allows no more. */+ public static final class Throttled extends RuntimeException {+ Throttled() {+ super(LambdaWebErrorPages.ENDPOINT_MAX_ENV_THROTTLE.errorType());+ }+ }+private final EmulatorConfig config;private final ContainerBuilder containerBuilder;private final ContainerLifecycleManager lifecycleManager;@@ -167,8 +179,8 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {private final ZipExtractor zipExtractor;private final LaunchedContainerAwsEnv awsEnv;private final LambdaExecutionRoleCredentials executionRoleCredentials;- private final ConcurrentHashMap<String, Environment> environments = new ConcurrentHashMap<>();- private final ConcurrentHashMap<String, Object> locks = new ConcurrentHashMap<>();+ /** The environments of each endpoint in each region. A list is only touched inside a map operation on its key. */+ private final ConcurrentHashMap<String, List<Environment>> pools = new ConcurrentHashMap<>();private final ScheduledExecutorService reaper = Executors.newSingleThreadScheduledExecutor(task -> {Thread thread = new Thread(task, "lambda-web-environments");thread.setDaemon(true);@@ -195,31 +207,123 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {}/**- * The running environment of {@code revision}, started if there is none, reserved for one- * request. The caller releases it when the request is over. Blocks on Docker and S3.+ * An environment of the endpoint for {@code revision} with room for one more request, reserved+ * for it. The least busy one is taken, the one idle longest on a tie, so requests rotate. When+ * all are full a new one starts. At {@code maxEnvironments} an idle environment of another+ * revision makes room, and without one the request is {@link Throttled}. {@code endpointKey}+ * names one creation of the endpoint in one region, so a recreated endpoint starts afresh. The+ * caller releases the environment when the request is over. Blocks on Docker and S3.*/- public Environment acquire(String accountId, String region, WebFunction fn, WebFunctionRevision revision) {- String key = String.join("|", accountId, region, fn.functionName, revision.revisionId);- synchronized (lockFor(key)) {- Environment existing = environments.get(key);- if (existing != null && existing.isAlive()) {- existing.reserve();- return existing;+ public Environment acquire(String accountId, String region, WebFunction fn, String endpointKey,+ Integer maxEnvironments, WebFunctionRevision revision) {+ int perEnvironment = Math.max(1, revision.serviceConfig.maxConcurrencyPerEnvironment);+ AtomicReference<Pick> picked = new AtomicReference<>();+ pools.compute(endpointKey, (key, current) -> {+ List<Environment> pool = current == null ? new ArrayList<>() : current;+ picked.set(pick(pool, endpointKey, accountId, revision.revisionId, perEnvironment, maxEnvironments));+ return pool;+ });+ Pick pick = picked.get();+ if (pick.retired() != null) {+ reaper.execute(() -> stopRetired(pick.retired()));+ }+ Environment chosen = pick.environment();+ if (!pick.starter()) {+ awaitReady(chosen);+ return chosen;+ }+ try {+ start(chosen, region, fn, revision);+ } catch (RuntimeException e) {+ chosen.alive.set(false);+ removeFromPool(chosen);+ chosen.ready.completeExceptionally(e);+ throw e;+ }+ if (!chosen.isAlive()) {+ IllegalStateException stopped = new IllegalStateException("The environment was stopped while it started");+ chosen.ready.completeExceptionally(stopped);+ stopContainer(chosen);+ throw stopped;+ }+ chosen.ready.complete(null);+ return chosen;+ }++ private record Pick(Environment environment, boolean starter, Environment retired) {+ }++ private static Pick pick(List<Environment> pool, String endpointKey, String accountId, String revisionId,+ int perEnvironment, Integer maxEnvironments) {+ Environment open = pool.stream()+ .filter(each -> each.isAlive() && each.revisionId.equals(revisionId)+ && each.inFlight.get() < perEnvironment)+ .min(Comparator.comparingInt((Environment each) -> each.inFlight.get())+ .thenComparingLong(each -> each.lastUsed.get()))+ .orElse(null);+ if (open != null) {+ open.reserve();+ return new Pick(open, false, null);+ }+ Environment retired = null;+ if (maxEnvironments != null && pool.size() >= maxEnvironments) {+ retired = pool.stream()+ .filter(each -> !each.revisionId.equals(revisionId) && isIdle(each, 0))+ .min(Comparator.comparingLong(each -> each.lastUsed.get()))+ .orElseThrow(Throttled::new);+ pool.remove(retired);+ }+ Environment started = new Environment(endpointKey, revisionId, accountId);+ pool.add(started);+ started.reserve();+ return new Pick(started, true, retired);+ }++ private static boolean isIdle(Environment environment, long idleNanos) {+ return environment.isAlive() && environment.isReady() && environment.inFlight.get() == 0+ && System.nanoTime() - environment.lastUsed.get() >= idleNanos;+ }++ private void stopRetired(Environment environment) {+ try {+ LOG.infov("Removing the Lambda Web environment {0} of revision {1} to make room",+ environment.containerId, environment.revisionId);+ discard(environment);+ } catch (RuntimeException e) {+ LOG.warnv(e, "Could not remove the Lambda Web environment {0}", environment.containerId);+ }+ }++ private static void awaitReady(Environment environment) {+ try {+ environment.ready.get();+ } catch (ExecutionException e) {+ if (e.getCause() instanceof RuntimeException runtime) {+ throw runtime;}- Environment started = start(key, accountId, region, fn, revision);- started.reserve();- environments.put(key, started);- return started;+ throw new IllegalStateException(e.getCause());+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();+ throw new IllegalStateException(e);}}+ private void removeFromPool(Environment environment) {+ pools.computeIfPresent(environment.poolKey, (key, pool) -> {+ pool.remove(environment);+ return pool.isEmpty() ? null : pool;+ });+ }+/** Stops an environment, with the one second AWS gives the process after SIGTERM. */public void discard(Environment environment) {+ removeFromPool(environment);if (!environment.alive.compareAndSet(true, false)) {- environments.remove(environment.key, environment);return;}- environments.remove(environment.key, environment);+ if (environment.containerId == null) {+ return;+ }try {exitCodeIfStopped(environment.containerId).ifPresent(exitCode ->LOG.infov("The Lambda Web environment {0} had exited with status {1}: {2}",@@ -227,6 +331,10 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {} catch (RuntimeException e) {LOG.debugv(e, "Could not inspect the Lambda Web environment {0}", environment.containerId);}+ stopContainer(environment);+ }++ private void stopContainer(Environment environment) {try {lifecycleManager.stopAndRemove(environment.containerId, null, 1);} finally {@@ -252,7 +360,18 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {@Overridepublic void stopManagedContainers() {- new ArrayList<>(environments.values()).forEach(this::discard);+ allEnvironments().forEach(this::discard);+ }++ private List<Environment> allEnvironments() {+ List<Environment> all = new ArrayList<>();+ for (String key : pools.keySet()) {+ pools.computeIfPresent(key, (ignored, pool) -> {+ all.addAll(pool);+ return pool;+ });+ }+ return all;}@PreDestroy@@ -260,30 +379,30 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {reaper.shutdownNow();}- private Object lockFor(String key) {- return locks.computeIfAbsent(key, ignored -> new Object());- }-private void reapIdle() {long idleNanos = TimeUnit.SECONDS.toNanos(config.services().lambda().containerIdleTimeoutSeconds());- for (Environment environment : new ArrayList<>(environments.values())) {- try {- synchronized (lockFor(environment.key)) {- if (environment.inFlight.get() != 0- || System.nanoTime() - environment.lastUsed.get() <= idleNanos) {- continue;- }- LOG.infov("Removing the idle Lambda Web environment {0}", environment.containerId);- discard(environment);+ for (Environment environment : allEnvironments()) {+ AtomicBoolean idle = new AtomicBoolean();+ pools.computeIfPresent(environment.poolKey, (key, pool) -> {+ if (isIdle(environment, idleNanos) && pool.remove(environment)) {+ idle.set(true);}+ return pool.isEmpty() ? null : pool;+ });+ if (!idle.get()) {+ continue;+ }+ try {+ LOG.infov("Removing the idle Lambda Web environment {0}", environment.containerId);+ discard(environment);} catch (RuntimeException e) {LOG.warnv(e, "Could not remove the idle Lambda Web environment {0}", environment.containerId);}}}- private Environment start(String key, String accountId, String region, WebFunction fn,- WebFunctionRevision revision) {+ private void start(Environment environment, String region, WebFunction fn, WebFunctionRevision revision) {+ String accountId = environment.accountId;String image = ecrRegistryManager.rewriteImageUri(imageResolver.resolve(revision.buildConfig.runtime));String shortId = UUID.randomUUID().toString().replace("-", "").substring(0, 8);String name = ContainerStorageHelper.prefixedDockerName(config, "floci-lambda-web",@@ -320,14 +439,15 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {ContainerInfo info = lifecycleManager.startCreated(containerId, spec);EndpointInfo endpoint = info.getEndpoint(APP_PORT);awaitListening(containerId);- String logStream = LOG_STREAM_DATE.format(Instant.now()) + "/" + fn.functionName + "[" + revision.revisionId- + "]" + UUID.randomUUID().toString().replace("-", "");- Environment environment = new Environment(key, accountId, containerId, endpoint.host(), endpoint.port(),- logStream, roleCredentials);+ environment.containerId = containerId;+ environment.host = endpoint.host();+ environment.port = endpoint.port();+ environment.logStream = LOG_STREAM_DATE.format(Instant.now()) + "/" + fn.functionName + "["+ + revision.revisionId + "]" + UUID.randomUUID().toString().replace("-", "");+ environment.roleCredentials = roleCredentials.orElse(null);watchExit(environment);LOG.infov("Started the Lambda Web environment {0} of {1} revision {2} at {3}:{4}",containerId, fn.functionName, revision.revisionId, endpoint.host(), endpoint.port());- return environment;} catch (RuntimeException e) {lifecycleManager.stopAndRemove(containerId, null, 1);roleCredentials.ifPresent(creds -> executionRoleCredentials.unregister(accountId, creds.accessKeyId()));@@ -351,8 +471,10 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {}private void forgetCredentials(Environment environment) {- environment.roleCredentials.ifPresent(creds ->- executionRoleCredentials.unregister(environment.accountId, creds.accessKeyId()));+ SessionCreds credentials = environment.roleCredentials;+ if (credentials != null) {+ executionRoleCredentials.unregister(environment.accountId, credentials.accessKeyId());+ }}/** The code goes in by {@code docker cp}, so it works when Floci itself runs in a container. */@@ -559,7 +681,7 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {return;}environment.exitedAt.compareAndSet(null, finishedAt(environment.containerId).orElse(Instant.now()));- environments.remove(environment.key, environment);+ removeFromPool(environment);reaper.execute(() -> {try {LOG.infov("The Lambda Web environment {0} exited with status {1}: {2}",diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebErrorPages.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebErrorPages.javaindex d62add6c65..942f91ee53 100644--- a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebErrorPages.java+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebErrorPages.java@@ -27,6 +27,11 @@ final class LambdaWebErrorPages {APPLICATION_ERROR);static final Page INIT_CODE_ERROR = new Page(502, "Bad Gateway", "Application.InitCodeError", APPLICATION_ERROR);static final Page RUNTIME_EXIT = new Page(502, "Bad Gateway", "Runtime.Exit", APPLICATION_ERROR);+ private static final String RATE_EXCEEDED = "Request rate exceeded. Please try again later.";+ static final Page ENDPOINT_MAX_ENV_THROTTLE = new Page(429, "Too Many Requests",+ "Platform.EndpointMaxEnvThrottle", RATE_EXCEEDED);+ static final Page ENDPOINT_REQUEST_THROTTLE = new Page(429, "Too Many Requests",+ "Platform.EndpointRequestThrottle", RATE_EXCEEDED);/** The edge in front of the platform refuses a method itself, with its own page and header casing. */static final String MALFORMED_REQUEST_TYPE = "Platform.MalformedRequest";diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebRateLimiter.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebRateLimiter.javanew file mode 100644index 0000000000..7b0133febb--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebRateLimiter.java@@ -0,0 +1,68 @@+package io.github.hectorvent.floci.services.lambdaweb;++import java.util.concurrent.ConcurrentHashMap;+import java.util.concurrent.TimeUnit;+import java.util.concurrent.atomic.AtomicBoolean;+import java.util.concurrent.atomic.AtomicLong;+import java.util.function.LongSupplier;++/**+ * The {@code rateLimit} of each endpoint: a bucket of {@code rateLimit} requests that refills at+ * {@code rateLimit} per second, so a brief burst above the rate passes and a sustained one does not.+ */+final class LambdaWebRateLimiter {++ private static final long SECOND = TimeUnit.SECONDS.toNanos(1);++ private record Bucket(int rate, double tokens, long at) {+ }++ private final ConcurrentHashMap<String, Bucket> buckets = new ConcurrentHashMap<>();+ private final LongSupplier nanoClock;+ private final AtomicLong sweptAt;++ LambdaWebRateLimiter() {+ this(System::nanoTime);+ }++ LambdaWebRateLimiter(LongSupplier nanoClock) {+ this.nanoClock = nanoClock;+ this.sweptAt = new AtomicLong(nanoClock.getAsLong());+ }++ /** Whether one more request to {@code endpointKey} fits under {@code rate} requests per second. */+ boolean tryAcquire(String endpointKey, int rate) {+ sweep();+ AtomicBoolean admitted = new AtomicBoolean();+ buckets.compute(endpointKey, (key, bucket) -> {+ double tokens;+ long now = nanoClock.getAsLong();+ if (bucket == null || bucket.rate() != rate) {+ tokens = rate;+ } else {+ now = Math.max(now, bucket.at());+ double refill = rate * (double) (now - bucket.at()) / SECOND;+ tokens = Math.min(rate, bucket.tokens() + refill);+ }+ if (tokens >= 1) {+ admitted.set(true);+ tokens -= 1;+ }+ return new Bucket(rate, tokens, now);+ });+ return admitted.get();+ }++ int size() {+ return buckets.size();+ }++ /** A bucket untouched for a second is full again, the same as no bucket. */+ private void sweep() {+ long now = nanoClock.getAsLong();+ long last = sweptAt.get();+ if (now - last >= SECOND && sweptAt.compareAndSet(last, now)) {+ buckets.values().removeIf(bucket -> now - bucket.at() >= SECOND);+ }+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.javaindex beedfb9251..6c700263a0 100644--- a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java@@ -18,7 +18,17 @@ import java.io.InputStream;import java.io.OutputStream;import java.net.Socket;import java.nio.charset.StandardCharsets;+import java.util.ArrayList;+import java.util.HashSet;+import java.util.List;import java.util.Map;+import java.util.Set;+import java.util.concurrent.Callable;+import java.util.concurrent.CyclicBarrier;+import java.util.concurrent.ExecutorService;+import java.util.concurrent.Executors;+import java.util.concurrent.Future;+import java.util.concurrent.TimeUnit;import java.util.zip.ZipEntry;import java.util.zip.ZipOutputStream;@@ -50,6 +60,7 @@ class LambdaWebDataPlaneDockerIntegrationTest {const [host, port] = process.env.AWS_LAMBDA_HTTP_ENDPOINT.split(':');http.createServer((req, res) => {if (req.url.startsWith('/slow')) { setTimeout(() => res.end('late'), 8000); return; }+ if (req.url.startsWith('/wait')) { setTimeout(() => res.end('waited'), 3000); return; }if (req.url.startsWith('/stream')) {res.writeHead(200, {'content-type': 'text/plain'});let i = 0;@@ -181,7 +192,97 @@ class LambdaWebDataPlaneDockerIntegrationTest {assertNotEquals(logStream, after.header("lambda-web-logstream"), "a new environment serves the next request");}+ @Test+ @Timeout(240)+ void environmentsScaleUpToMaxEnvironmentsAndTheRateLimitThrottles() throws Exception {+ String domain = createFunction(30, 1);+ given().contentType("application/json").body("{\"scalingConfig\": {\"maxEnvironments\": 2}}")+ .when().patch(BASE + "/web-functions/" + functionName + "/endpoints/dev")+ .then().statusCode(202);++ // One request per environment: two run, the rest are refused, as recorded on AWS.+ List<String> responses = parallel(5, () -> raw("GET /wait HTTP/1.1\r\nHost: " + domain+ + "\r\nConnection: close\r\n\r\n"));+ long served = responses.stream().filter(response -> response.startsWith("HTTP/1.1 200 ")).count();+ List<String> throttled = responses.stream().filter(response -> response.startsWith("HTTP/1.1 429 ")).toList();+ assertEquals(2, served, String.join("\n----\n", responses));+ assertEquals(3, throttled.size(), String.join("\n----\n", responses));+ for (String response : throttled) {+ assertTrue(response.contains("\r\nlambda-web-error-type: Platform.EndpointMaxEnvThrottle\r\n"), response);+ assertTrue(response.contains("<p>Request rate exceeded. Please try again later.</p>"), response);+ }++ // Requests one after another rotate over the two environments.+ Set<String> logStreams = new HashSet<>();+ for (int i = 0; i < 4; i++) {+ logStreams.add(given().header("Host", domain).when().get("/status/200")+ .then().statusCode(200).extract().header("lambda-web-logstream"));+ }+ assertEquals(2, logStreams.size(), logStreams.toString());++ // A new revision takes the place of an idle environment of the old one.+ String revisionId = given().contentType("application/json")+ .body("""+ {"buildConfig": {"codeConfig": {"s3Object": {"bucket": "%s", "key": "%s.zip"}},+ "runtimeConfig": {"runtime": "nodejs24.x"}},+ "serviceConfig": {"executionRoleArn": "%s", "timeoutSeconds": 30}}+ """.formatted(BUCKET, functionName, ROLE))+ .when().post(BASE + "/web-functions/" + functionName + "/revisions")+ .then().statusCode(202)+ .extract().path("revisionId");+ String moved = given().header("Host", domain).when().get("/status/200")+ .then().statusCode(200).extract().header("lambda-web-logstream");+ assertTrue(moved.contains("[" + revisionId + "]"), moved);+ logStreams.add(moved);++ // A recreated endpoint shares no environment or rate with the deleted one.+ String endpoints = BASE + "/web-functions/" + functionName + "/endpoints";+ given().when().delete(endpoints + "/dev").then().statusCode(204);+ String recreated = given().contentType("application/json")+ .body("{\"endpointName\": \"dev\", \"endpointType\": \"HomeRegion\","+ + " \"authType\": \"ApplicationManaged\", \"autoDeploymentMode\": \"LatestRevision\","+ + " \"throttleConfig\": {\"rateLimit\": 100}}")+ .when().put(endpoints)+ .then().statusCode(202)+ .extract().path("domainName");+ String fresh = given().header("Host", recreated).when().get("/status/200")+ .then().statusCode(200).extract().header("lambda-web-logstream");+ assertFalse(logStreams.contains(fresh), fresh + " " + logStreams);++ // Requests that all arrive at once are only admitted up to the rate.+ int count = 200;+ CyclicBarrier together = new CyclicBarrier(count);+ List<String> burst = parallel(count, () -> raw("GET /status/200 HTTP/1.1\r\nHost: " + recreated+ + "\r\nConnection: close\r\n\r\n", together));+ List<String> limited = burst.stream().filter(response -> response.startsWith("HTTP/1.1 429 ")).toList();+ assertFalse(limited.isEmpty(), "a burst well past the rate is throttled");+ for (String response : limited) {+ assertTrue(response.contains("\r\nlambda-web-error-type: Platform.EndpointRequestThrottle\r\n"), response);+ }+ }++ private static List<String> parallel(int count, Callable<String> call) throws Exception {+ ExecutorService pool = Executors.newFixedThreadPool(count);+ try {+ List<Future<String>> futures = new ArrayList<>();+ for (int i = 0; i < count; i++) {+ futures.add(pool.submit(call));+ }+ List<String> results = new ArrayList<>();+ for (Future<String> future : futures) {+ results.add(future.get(60, TimeUnit.SECONDS));+ }+ return results;+ } finally {+ pool.shutdownNow();+ }+ }+private String createFunction(int timeoutSeconds) throws IOException {+ return createFunction(timeoutSeconds, 64);+ }++ private String createFunction(int timeoutSeconds, int maxConcurrencyPerEnvironment) throws IOException {functionName = "dp-docker-" + Long.toString(System.nanoTime(), 36);s3Service.createBucket(BUCKET, "us-east-1");s3Service.putObject(BUCKET, functionName + ".zip", zip(APP), "application/zip", Map.of());@@ -192,19 +293,29 @@ class LambdaWebDataPlaneDockerIntegrationTest {"buildConfig": {"codeConfig": {"s3Object": {"bucket": "%s", "key": "%s.zip"}},"runtimeConfig": {"runtime": "nodejs24.x"}},"serviceConfig": {"executionRoleArn": "%s", "timeoutSeconds": %d,+ "maxConcurrencyPerEnvironment": %d,"environmentVariables": {"CUSTOM": "yes"}}},"endpointConfig": {"endpointName": "dev", "endpointType": "HomeRegion","authType": "ApplicationManaged", "autoDeploymentMode": "LatestRevision"}}- """.formatted(functionName, BUCKET, functionName, ROLE, timeoutSeconds))+ """.formatted(functionName, BUCKET, functionName, ROLE, timeoutSeconds,+ maxConcurrencyPerEnvironment)).when().put(BASE + "/web-functions").then().statusCode(202).extract().path("endpoint.domainName");}/** Sends one raw HTTP/1.1 request with {@code Connection: close} and reads until the server closes. */- private static String raw(String request) throws IOException {+ private static String raw(String request) throws Exception {+ return raw(request, null);+ }++ /** As {@link #raw(String)}, sent once every party of {@code together} has connected. */+ private static String raw(String request, CyclicBarrier together) throws Exception {try (Socket socket = new Socket("localhost", RestAssured.port)) {socket.setSoTimeout(30_000);+ if (together != null) {+ together.await(30, TimeUnit.SECONDS);+ }OutputStream out = socket.getOutputStream();out.write(request.getBytes(StandardCharsets.US_ASCII));out.flush();diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneIntegrationTest.javaindex dc1682927a..1a6af4b93e 100644--- a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneIntegrationTest.java@@ -51,7 +51,7 @@ class LambdaWebDataPlaneIntegrationTest {}@Test- void aDisabledEndpointAnswersBeforeAnyEnvironmentStarts() {+ void aDisabledEndpointAnswersBeforeAnyEnvironmentStarts() throws IOException {String functionName = "dp-disabled-" + Long.toString(System.nanoTime(), 36);String domain = createFunction(functionName);given().contentType("application/json").body("{\"throttleConfig\": {\"rateLimit\": 0}}")@@ -65,9 +65,40 @@ class LambdaWebDataPlaneIntegrationTest {.body(equalTo(LambdaWebErrorPages.html(LambdaWebErrorPages.ENDPOINT_DISABLED,response.header("lambda-web-request-id"))));+ // The page keeps the connection, as on AWS. A body sent after it is skipped and the next request is served.+ try (Socket socket = new Socket("localhost", RestAssured.port)) {+ socket.setSoTimeout(10_000);+ OutputStream out = socket.getOutputStream();+ InputStream in = socket.getInputStream();+ out.write(("POST /upload HTTP/1.1\r\nHost: " + domain + "\r\nContent-Length: 5\r\n\r\n")+ .getBytes(StandardCharsets.US_ASCII));+ out.flush();+ String first = readUntil(in, "</html>");+ assertTrue(first.startsWith("HTTP/1.1 503 "), first);+ assertTrue(first.contains("\r\nconnection: keep-alive\r\n"), first);+ out.write(("hello" + "GET /next HTTP/1.1\r\nHost: " + domain + "\r\nConnection: close\r\n\r\n")+ .getBytes(StandardCharsets.US_ASCII));+ out.flush();+ String second = new String(in.readAllBytes(), StandardCharsets.UTF_8);+ assertTrue(second.startsWith("HTTP/1.1 503 "), second);+ assertTrue(second.contains("\r\nlambda-web-error-type: Platform.EndpointDisabled\r\n"), second);+ }+given().when().delete(BASE + "/web-functions/" + functionName).then().statusCode(204);}+ private static String readUntil(InputStream in, String end) throws IOException {+ StringBuilder read = new StringBuilder();+ while (!read.toString().endsWith(end)) {+ int next = in.read();+ if (next < 0) {+ break;+ }+ read.append((char) next);+ }+ return read.toString();+ }+@Testvoid httpOneDotZeroIsRefused() throws IOException {String response = raw("GET / HTTP/1.0\r\nHost: " + UNKNOWN_HOST + "\r\n\r\n");diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebErrorPagesTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebErrorPagesTest.javaindex 5599ac5351..6a697d4916 100644--- a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebErrorPagesTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebErrorPagesTest.java@@ -23,6 +23,8 @@ class LambdaWebErrorPagesTest {assertEquals(228, length(LambdaWebErrorPages.TIMEOUT_LISTEN_ERROR));assertEquals(228, length(LambdaWebErrorPages.INIT_CODE_ERROR));assertEquals(228, length(LambdaWebErrorPages.RUNTIME_EXIT));+ assertEquals(237, length(LambdaWebErrorPages.ENDPOINT_MAX_ENV_THROTTLE));+ assertEquals(237, length(LambdaWebErrorPages.ENDPOINT_REQUEST_THROTTLE));assertEquals(180, LambdaWebErrorPages.MALFORMED_REQUEST_HTML.getBytes(StandardCharsets.UTF_8).length);}diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebRateLimiterTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebRateLimiterTest.javanew file mode 100644index 0000000000..f075cd2fea--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebRateLimiterTest.java@@ -0,0 +1,69 @@+package io.github.hectorvent.floci.services.lambdaweb;++import org.junit.jupiter.api.Test;++import java.util.concurrent.TimeUnit;+import java.util.concurrent.atomic.AtomicLong;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertTrue;++class LambdaWebRateLimiterTest {++ private final AtomicLong now = new AtomicLong();+ private final LambdaWebRateLimiter limiter = new LambdaWebRateLimiter(now::get);++ @Test+ void aBurstUpToTheRatePassesAndTheNextRequestIsThrottled() {+ assertEquals(100, admitted("ep", 100, 150));+ }++ @Test+ void theBucketRefillsAtTheRatePerSecond() {+ assertEquals(100, admitted("ep", 100, 100));+ assertFalse(limiter.tryAcquire("ep", 100));+ now.addAndGet(TimeUnit.MILLISECONDS.toNanos(500));+ assertEquals(50, admitted("ep", 100, 80));+ now.addAndGet(TimeUnit.SECONDS.toNanos(10));+ assertEquals(100, admitted("ep", 100, 150), "the bucket holds at most one second of requests");+ }++ @Test+ void endpointsHaveTheirOwnBucketsAndANewRateStartsAFullOne() {+ assertEquals(100, admitted("a", 100, 100));+ assertTrue(limiter.tryAcquire("b", 100));+ assertFalse(limiter.tryAcquire("a", 100));+ assertEquals(200, admitted("a", 200, 250));+ }++ @Test+ void aClockReadOutOfOrderTakesNoTokensBack() {+ now.set(1_000);+ assertTrue(limiter.tryAcquire("ep", 2));+ now.set(0);+ assertTrue(limiter.tryAcquire("ep", 2));+ assertFalse(limiter.tryAcquire("ep", 2));+ }++ @Test+ void bucketsThatRefilledAreDropped() {+ assertEquals(100, admitted("a", 100, 100));+ assertTrue(limiter.tryAcquire("b", 100));+ assertEquals(2, limiter.size());+ now.addAndGet(TimeUnit.SECONDS.toNanos(1));+ assertTrue(limiter.tryAcquire("c", 100));+ assertEquals(1, limiter.size());+ assertEquals(100, admitted("a", 100, 150));+ }++ private int admitted(String endpoint, int rate, int requests) {+ int admitted = 0;+ for (int i = 0; i < requests; i++) {+ if (limiter.tryAcquire(endpoint, rate)) {+ admitted++;+ }+ }+ return admitted;+ }+}