Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(lambdaweb): scale environments per endpoint and apply maxEnvironments and rateLimit (#5785)

TATSUNO “Taz” Yasuhiro832f1008bb6e
diff --git a/docs/services/lambda-web-functions.md b/docs/services/lambda-web-functions.md
index f4e3b19aff..7ab7f4114d 100644
--- a/docs/services/lambda-web-functions.md
+++ b/docs/services/lambda-web-functions.md
@@ -41,7 +41,7 @@ The `Lambda Web` service model (apiVersion 2025-03-07, `aws lambda-web` in the A
- **ARNs** given as identifiers must name the request's region and the caller's account, as on AWS. A revision or endpoint ARN resolves by its id or name only.
- **Page tokens** are base64 cursors that carry the request and the position of the page, not the UUIDs AWS issues. A token reused with other parameters gets the AWS "does not match the request" error, and any other token the "invalid or has expired" error.
- **Domain names** take the form `<id>.001.lambda-web.<region>.<floci host>` and `<id>.lambda-web.global.<floci host>`, with the host and port of `floci.base-url`. A `PerRegion` endpoint gets one domain per region in `regionalEndpoints`.
-- **Not implemented yet:** `IamAuth` on requests, the automatic deletion of revisions past 50, the scaling limits (`maxConcurrencyPerEnvironment`, `maxEnvironments`, a `rateLimit` other than 0), the metadata API, the logs and metrics, and serving a `MultiRegion` or `PerRegion` endpoint in its other regions.
+- **Not implemented yet:** `IamAuth` on requests, the automatic deletion of revisions past 50, the metadata API, the logs and metrics, and serving a `MultiRegion` or `PerRegion` endpoint in its other regions.
### `aws lambda-web deploy`
@@ -64,7 +64,9 @@ curl -H "Host: <domainName>" http://localhost:4566/
The `Host` must match the domain exactly, as on AWS: another case or another port is `404` with `lambda-web-error-type: Platform.InvalidEndpoint`.
-Floci runs each revision in one execution environment, a `public.ecr.aws/lambda/nodejs:24` container started by the first request. The code is the revision's S3 object unpacked into `/var/task`, the entry file is `index.js` or the file `AWS_LAMBDA_NODEJS_ENTRYPOINT` names, and node runs with the flags AWS uses. The app sees `AWS_LAMBDA_HTTP_ENDPOINT=0.0.0.0:3000`, `AWS_LAMBDA_INITIALIZATION_TYPE=native-http`, `AWS_EXECUTION_ENV`, `AWS_LAMBDA_FUNCTION_MEMORY_SIZE=2048`, `AWS_LAMBDA_LOG_LEVEL`, `AWS_ACCOUNT_ID`, `AWS_REGION`, credentials and `AWS_ENDPOINT_URL` for Floci, `NODE_PATH`, and its own variables. The server has 10 seconds to listen. An environment that sees no request for `floci.services.lambda.container-idle-timeout-seconds` is stopped with `SIGTERM` and one second, as on scale to zero. A process that exits between requests is replaced by a fresh environment on the next request.
+An execution environment is a `public.ecr.aws/lambda/nodejs:24` container that a request starts when every environment of the endpoint is full. The code is the revision's S3 object unpacked into `/var/task`, the entry file is `index.js` or the file `AWS_LAMBDA_NODEJS_ENTRYPOINT` names, and node runs with the flags AWS uses. The app sees `AWS_LAMBDA_HTTP_ENDPOINT=0.0.0.0:3000`, `AWS_LAMBDA_INITIALIZATION_TYPE=native-http`, `AWS_EXECUTION_ENV`, `AWS_LAMBDA_FUNCTION_MEMORY_SIZE=2048`, `AWS_LAMBDA_LOG_LEVEL`, `AWS_ACCOUNT_ID`, `AWS_REGION`, credentials and `AWS_ENDPOINT_URL` for Floci, `NODE_PATH`, and its own variables. The server has 10 seconds to listen. An environment that sees no request for `floci.services.lambda.container-idle-timeout-seconds` is stopped with `SIGTERM` and one second, as on scale to zero. A process that exits between requests is replaced by a fresh environment on the next request.
+
+Each endpoint scales on its own, as on AWS. An environment takes up to the revision's `maxConcurrencyPerEnvironment` requests at once, the least busy one first, so requests rotate over the environments. A request that finds every environment full starts another one, or waits for one that is starting and has room. When the endpoint already runs `maxEnvironments`, an idle environment of another revision makes room, and without one the request is refused with `429` and `Platform.EndpointMaxEnvThrottle`. A `rateLimit` admits a burst of up to `rateLimit` requests that refills at `rateLimit` per second, and refuses the rest with `429` and `Platform.EndpointRequestThrottle`. AWS adjusts the concurrency of each environment under `maxConcurrencyPerEnvironment` from the measured load, so a burst there can spread over more environments than in Floci, and the account quotas are not applied.
Requests are forwarded over HTTP/1.1 with the headers AWS adds and drops: `lambda-web-request-id`, `lambda-web-endpoint-name`, `x-forwarded-proto`, `x-forwarded-port` and `Connection: close` are set, the client address is appended as an `X-Forwarded-For` line, and `Connection`, `Keep-Alive`, `Upgrade`, `Expect`, `Trailer`, the `Proxy-*` headers, `X-Real-IP`, the method override headers and any `lambda-web-*` header the client sent are dropped. A body without a `Content-Length` reaches the app chunked. Responses carry `lambda-web-request-id` and `lambda-web-logstream`, lose their connection headers and trailers, and stream as the app writes them. Requests with two revisions in `revisionWeights` are split by weight.
@@ -78,6 +80,8 @@ The platform answers these cases itself, with the AWS HTML pages and `lambda-web
| --- | --- | --- |
| 400 | `Platform.MalformedRequest` | A method other than `GET`, `HEAD`, `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` |
| 404 | `Platform.InvalidEndpoint` | The `Host` is a `lambda-web` domain Floci did not mint |
+| 429 | `Platform.EndpointMaxEnvThrottle` | Every environment is full and the endpoint runs `maxEnvironments` |
+| 429 | `Platform.EndpointRequestThrottle` | The requests exceed the endpoint's `rateLimit` |
| 502 | `Application.TimeoutListenError`, `Application.InitCodeError`, `Runtime.Exit` | The server did not listen within 10 seconds, threw at import, or exited at start |
| 503 | `Platform.EndpointDisabled` | The endpoint's `rateLimit` is 0 |
| 503 | `Runtime.OriginRequestError` | The process died while handling the request |
diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlane.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlane.java
index ffbfdc754a..77b2b13f68 100644
--- a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlane.java
+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlane.java
@@ -4,6 +4,7 @@ import io.github.hectorvent.floci.core.common.RequestHost;
import io.github.hectorvent.floci.core.common.ServiceConfigAccess;
import io.github.hectorvent.floci.services.lambdaweb.LambdaWebEnvironmentManager.Environment;
import io.github.hectorvent.floci.services.lambdaweb.LambdaWebEnvironmentManager.StartFailure;
+import io.github.hectorvent.floci.services.lambdaweb.LambdaWebEnvironmentManager.Throttled;
import io.github.hectorvent.floci.services.lambdaweb.LambdaWebService.WebFunctionTarget;
import io.github.hectorvent.floci.services.lambdaweb.model.WebFunctionRevision;
import io.vertx.core.Future;
@@ -69,6 +70,7 @@ public class LambdaWebDataPlane {
private final ServiceConfigAccess serviceConfigAccess;
private final Vertx vertx;
private final HttpClient client;
+ private final LambdaWebRateLimiter rateLimiter = new LambdaWebRateLimiter();
@Inject
public LambdaWebDataPlane(LambdaWebService service, LambdaWebEnvironmentManager environments,
@@ -109,11 +111,15 @@ public class LambdaWebDataPlane {
return;
}
WebFunctionTarget target = located.get();
- if (target.endpoint().throttleConfig != null
- && Integer.valueOf(0).equals(target.endpoint().throttleConfig.rateLimit)) {
+ Integer rateLimit = target.endpoint().throttleConfig == null ? null : target.endpoint().throttleConfig.rateLimit;
+ if (Integer.valueOf(0).equals(rateLimit)) {
page(request, LambdaWebErrorPages.ENDPOINT_DISABLED, requestId, false);
return;
}
+ if (rateLimit != null && !rateLimiter.tryAcquire(endpointKey(target), rateLimit)) {
+ page(request, LambdaWebErrorPages.ENDPOINT_REQUEST_THROTTLE, requestId, false);
+ return;
+ }
WebFunctionRevision revision = LambdaWebService.pickRevision(target);
if (revision == null) {
page(request, LambdaWebErrorPages.ORIGIN_REQUEST_ERROR, requestId, false);
@@ -126,6 +132,11 @@ public class LambdaWebDataPlane {
acquire(request, host, target, revision, requestId, null);
}
+ /** The domain is minted for each creation, so a recreated endpoint shares no state with the deleted one. */
+ private static String endpointKey(WebFunctionTarget target) {
+ return String.join("|", target.accountId(), target.region(), target.endpoint().domainName);
+ }
+
/** Starts or reuses the revision's environment off the event loop, then forwards. */
private void acquire(HttpServerRequest request, String host, WebFunctionTarget target,
WebFunctionRevision revision, String requestId, Environment dead) {
@@ -133,7 +144,10 @@ public class LambdaWebDataPlane {
if (dead != null) {
environments.discard(dead);
}
- return environments.acquire(target.accountId(), target.region(), target.function(), revision);
+ Integer maxEnvironments = target.endpoint().scalingConfig == null
+ ? null : target.endpoint().scalingConfig.maxEnvironments;
+ return environments.acquire(target.accountId(), target.region(), target.function(),
+ endpointKey(target), maxEnvironments, revision);
}, false)
.onSuccess(environment -> forward(request, host, target, revision, environment, requestId, dead == null))
.onFailure(error -> {
@@ -142,6 +156,10 @@ public class LambdaWebDataPlane {
page(request, failure.page(), requestId, false);
return;
}
+ if (error instanceof Throttled) {
+ page(request, LambdaWebErrorPages.ENDPOINT_MAX_ENV_THROTTLE, requestId, false);
+ return;
+ }
LOG.errorv(error, "Could not start an environment for {0}", host);
internalError(request, requestId);
});
diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java
index 2e2d639fd4..9c082423af 100644
--- a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java
@@ -49,7 +49,9 @@ import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
@@ -60,9 +62,9 @@ import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Stream;
/**
- * The execution environments of Web Function revisions: one Node.js server container per revision,
- * started by the first request, reused while requests keep coming and removed after the Lambda idle
- * timeout. The runtime loads {@code index.js}, or the file {@code AWS_LAMBDA_NODEJS_ENTRYPOINT} names,
+ * The execution environments of Web Function endpoints: Node.js server containers of a revision,
+ * started by requests that find no room, reused while requests keep coming and removed after the
+ * Lambda idle timeout. The runtime loads {@code index.js}, or the file {@code AWS_LAMBDA_NODEJS_ENTRYPOINT} names,
* from {@code /var/task} with the node flags AWS uses.
*/
@ApplicationScoped
@@ -86,31 +88,34 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
private static final Duration EXIT_WATCH_REARM = Duration.ofSeconds(1);
/**
- * A started container. {@code acquire} reserves it for one request and {@code release} gives
- * the reservation back; the idle reaper only stops an environment with no reservation.
+ * One container of an endpoint's pool. It joins the pool before it starts, so requests that
+ * arrive while it initializes wait for it instead of starting more. {@code acquire} reserves it
+ * for one request and {@code release} gives the reservation back; the idle reaper only stops an
+ * environment with no reservation.
*/
public static final class Environment {
- private final String key;
+ private final String poolKey;
+ private final String revisionId;
private final String accountId;
- private final String containerId;
- private final String host;
- private final int port;
- private final String logStream;
- private final Optional<SessionCreds> roleCredentials;
+ private final CompletableFuture<Void> ready = new CompletableFuture<>();
+ private volatile String containerId;
+ private volatile String host;
+ private volatile int port;
+ private volatile String logStream;
+ private volatile SessionCreds roleCredentials;
private final AtomicInteger inFlight = new AtomicInteger();
private final AtomicLong lastUsed = new AtomicLong(System.nanoTime());
private final AtomicBoolean alive = new AtomicBoolean(true);
private final AtomicReference<Instant> exitedAt = new AtomicReference<>();
- Environment(String key, String accountId, String containerId, String host, int port, String logStream,
- Optional<SessionCreds> roleCredentials) {
- this.key = key;
+ Environment(String poolKey, String revisionId, String accountId) {
+ this.poolKey = poolKey;
+ this.revisionId = revisionId;
this.accountId = accountId;
- this.containerId = containerId;
- this.host = host;
- this.port = port;
- this.logStream = logStream;
- this.roleCredentials = roleCredentials;
+ }
+
+ private boolean isReady() {
+ return ready.isDone() && !ready.isCompletedExceptionally();
}
public String containerId() {
@@ -158,6 +163,13 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
}
}
+ /** Every environment of the endpoint is full and {@code maxEnvironments} allows no more. */
+ public static final class Throttled extends RuntimeException {
+ Throttled() {
+ super(LambdaWebErrorPages.ENDPOINT_MAX_ENV_THROTTLE.errorType());
+ }
+ }
+
private final EmulatorConfig config;
private final ContainerBuilder containerBuilder;
private final ContainerLifecycleManager lifecycleManager;
@@ -167,8 +179,8 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
private final ZipExtractor zipExtractor;
private final LaunchedContainerAwsEnv awsEnv;
private final LambdaExecutionRoleCredentials executionRoleCredentials;
- private final ConcurrentHashMap<String, Environment> environments = new ConcurrentHashMap<>();
- private final ConcurrentHashMap<String, Object> locks = new ConcurrentHashMap<>();
+ /** The environments of each endpoint in each region. A list is only touched inside a map operation on its key. */
+ private final ConcurrentHashMap<String, List<Environment>> pools = new ConcurrentHashMap<>();
private final ScheduledExecutorService reaper = Executors.newSingleThreadScheduledExecutor(task -> {
Thread thread = new Thread(task, "lambda-web-environments");
thread.setDaemon(true);
@@ -195,31 +207,123 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
}
/**
- * The running environment of {@code revision}, started if there is none, reserved for one
- * request. The caller releases it when the request is over. Blocks on Docker and S3.
+ * An environment of the endpoint for {@code revision} with room for one more request, reserved
+ * for it. The least busy one is taken, the one idle longest on a tie, so requests rotate. When
+ * all are full a new one starts. At {@code maxEnvironments} an idle environment of another
+ * revision makes room, and without one the request is {@link Throttled}. {@code endpointKey}
+ * names one creation of the endpoint in one region, so a recreated endpoint starts afresh. The
+ * caller releases the environment when the request is over. Blocks on Docker and S3.
*/
- public Environment acquire(String accountId, String region, WebFunction fn, WebFunctionRevision revision) {
- String key = String.join("|", accountId, region, fn.functionName, revision.revisionId);
- synchronized (lockFor(key)) {
- Environment existing = environments.get(key);
- if (existing != null && existing.isAlive()) {
- existing.reserve();
- return existing;
+ public Environment acquire(String accountId, String region, WebFunction fn, String endpointKey,
+ Integer maxEnvironments, WebFunctionRevision revision) {
+ int perEnvironment = Math.max(1, revision.serviceConfig.maxConcurrencyPerEnvironment);
+ AtomicReference<Pick> picked = new AtomicReference<>();
+ pools.compute(endpointKey, (key, current) -> {
+ List<Environment> pool = current == null ? new ArrayList<>() : current;
+ picked.set(pick(pool, endpointKey, accountId, revision.revisionId, perEnvironment, maxEnvironments));
+ return pool;
+ });
+ Pick pick = picked.get();
+ if (pick.retired() != null) {
+ reaper.execute(() -> stopRetired(pick.retired()));
+ }
+ Environment chosen = pick.environment();
+ if (!pick.starter()) {
+ awaitReady(chosen);
+ return chosen;
+ }
+ try {
+ start(chosen, region, fn, revision);
+ } catch (RuntimeException e) {
+ chosen.alive.set(false);
+ removeFromPool(chosen);
+ chosen.ready.completeExceptionally(e);
+ throw e;
+ }
+ if (!chosen.isAlive()) {
+ IllegalStateException stopped = new IllegalStateException("The environment was stopped while it started");
+ chosen.ready.completeExceptionally(stopped);
+ stopContainer(chosen);
+ throw stopped;
+ }
+ chosen.ready.complete(null);
+ return chosen;
+ }
+
+ private record Pick(Environment environment, boolean starter, Environment retired) {
+ }
+
+ private static Pick pick(List<Environment> pool, String endpointKey, String accountId, String revisionId,
+ int perEnvironment, Integer maxEnvironments) {
+ Environment open = pool.stream()
+ .filter(each -> each.isAlive() && each.revisionId.equals(revisionId)
+ && each.inFlight.get() < perEnvironment)
+ .min(Comparator.comparingInt((Environment each) -> each.inFlight.get())
+ .thenComparingLong(each -> each.lastUsed.get()))
+ .orElse(null);
+ if (open != null) {
+ open.reserve();
+ return new Pick(open, false, null);
+ }
+ Environment retired = null;
+ if (maxEnvironments != null && pool.size() >= maxEnvironments) {
+ retired = pool.stream()
+ .filter(each -> !each.revisionId.equals(revisionId) && isIdle(each, 0))
+ .min(Comparator.comparingLong(each -> each.lastUsed.get()))
+ .orElseThrow(Throttled::new);
+ pool.remove(retired);
+ }
+ Environment started = new Environment(endpointKey, revisionId, accountId);
+ pool.add(started);
+ started.reserve();
+ return new Pick(started, true, retired);
+ }
+
+ private static boolean isIdle(Environment environment, long idleNanos) {
+ return environment.isAlive() && environment.isReady() && environment.inFlight.get() == 0
+ && System.nanoTime() - environment.lastUsed.get() >= idleNanos;
+ }
+
+ private void stopRetired(Environment environment) {
+ try {
+ LOG.infov("Removing the Lambda Web environment {0} of revision {1} to make room",
+ environment.containerId, environment.revisionId);
+ discard(environment);
+ } catch (RuntimeException e) {
+ LOG.warnv(e, "Could not remove the Lambda Web environment {0}", environment.containerId);
+ }
+ }
+
+ private static void awaitReady(Environment environment) {
+ try {
+ environment.ready.get();
+ } catch (ExecutionException e) {
+ if (e.getCause() instanceof RuntimeException runtime) {
+ throw runtime;
}
- Environment started = start(key, accountId, region, fn, revision);
- started.reserve();
- environments.put(key, started);
- return started;
+ throw new IllegalStateException(e.getCause());
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new IllegalStateException(e);
}
}
+ private void removeFromPool(Environment environment) {
+ pools.computeIfPresent(environment.poolKey, (key, pool) -> {
+ pool.remove(environment);
+ return pool.isEmpty() ? null : pool;
+ });
+ }
+
/** Stops an environment, with the one second AWS gives the process after SIGTERM. */
public void discard(Environment environment) {
+ removeFromPool(environment);
if (!environment.alive.compareAndSet(true, false)) {
- environments.remove(environment.key, environment);
return;
}
- environments.remove(environment.key, environment);
+ if (environment.containerId == null) {
+ return;
+ }
try {
exitCodeIfStopped(environment.containerId).ifPresent(exitCode ->
LOG.infov("The Lambda Web environment {0} had exited with status {1}: {2}",
@@ -227,6 +331,10 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
} catch (RuntimeException e) {
LOG.debugv(e, "Could not inspect the Lambda Web environment {0}", environment.containerId);
}
+ stopContainer(environment);
+ }
+
+ private void stopContainer(Environment environment) {
try {
lifecycleManager.stopAndRemove(environment.containerId, null, 1);
} finally {
@@ -252,7 +360,18 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
@Override
public void stopManagedContainers() {
- new ArrayList<>(environments.values()).forEach(this::discard);
+ allEnvironments().forEach(this::discard);
+ }
+
+ private List<Environment> allEnvironments() {
+ List<Environment> all = new ArrayList<>();
+ for (String key : pools.keySet()) {
+ pools.computeIfPresent(key, (ignored, pool) -> {
+ all.addAll(pool);
+ return pool;
+ });
+ }
+ return all;
}
@PreDestroy
@@ -260,30 +379,30 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
reaper.shutdownNow();
}
- private Object lockFor(String key) {
- return locks.computeIfAbsent(key, ignored -> new Object());
- }
-
private void reapIdle() {
long idleNanos = TimeUnit.SECONDS.toNanos(config.services().lambda().containerIdleTimeoutSeconds());
- for (Environment environment : new ArrayList<>(environments.values())) {
- try {
- synchronized (lockFor(environment.key)) {
- if (environment.inFlight.get() != 0
- || System.nanoTime() - environment.lastUsed.get() <= idleNanos) {
- continue;
- }
- LOG.infov("Removing the idle Lambda Web environment {0}", environment.containerId);
- discard(environment);
+ for (Environment environment : allEnvironments()) {
+ AtomicBoolean idle = new AtomicBoolean();
+ pools.computeIfPresent(environment.poolKey, (key, pool) -> {
+ if (isIdle(environment, idleNanos) && pool.remove(environment)) {
+ idle.set(true);
}
+ return pool.isEmpty() ? null : pool;
+ });
+ if (!idle.get()) {
+ continue;
+ }
+ try {
+ LOG.infov("Removing the idle Lambda Web environment {0}", environment.containerId);
+ discard(environment);
} catch (RuntimeException e) {
LOG.warnv(e, "Could not remove the idle Lambda Web environment {0}", environment.containerId);
}
}
}
- private Environment start(String key, String accountId, String region, WebFunction fn,
- WebFunctionRevision revision) {
+ private void start(Environment environment, String region, WebFunction fn, WebFunctionRevision revision) {
+ String accountId = environment.accountId;
String image = ecrRegistryManager.rewriteImageUri(imageResolver.resolve(revision.buildConfig.runtime));
String shortId = UUID.randomUUID().toString().replace("-", "").substring(0, 8);
String name = ContainerStorageHelper.prefixedDockerName(config, "floci-lambda-web",
@@ -320,14 +439,15 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
ContainerInfo info = lifecycleManager.startCreated(containerId, spec);
EndpointInfo endpoint = info.getEndpoint(APP_PORT);
awaitListening(containerId);
- String logStream = LOG_STREAM_DATE.format(Instant.now()) + "/" + fn.functionName + "[" + revision.revisionId
- + "]" + UUID.randomUUID().toString().replace("-", "");
- Environment environment = new Environment(key, accountId, containerId, endpoint.host(), endpoint.port(),
- logStream, roleCredentials);
+ environment.containerId = containerId;
+ environment.host = endpoint.host();
+ environment.port = endpoint.port();
+ environment.logStream = LOG_STREAM_DATE.format(Instant.now()) + "/" + fn.functionName + "["
+ + revision.revisionId + "]" + UUID.randomUUID().toString().replace("-", "");
+ environment.roleCredentials = roleCredentials.orElse(null);
watchExit(environment);
LOG.infov("Started the Lambda Web environment {0} of {1} revision {2} at {3}:{4}",
containerId, fn.functionName, revision.revisionId, endpoint.host(), endpoint.port());
- return environment;
} catch (RuntimeException e) {
lifecycleManager.stopAndRemove(containerId, null, 1);
roleCredentials.ifPresent(creds -> executionRoleCredentials.unregister(accountId, creds.accessKeyId()));
@@ -351,8 +471,10 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
}
private void forgetCredentials(Environment environment) {
- environment.roleCredentials.ifPresent(creds ->
- executionRoleCredentials.unregister(environment.accountId, creds.accessKeyId()));
+ SessionCreds credentials = environment.roleCredentials;
+ if (credentials != null) {
+ executionRoleCredentials.unregister(environment.accountId, credentials.accessKeyId());
+ }
}
/** The code goes in by {@code docker cp}, so it works when Floci itself runs in a container. */
@@ -559,7 +681,7 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
return;
}
environment.exitedAt.compareAndSet(null, finishedAt(environment.containerId).orElse(Instant.now()));
- environments.remove(environment.key, environment);
+ removeFromPool(environment);
reaper.execute(() -> {
try {
LOG.infov("The Lambda Web environment {0} exited with status {1}: {2}",
diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebErrorPages.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebErrorPages.java
index d62add6c65..942f91ee53 100644
--- a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebErrorPages.java
+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebErrorPages.java
@@ -27,6 +27,11 @@ final class LambdaWebErrorPages {
APPLICATION_ERROR);
static final Page INIT_CODE_ERROR = new Page(502, "Bad Gateway", "Application.InitCodeError", APPLICATION_ERROR);
static final Page RUNTIME_EXIT = new Page(502, "Bad Gateway", "Runtime.Exit", APPLICATION_ERROR);
+ private static final String RATE_EXCEEDED = "Request rate exceeded. Please try again later.";
+ static final Page ENDPOINT_MAX_ENV_THROTTLE = new Page(429, "Too Many Requests",
+ "Platform.EndpointMaxEnvThrottle", RATE_EXCEEDED);
+ static final Page ENDPOINT_REQUEST_THROTTLE = new Page(429, "Too Many Requests",
+ "Platform.EndpointRequestThrottle", RATE_EXCEEDED);
/** The edge in front of the platform refuses a method itself, with its own page and header casing. */
static final String MALFORMED_REQUEST_TYPE = "Platform.MalformedRequest";
diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebRateLimiter.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebRateLimiter.java
new file mode 100644
index 0000000000..7b0133febb
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebRateLimiter.java
@@ -0,0 +1,68 @@
+package io.github.hectorvent.floci.services.lambdaweb;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.LongSupplier;
+
+/**
+ * The {@code rateLimit} of each endpoint: a bucket of {@code rateLimit} requests that refills at
+ * {@code rateLimit} per second, so a brief burst above the rate passes and a sustained one does not.
+ */
+final class LambdaWebRateLimiter {
+
+ private static final long SECOND = TimeUnit.SECONDS.toNanos(1);
+
+ private record Bucket(int rate, double tokens, long at) {
+ }
+
+ private final ConcurrentHashMap<String, Bucket> buckets = new ConcurrentHashMap<>();
+ private final LongSupplier nanoClock;
+ private final AtomicLong sweptAt;
+
+ LambdaWebRateLimiter() {
+ this(System::nanoTime);
+ }
+
+ LambdaWebRateLimiter(LongSupplier nanoClock) {
+ this.nanoClock = nanoClock;
+ this.sweptAt = new AtomicLong(nanoClock.getAsLong());
+ }
+
+ /** Whether one more request to {@code endpointKey} fits under {@code rate} requests per second. */
+ boolean tryAcquire(String endpointKey, int rate) {
+ sweep();
+ AtomicBoolean admitted = new AtomicBoolean();
+ buckets.compute(endpointKey, (key, bucket) -> {
+ double tokens;
+ long now = nanoClock.getAsLong();
+ if (bucket == null || bucket.rate() != rate) {
+ tokens = rate;
+ } else {
+ now = Math.max(now, bucket.at());
+ double refill = rate * (double) (now - bucket.at()) / SECOND;
+ tokens = Math.min(rate, bucket.tokens() + refill);
+ }
+ if (tokens >= 1) {
+ admitted.set(true);
+ tokens -= 1;
+ }
+ return new Bucket(rate, tokens, now);
+ });
+ return admitted.get();
+ }
+
+ int size() {
+ return buckets.size();
+ }
+
+ /** A bucket untouched for a second is full again, the same as no bucket. */
+ private void sweep() {
+ long now = nanoClock.getAsLong();
+ long last = sweptAt.get();
+ if (now - last >= SECOND && sweptAt.compareAndSet(last, now)) {
+ buckets.values().removeIf(bucket -> now - bucket.at() >= SECOND);
+ }
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java
index beedfb9251..6c700263a0 100644
--- a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java
@@ -18,7 +18,17 @@ import java.io.InputStream;
import java.io.OutputStream;
import java.net.Socket;
import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CyclicBarrier;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
@@ -50,6 +60,7 @@ class LambdaWebDataPlaneDockerIntegrationTest {
const [host, port] = process.env.AWS_LAMBDA_HTTP_ENDPOINT.split(':');
http.createServer((req, res) => {
if (req.url.startsWith('/slow')) { setTimeout(() => res.end('late'), 8000); return; }
+ if (req.url.startsWith('/wait')) { setTimeout(() => res.end('waited'), 3000); return; }
if (req.url.startsWith('/stream')) {
res.writeHead(200, {'content-type': 'text/plain'});
let i = 0;
@@ -181,7 +192,97 @@ class LambdaWebDataPlaneDockerIntegrationTest {
assertNotEquals(logStream, after.header("lambda-web-logstream"), "a new environment serves the next request");
}
+ @Test
+ @Timeout(240)
+ void environmentsScaleUpToMaxEnvironmentsAndTheRateLimitThrottles() throws Exception {
+ String domain = createFunction(30, 1);
+ given().contentType("application/json").body("{\"scalingConfig\": {\"maxEnvironments\": 2}}")
+ .when().patch(BASE + "/web-functions/" + functionName + "/endpoints/dev")
+ .then().statusCode(202);
+
+ // One request per environment: two run, the rest are refused, as recorded on AWS.
+ List<String> responses = parallel(5, () -> raw("GET /wait HTTP/1.1\r\nHost: " + domain
+ + "\r\nConnection: close\r\n\r\n"));
+ long served = responses.stream().filter(response -> response.startsWith("HTTP/1.1 200 ")).count();
+ List<String> throttled = responses.stream().filter(response -> response.startsWith("HTTP/1.1 429 ")).toList();
+ assertEquals(2, served, String.join("\n----\n", responses));
+ assertEquals(3, throttled.size(), String.join("\n----\n", responses));
+ for (String response : throttled) {
+ assertTrue(response.contains("\r\nlambda-web-error-type: Platform.EndpointMaxEnvThrottle\r\n"), response);
+ assertTrue(response.contains("<p>Request rate exceeded. Please try again later.</p>"), response);
+ }
+
+ // Requests one after another rotate over the two environments.
+ Set<String> logStreams = new HashSet<>();
+ for (int i = 0; i < 4; i++) {
+ logStreams.add(given().header("Host", domain).when().get("/status/200")
+ .then().statusCode(200).extract().header("lambda-web-logstream"));
+ }
+ assertEquals(2, logStreams.size(), logStreams.toString());
+
+ // A new revision takes the place of an idle environment of the old one.
+ String revisionId = given().contentType("application/json")
+ .body("""
+ {"buildConfig": {"codeConfig": {"s3Object": {"bucket": "%s", "key": "%s.zip"}},
+ "runtimeConfig": {"runtime": "nodejs24.x"}},
+ "serviceConfig": {"executionRoleArn": "%s", "timeoutSeconds": 30}}
+ """.formatted(BUCKET, functionName, ROLE))
+ .when().post(BASE + "/web-functions/" + functionName + "/revisions")
+ .then().statusCode(202)
+ .extract().path("revisionId");
+ String moved = given().header("Host", domain).when().get("/status/200")
+ .then().statusCode(200).extract().header("lambda-web-logstream");
+ assertTrue(moved.contains("[" + revisionId + "]"), moved);
+ logStreams.add(moved);
+
+ // A recreated endpoint shares no environment or rate with the deleted one.
+ String endpoints = BASE + "/web-functions/" + functionName + "/endpoints";
+ given().when().delete(endpoints + "/dev").then().statusCode(204);
+ String recreated = given().contentType("application/json")
+ .body("{\"endpointName\": \"dev\", \"endpointType\": \"HomeRegion\","
+ + " \"authType\": \"ApplicationManaged\", \"autoDeploymentMode\": \"LatestRevision\","
+ + " \"throttleConfig\": {\"rateLimit\": 100}}")
+ .when().put(endpoints)
+ .then().statusCode(202)
+ .extract().path("domainName");
+ String fresh = given().header("Host", recreated).when().get("/status/200")
+ .then().statusCode(200).extract().header("lambda-web-logstream");
+ assertFalse(logStreams.contains(fresh), fresh + " " + logStreams);
+
+ // Requests that all arrive at once are only admitted up to the rate.
+ int count = 200;
+ CyclicBarrier together = new CyclicBarrier(count);
+ List<String> burst = parallel(count, () -> raw("GET /status/200 HTTP/1.1\r\nHost: " + recreated
+ + "\r\nConnection: close\r\n\r\n", together));
+ List<String> limited = burst.stream().filter(response -> response.startsWith("HTTP/1.1 429 ")).toList();
+ assertFalse(limited.isEmpty(), "a burst well past the rate is throttled");
+ for (String response : limited) {
+ assertTrue(response.contains("\r\nlambda-web-error-type: Platform.EndpointRequestThrottle\r\n"), response);
+ }
+ }
+
+ private static List<String> parallel(int count, Callable<String> call) throws Exception {
+ ExecutorService pool = Executors.newFixedThreadPool(count);
+ try {
+ List<Future<String>> futures = new ArrayList<>();
+ for (int i = 0; i < count; i++) {
+ futures.add(pool.submit(call));
+ }
+ List<String> results = new ArrayList<>();
+ for (Future<String> future : futures) {
+ results.add(future.get(60, TimeUnit.SECONDS));
+ }
+ return results;
+ } finally {
+ pool.shutdownNow();
+ }
+ }
+
private String createFunction(int timeoutSeconds) throws IOException {
+ return createFunction(timeoutSeconds, 64);
+ }
+
+ private String createFunction(int timeoutSeconds, int maxConcurrencyPerEnvironment) throws IOException {
functionName = "dp-docker-" + Long.toString(System.nanoTime(), 36);
s3Service.createBucket(BUCKET, "us-east-1");
s3Service.putObject(BUCKET, functionName + ".zip", zip(APP), "application/zip", Map.of());
@@ -192,19 +293,29 @@ class LambdaWebDataPlaneDockerIntegrationTest {
"buildConfig": {"codeConfig": {"s3Object": {"bucket": "%s", "key": "%s.zip"}},
"runtimeConfig": {"runtime": "nodejs24.x"}},
"serviceConfig": {"executionRoleArn": "%s", "timeoutSeconds": %d,
+ "maxConcurrencyPerEnvironment": %d,
"environmentVariables": {"CUSTOM": "yes"}}},
"endpointConfig": {"endpointName": "dev", "endpointType": "HomeRegion",
"authType": "ApplicationManaged", "autoDeploymentMode": "LatestRevision"}}
- """.formatted(functionName, BUCKET, functionName, ROLE, timeoutSeconds))
+ """.formatted(functionName, BUCKET, functionName, ROLE, timeoutSeconds,
+ maxConcurrencyPerEnvironment))
.when().put(BASE + "/web-functions")
.then().statusCode(202)
.extract().path("endpoint.domainName");
}
/** Sends one raw HTTP/1.1 request with {@code Connection: close} and reads until the server closes. */
- private static String raw(String request) throws IOException {
+ private static String raw(String request) throws Exception {
+ return raw(request, null);
+ }
+
+ /** As {@link #raw(String)}, sent once every party of {@code together} has connected. */
+ private static String raw(String request, CyclicBarrier together) throws Exception {
try (Socket socket = new Socket("localhost", RestAssured.port)) {
socket.setSoTimeout(30_000);
+ if (together != null) {
+ together.await(30, TimeUnit.SECONDS);
+ }
OutputStream out = socket.getOutputStream();
out.write(request.getBytes(StandardCharsets.US_ASCII));
out.flush();
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneIntegrationTest.java
index dc1682927a..1a6af4b93e 100644
--- a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneIntegrationTest.java
@@ -51,7 +51,7 @@ class LambdaWebDataPlaneIntegrationTest {
}
@Test
- void aDisabledEndpointAnswersBeforeAnyEnvironmentStarts() {
+ void aDisabledEndpointAnswersBeforeAnyEnvironmentStarts() throws IOException {
String functionName = "dp-disabled-" + Long.toString(System.nanoTime(), 36);
String domain = createFunction(functionName);
given().contentType("application/json").body("{\"throttleConfig\": {\"rateLimit\": 0}}")
@@ -65,9 +65,40 @@ class LambdaWebDataPlaneIntegrationTest {
.body(equalTo(LambdaWebErrorPages.html(LambdaWebErrorPages.ENDPOINT_DISABLED,
response.header("lambda-web-request-id"))));
+ // The page keeps the connection, as on AWS. A body sent after it is skipped and the next request is served.
+ try (Socket socket = new Socket("localhost", RestAssured.port)) {
+ socket.setSoTimeout(10_000);
+ OutputStream out = socket.getOutputStream();
+ InputStream in = socket.getInputStream();
+ out.write(("POST /upload HTTP/1.1\r\nHost: " + domain + "\r\nContent-Length: 5\r\n\r\n")
+ .getBytes(StandardCharsets.US_ASCII));
+ out.flush();
+ String first = readUntil(in, "</html>");
+ assertTrue(first.startsWith("HTTP/1.1 503 "), first);
+ assertTrue(first.contains("\r\nconnection: keep-alive\r\n"), first);
+ out.write(("hello" + "GET /next HTTP/1.1\r\nHost: " + domain + "\r\nConnection: close\r\n\r\n")
+ .getBytes(StandardCharsets.US_ASCII));
+ out.flush();
+ String second = new String(in.readAllBytes(), StandardCharsets.UTF_8);
+ assertTrue(second.startsWith("HTTP/1.1 503 "), second);
+ assertTrue(second.contains("\r\nlambda-web-error-type: Platform.EndpointDisabled\r\n"), second);
+ }
+
given().when().delete(BASE + "/web-functions/" + functionName).then().statusCode(204);
}
+ private static String readUntil(InputStream in, String end) throws IOException {
+ StringBuilder read = new StringBuilder();
+ while (!read.toString().endsWith(end)) {
+ int next = in.read();
+ if (next < 0) {
+ break;
+ }
+ read.append((char) next);
+ }
+ return read.toString();
+ }
+
@Test
void httpOneDotZeroIsRefused() throws IOException {
String response = raw("GET / HTTP/1.0\r\nHost: " + UNKNOWN_HOST + "\r\n\r\n");
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebErrorPagesTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebErrorPagesTest.java
index 5599ac5351..6a697d4916 100644
--- a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebErrorPagesTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebErrorPagesTest.java
@@ -23,6 +23,8 @@ class LambdaWebErrorPagesTest {
assertEquals(228, length(LambdaWebErrorPages.TIMEOUT_LISTEN_ERROR));
assertEquals(228, length(LambdaWebErrorPages.INIT_CODE_ERROR));
assertEquals(228, length(LambdaWebErrorPages.RUNTIME_EXIT));
+ assertEquals(237, length(LambdaWebErrorPages.ENDPOINT_MAX_ENV_THROTTLE));
+ assertEquals(237, length(LambdaWebErrorPages.ENDPOINT_REQUEST_THROTTLE));
assertEquals(180, LambdaWebErrorPages.MALFORMED_REQUEST_HTML.getBytes(StandardCharsets.UTF_8).length);
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebRateLimiterTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebRateLimiterTest.java
new file mode 100644
index 0000000000..f075cd2fea
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebRateLimiterTest.java
@@ -0,0 +1,69 @@
+package io.github.hectorvent.floci.services.lambdaweb;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+class LambdaWebRateLimiterTest {
+
+ private final AtomicLong now = new AtomicLong();
+ private final LambdaWebRateLimiter limiter = new LambdaWebRateLimiter(now::get);
+
+ @Test
+ void aBurstUpToTheRatePassesAndTheNextRequestIsThrottled() {
+ assertEquals(100, admitted("ep", 100, 150));
+ }
+
+ @Test
+ void theBucketRefillsAtTheRatePerSecond() {
+ assertEquals(100, admitted("ep", 100, 100));
+ assertFalse(limiter.tryAcquire("ep", 100));
+ now.addAndGet(TimeUnit.MILLISECONDS.toNanos(500));
+ assertEquals(50, admitted("ep", 100, 80));
+ now.addAndGet(TimeUnit.SECONDS.toNanos(10));
+ assertEquals(100, admitted("ep", 100, 150), "the bucket holds at most one second of requests");
+ }
+
+ @Test
+ void endpointsHaveTheirOwnBucketsAndANewRateStartsAFullOne() {
+ assertEquals(100, admitted("a", 100, 100));
+ assertTrue(limiter.tryAcquire("b", 100));
+ assertFalse(limiter.tryAcquire("a", 100));
+ assertEquals(200, admitted("a", 200, 250));
+ }
+
+ @Test
+ void aClockReadOutOfOrderTakesNoTokensBack() {
+ now.set(1_000);
+ assertTrue(limiter.tryAcquire("ep", 2));
+ now.set(0);
+ assertTrue(limiter.tryAcquire("ep", 2));
+ assertFalse(limiter.tryAcquire("ep", 2));
+ }
+
+ @Test
+ void bucketsThatRefilledAreDropped() {
+ assertEquals(100, admitted("a", 100, 100));
+ assertTrue(limiter.tryAcquire("b", 100));
+ assertEquals(2, limiter.size());
+ now.addAndGet(TimeUnit.SECONDS.toNanos(1));
+ assertTrue(limiter.tryAcquire("c", 100));
+ assertEquals(1, limiter.size());
+ assertEquals(100, admitted("a", 100, 150));
+ }
+
+ private int admitted(String endpoint, int rate, int requests) {
+ int admitted = 0;
+ for (int i = 0; i < requests; i++) {
+ if (limiter.tryAcquire(endpoint, rate)) {
+ admitted++;
+ }
+ }
+ return admitted;
+ }
+}