Light, fluffy, and always free - The AWS Local Emulator alternative
fix(docker): make the Docker client pool size configurable, default 1024 (#4276)
Goooooooooody76f8c0934134
diff --git a/docs/configuration/advanced/application-yml.md b/docs/configuration/advanced/application-yml.mdindex 3ba8ad1d2b..b33006a258 100644--- a/docs/configuration/advanced/application-yml.md+++ b/docs/configuration/advanced/application-yml.md@@ -109,6 +109,7 @@ floci:log-max-size: "10m" # Max size per container log file before rotationlog-max-file: "3" # Number of rotated log files to retaindocker-host: unix:///var/run/docker.sock # Docker daemon socket (shared by Lambda, RDS, ElastiCache)+ max-connections: 1024 # Docker client connection pool; each live Lambda container holds 2docker-config-path: "" # Path to dir containing Docker's config.json (e.g. /root/.docker)registry-credentials: [] # Per-registry explicit credentials for private registriesdiff --git a/docs/configuration/docker.md b/docs/configuration/docker.mdindex bd1f9d8e8b..fd886665ad 100644--- a/docs/configuration/docker.md+++ b/docs/configuration/docker.md@@ -48,6 +48,20 @@ services:Use this only when the socket's permissions require root. Without the option, Floci retains its unprivileged default. The setting applies to the container entrypoint, not to `floci.docker` configuration.+## Connection Pool++Every Docker call Floci makes goes through one shared client with a bounded connection pool. Some of those connections stay open for as long as a container runs: each Lambda container holds two (its followed log stream and the watcher that notices its runtime exiting) plus one per Lambda extension, and other container-backed services hold one for their log stream. When those long-lived connections fill the pool, every other Docker call (create, start, stop, remove) waits for one to free up, and Floci stops making progress.++The default of 1024 connections covers the 500 concurrent Lambda containers the default [Runtime API port range](../services/lambda.md#configuration) allows. Raise it if you widen that range or run many other containers at the same time:++```yaml+floci:+ docker:+ max-connections: 1024+```++Environment variable: `FLOCI_DOCKER_MAX_CONNECTIONS`+## Private Registry AuthenticationAny service that pulls a container image from a private registry (Lambda image functions, custom OpenSearch images, private Postgres images, etc.) needs Docker credentials. Two approaches are supported and can be combined.@@ -257,6 +271,7 @@ What each setting does and why it is needed:|---|---|---|| `FLOCI_DOCKER_DOCKER_HOST` | `unix:///var/run/docker.sock` | Docker daemon socket || `FLOCI_DOCKER_DOCKER_CONFIG_PATH` | _(unset)_ | Path to directory containing Docker's `config.json` |+| `FLOCI_DOCKER_MAX_CONNECTIONS` | `1024` | Connection pool size for Floci's Docker client. Each live Lambda container holds two connections, so about half this many can run at once || `FLOCI_DOCKER_REGISTRY_CREDENTIALS_0__SERVER` | _(unset)_ | Registry hostname for credential entry 0 || `FLOCI_DOCKER_REGISTRY_CREDENTIALS_0__USERNAME` | _(unset)_ | Username for credential entry 0 || `FLOCI_DOCKER_REGISTRY_CREDENTIALS_0__PASSWORD` | _(unset)_ | Password for credential entry 0 |diff --git a/docs/configuration/environment-variables.md b/docs/configuration/environment-variables.mdindex 3ae89de950..28c3edc8d1 100644--- a/docs/configuration/environment-variables.md+++ b/docs/configuration/environment-variables.md@@ -110,6 +110,7 @@ This foundation release provides opt-in security-group filtering for EC2 Docker| `FLOCI_NETWORK_SECURITY_GROUP_ENFORCEMENT_HELPER_IMAGE` | `floci/network-helper:local` | Linux helper image containing nftables; Floci builds the default image locally when missing || `FLOCI_DOCKER_DOCKER_HOST` | `unix:///var/run/docker.sock` | Docker daemon socket path or TCP address || `FLOCI_DOCKER_DOCKER_CONFIG_PATH` | _(none)_ | Path to a directory containing Docker's `config.json` for registry auth |+| `FLOCI_DOCKER_MAX_CONNECTIONS` | `1024` | Connection pool size for Floci's Docker client. Each live Lambda container holds two connections (log stream and exit watcher), so about half this many can run at once. See [Docker Configuration](docker.md#connection-pool) || `FLOCI_DOCKER_IMAGE_REGISTRY_BASE` | _(none)_ | Optional registry/repository base for every Docker image Floci launches. When set, `postgres:16-alpine` resolves as `<base>/postgres:16-alpine` and `public.ecr.aws/docker/library/ubuntu:24.04` resolves as `<base>/public.ecr.aws/docker/library/ubuntu:24.04` || `FLOCI_DOCKER_LOG_MAX_SIZE` | `10m` | Log rotation max size for spawned containers (e.g. `10m`, `1g`) || `FLOCI_DOCKER_LOG_MAX_FILE` | `3` | Number of rotated log files to keep for spawned containers |diff --git a/docs/services/lambda.md b/docs/services/lambda.mdindex fd1e04aad1..2b64a6b5be 100644--- a/docs/services/lambda.md+++ b/docs/services/lambda.md@@ -351,7 +351,7 @@ These AWS Lambda operations have no handler in Floci. Calls will return `404` or| `FLOCI_SERVICES_LAMBDA_DEFAULT_MEMORY_MB` | `128` | Default function memory (MB) || `FLOCI_SERVICES_LAMBDA_DEFAULT_TIMEOUT_SECONDS` | `3` | Default function timeout (seconds) || `FLOCI_SERVICES_LAMBDA_RUNTIME_API_BASE_PORT` | `12000` | First port in the Lambda Runtime API range |-| `FLOCI_SERVICES_LAMBDA_RUNTIME_API_MAX_PORT` | `12499` | Last port in the Lambda Runtime API range. One port is held per running container, so the range width caps concurrent executions |+| `FLOCI_SERVICES_LAMBDA_RUNTIME_API_MAX_PORT` | `12499` | Last port in the Lambda Runtime API range. One port is held per running container, so the range width caps concurrent executions. Each running container also holds two Docker connections, so a wider range needs a matching `FLOCI_DOCKER_MAX_CONNECTIONS` || `FLOCI_SERVICES_LAMBDA_CODE_PATH` | `./data/lambda-code` | Directory where Lambda ZIP files are stored || `FLOCI_SERVICES_LAMBDA_POLL_INTERVAL_MS` | `1000` | Event-source mapping poll interval (milliseconds) || `FLOCI_SERVICES_LAMBDA_CONTAINER_IDLE_TIMEOUT_SECONDS` | `300` | Idle container shutdown timeout (seconds) |diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.javaindex ca9fe16755..18ae70b2af 100644--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java@@ -3324,6 +3324,22 @@ public interface EmulatorConfig {@WithDefault("unix:///var/run/docker.sock")String dockerHost();+ /**+ * Size of the connection pool behind the shared Docker client. Every Docker call leases a+ * connection from it, and some hold one for as long as a container runs: a Lambda+ * container holds two (its followed log stream and its exit watcher) plus one per+ * extension, and other container-backed services hold one for their log stream. Once+ * those fill the pool, create, start, stop and remove wait for a free connection,+ * including the calls that would release one, so the emulator stalls.+ *+ * <p>1024 rather than the former hard-coded 100, which capped live Lambda containers at+ * about 50: a tenth of the 500 that the runtime API port range+ * ({@link LambdaServiceConfig#runtimeApiBasePort}) exists to allow. Connections are+ * opened on demand, so a small stack never approaches the limit.+ */+ @WithDefault("1024")+ int maxConnections();+/*** Optional namespace inserted into Floci-managed child container and volume names.* Useful when multiple Floci processes share one Docker daemon.diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/DockerClientProducer.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/DockerClientProducer.javaindex f65e8ff81a..6f812d3f36 100644--- a/src/main/java/io/github/hectorvent/floci/core/common/docker/DockerClientProducer.java+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/DockerClientProducer.java@@ -396,12 +396,25 @@ public class DockerClientProducer {return System.getProperty("os.name", "").toLowerCase().contains("win");}+ /**+ * Rejects a pool size below 1 with a message naming the setting, instead of leaving the+ * connection pool to reject it with a generic error.+ */+ static int validateMaxConnections(int maxConnections) {+ if (maxConnections < 1) {+ throw new IllegalArgumentException("floci.docker.max-connections (FLOCI_DOCKER_MAX_CONNECTIONS) "+ + "must be at least 1, got: " + maxConnections);+ }+ return maxConnections;+ }+@Produces@ApplicationScopedpublic DockerClient dockerClient() {+ int maxConnections = validateMaxConnections(config.docker().maxConnections());ResolvedDockerConnection connection = resolveDockerConnection(config.docker(), System::getenv);String dockerHost = connection.host();- LOG.infov("Creating DockerClient for host: {0}", dockerHost);+ LOG.infov("Creating DockerClient for host: {0} (max connections: {1})", dockerHost, maxConnections);// createDefaultConfigBuilder() reads DOCKER_HOST directly from System.getenv() and passes// it to withDockerHost(), which calls URI.create() immediately. If DOCKER_HOST is set@@ -430,7 +443,7 @@ public class DockerClientProducer {ApacheDockerHttpClient httpClient = new ApacheDockerHttpClient.Builder().dockerHost(clientConfig.getDockerHost())- .maxConnections(100)+ .maxConnections(maxConnections).connectionTimeout(Duration.ofSeconds(30)).responseTimeout(Duration.ofMinutes(5)).build();diff --git a/src/main/resources/application.yml b/src/main/resources/application.ymlindex 7f495bce67..7f9a55abbd 100644--- a/src/main/resources/application.yml+++ b/src/main/resources/application.yml@@ -289,6 +289,13 @@ floci:log-max-size: "10m"log-max-file: "3"docker-host: unix:///var/run/docker.sock+ # FLOCI_DOCKER_MAX_CONNECTIONS: connection pool size for every Docker call Floci makes. Each+ # live Lambda container holds 2 connections for its whole lifetime (log follow + exit watcher),+ # so the warm-container ceiling is about max-connections / 2; past it, create/start/stop/remove+ # wait for a free connection and the emulator stalls. 1024 covers the 500 containers the+ # runtime API port range allows. Keep in step with the @WithDefault on+ # EmulatorConfig.DockerConfig.maxConnections: a value set here WINS over the annotation.+ max-connections: 1024resource-namespace: "" # FLOCI_DOCKER_RESOURCE_NAMESPACE; scopes child container/volume namesdocker-config-path: ""# extra-labels: # Extra Docker labels applied to every Floci-created container and volumediff --git a/src/test/java/io/github/hectorvent/floci/config/ApplicationDefaultsTest.java b/src/test/java/io/github/hectorvent/floci/config/ApplicationDefaultsTest.javaindex 5fb5fe0f9b..e5d5a8fc1b 100644--- a/src/test/java/io/github/hectorvent/floci/config/ApplicationDefaultsTest.java+++ b/src/test/java/io/github/hectorvent/floci/config/ApplicationDefaultsTest.java@@ -110,4 +110,26 @@ class ApplicationDefaultsTest {assertEquals("true", fallback.value(),"a stock Floci should refuse a PostgreSQL IAM token generated for another endpoint, as RDS does");}++ @Test+ void productionConfigSizesTheDockerPoolForTheLambdaRuntimeApiPortRange() throws IOException, NoSuchMethodException {+ JsonNode floci = new YAMLMapper().readTree(Path.of("src/main/resources/application.yml").toFile())+ .path("floci");+ int maxConnections = floci.path("docker").path("max-connections").asInt();+ JsonNode lambda = floci.path("services").path("lambda");+ int runtimeApiPorts = lambda.path("runtime-api-max-port").asInt()+ - lambda.path("runtime-api-base-port").asInt() + 1;++ assertTrue(maxConnections >= 2 * runtimeApiPorts,+ "each live Lambda container holds two Docker connections (log follow and exit watcher), so "+ + "max-connections " + maxConnections + " should cover the " + runtimeApiPorts+ + " containers the runtime API port range allows");++ WithDefault fallback = EmulatorConfig.DockerConfig.class+ .getMethod("maxConnections")+ .getAnnotation(WithDefault.class);+ assertNotNull(fallback, "maxConnections should declare a fallback default");+ assertEquals(String.valueOf(maxConnections), fallback.value(),+ "the @WithDefault must match application.yml, which wins over it");+ }}diff --git a/src/test/java/io/github/hectorvent/floci/core/common/docker/DockerClientProducerTest.java b/src/test/java/io/github/hectorvent/floci/core/common/docker/DockerClientProducerTest.javaindex 9d7e55af07..f3cf1dea0c 100644--- a/src/test/java/io/github/hectorvent/floci/core/common/docker/DockerClientProducerTest.java+++ b/src/test/java/io/github/hectorvent/floci/core/common/docker/DockerClientProducerTest.java@@ -1,11 +1,19 @@package io.github.hectorvent.floci.core.common.docker;+import com.github.dockerjava.api.DockerClient;+import com.github.dockerjava.api.async.ResultCallback;+import com.github.dockerjava.api.model.Frame;+import io.github.hectorvent.floci.config.EmulatorConfig;import org.junit.jupiter.api.Test;import org.junit.jupiter.api.io.TempDir;import org.junit.jupiter.params.ParameterizedTest;import org.junit.jupiter.params.provider.MethodSource;+import org.junit.jupiter.params.provider.ValueSource;import java.io.IOException;+import java.net.InetAddress;+import java.net.ServerSocket;+import java.net.Socket;import java.net.URI;import java.nio.file.Files;import java.nio.file.Path;@@ -13,9 +21,16 @@ import java.nio.file.Paths;import java.security.MessageDigest;import java.security.NoSuchAlgorithmException;import java.util.Optional;+import java.util.concurrent.BlockingQueue;+import java.util.concurrent.LinkedBlockingQueue;+import java.util.concurrent.TimeUnit;import java.util.stream.Stream;+import static org.hamcrest.MatcherAssert.assertThat;+import static org.hamcrest.Matchers.containsString;import static org.junit.jupiter.api.Assertions.*;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.when;/*** Bug condition exploration test for Docker host scheme normalization.@@ -610,4 +625,74 @@ class DockerClientProducerTest {"unix:///var/run/docker.sock", null, false, tempDir, null),"An incomplete TLS material directory should fail loudly rather than connect insecurely");}++ @ParameterizedTest+ @ValueSource(ints = {0, -1})+ void validateMaxConnections_belowOne_throwsNamingTheSetting(int maxConnections) {+ IllegalArgumentException e = assertThrows(IllegalArgumentException.class,+ () -> DockerClientProducer.validateMaxConnections(maxConnections));++ assertThat(e.getMessage(), containsString("floci.docker.max-connections"));+ }++ @Test+ void validateMaxConnections_positive_returnsItUnchanged() {+ assertEquals(1, DockerClientProducer.validateMaxConnections(1));+ assertEquals(1024, DockerClientProducer.validateMaxConnections(1024));+ }++ /**+ * The pool size comes from {@code floci.docker.max-connections}. A followed log stream holds+ * its connection until the container exits, and this fake daemon never answers, so with a+ * pool of two only two of three streams may reach it; the third waits for a free connection.+ */+ @Test+ void dockerClient_opensAtMostMaxConnectionsToTheDaemon() throws Exception {+ try (ServerSocket daemon = new ServerSocket(0, 50, InetAddress.getLoopbackAddress())) {+ BlockingQueue<Socket> accepted = new LinkedBlockingQueue<>();+ Thread acceptor = new Thread(() -> {+ try {+ while (!daemon.isClosed()) {+ accepted.add(daemon.accept());+ }+ } catch (IOException expected) {+ // Closing the server socket at the end of the test ends the accept loop.+ }+ });+ acceptor.setDaemon(true);+ acceptor.start();++ EmulatorConfig config = mock(EmulatorConfig.class);+ EmulatorConfig.DockerConfig docker = mock(EmulatorConfig.DockerConfig.class);+ when(config.docker()).thenReturn(docker);+ when(docker.dockerHost()).thenReturn("tcp://127.0.0.1:" + daemon.getLocalPort());+ when(docker.dockerConfigPath()).thenReturn(Optional.empty());+ when(docker.maxConnections()).thenReturn(2);++ DockerClient client = new DockerClientProducer(config).dockerClient();+ try {+ for (int i = 0; i < 3; i++) {+ client.logContainerCmd("container-" + i)+ .withStdOut(true)+ .withFollowStream(true)+ .exec(new ResultCallback.Adapter<Frame>() {+ @Override+ public void onError(Throwable ignored) {+ // Expected once the client is closed under the pending streams.+ }+ });+ }++ assertNotNull(accepted.poll(10, TimeUnit.SECONDS), "first stream should connect");+ assertNotNull(accepted.poll(10, TimeUnit.SECONDS), "second stream should connect");+ assertNull(accepted.poll(1, TimeUnit.SECONDS),+ "third stream should wait for a pooled connection instead of opening one");+ } finally {+ client.close();+ for (Socket socket : accepted) {+ socket.close();+ }+ }+ }+ }}