Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

fix(docker): retry transient daemon I/O at the transport seam and split streaming pools (#4614)

Sam Fakhreddine153153755962
diff --git a/docs/configuration/advanced/application-yml.md b/docs/configuration/advanced/application-yml.md
index 4d84797df1..70c25af738 100644
--- a/docs/configuration/advanced/application-yml.md
+++ b/docs/configuration/advanced/application-yml.md
@@ -120,7 +120,8 @@ floci:
log-max-size: "10m" # Max size per container log file before rotation
log-max-file: "3" # Number of rotated log files to retain
docker-host: unix:///var/run/docker.sock # Docker daemon socket (shared by Lambda, RDS, ElastiCache)
- max-connections: 1024 # Docker client connection pool; each live Lambda container holds 2
+ max-connections: 1024 # Docker client connection pool; each live Lambda container holds 1
+ streaming-max-connections: 512 # Separate pool for container log-follow streams
docker-config-path: "" # Path to dir containing Docker's config.json (e.g. /root/.docker)
registry-credentials: [] # Per-registry explicit credentials for private registries
diff --git a/docs/configuration/docker.md b/docs/configuration/docker.md
index c79c7015d2..04c3742848 100644
--- a/docs/configuration/docker.md
+++ b/docs/configuration/docker.md
@@ -50,7 +50,7 @@ Use this only when the socket's permissions require root. Without the option, Fl
## Connection Pool
-Every Docker call Floci makes goes through one shared client with a bounded connection pool. Some of those connections stay open for as long as a container runs: each Lambda container holds two (its followed log stream and the watcher that notices its runtime exiting) plus one per Lambda extension, and other container-backed services hold one for their log stream. When those long-lived connections fill the pool, every other Docker call (create, start, stop, remove) waits for one to free up, and Floci stops making progress.
+Every Docker call Floci makes goes through one shared client with a bounded connection pool. Some of those connections stay open for as long as a container runs: each Lambda container holds one (the watcher that notices its runtime exiting) plus one per Lambda extension. Log-follow streams use a second pool, described below. When those long-lived connections fill the pool, every other Docker call (create, start, stop, remove) waits for one to free up, and Floci stops making progress.
The default of 1024 connections covers the 500 concurrent Lambda containers the default [Runtime API port range](../services/lambda.md#configuration) allows. Raise it if you widen that range or run many other containers at the same time:
@@ -62,6 +62,16 @@ floci:
Environment variable: `FLOCI_DOCKER_MAX_CONNECTIONS`
+Container log-follow streams use a second, separate pool so they can never starve create, start, stop and remove calls. Its size defaults to 512 and follows the same rule (at least 1). Exec-output and container-wait streams (CodeBuild phases, Lambda extensions and exit watchers, EKS audit followers) still use the main pool and count against `max-connections`:
+
+```yaml
+floci:
+ docker:
+ streaming-max-connections: 512
+```
+
+Environment variable: `FLOCI_DOCKER_STREAMING_MAX_CONNECTIONS`
+
## Private Registry Authentication
Any service that pulls a container image from a private registry (Lambda image functions, custom OpenSearch images, private Postgres images, etc.) needs Docker credentials. Two approaches are supported and can be combined.
@@ -286,6 +296,12 @@ What each setting does and why it is needed:
auto-detection entirely. See the [Lambda docs](../services/lambda.md#configuration)
for details.
+## Transient I/O retry
+
+All of Floci's short-lived docker calls (create/start/inspect/remove container, volume management, image operations) travel one shared daemon socket, and under fan-out load the daemon occasionally drops a connection mid-call with `java.io.IOException: Broken pipe`. Floci retries these transient failures centrally, at the docker transport layer, up to 6 attempts with a capped exponential backoff (500ms base, 8s cap); so every call site is covered without per-call configuration, and a genuine daemon rejection (a 4xx, a name conflict) still surfaces immediately.
+
+A request is only replayed when doing so cannot change semantics: requests carrying a one-shot upload stream (e.g. copying an archive into a container), bidirectional attach streams, and `exec` requests (which would re-run the command) are never retried. Log-follow connections use a separate transport that does not retry at all. Bodyless actions whose replay changes the outcome (unnamed container create, network create, container restart, network connect and disconnect) are not retried either. A tar-writer failure on the Floci host (for example an unreadable source file) is reported at once instead of retried.
+
## Full Reference
| Environment variable | Default | Description |
diff --git a/docs/configuration/environment-variables.md b/docs/configuration/environment-variables.md
index fe6975f581..aa2ebdf4b5 100644
--- a/docs/configuration/environment-variables.md
+++ b/docs/configuration/environment-variables.md
@@ -111,7 +111,8 @@ This foundation release provides opt-in security-group filtering for EC2 Docker
| `FLOCI_NETWORK_SECURITY_GROUP_ENFORCEMENT_HELPER_IMAGE` | `floci/network-helper:local` | Linux helper image containing nftables; Floci builds the default image locally when missing |
| `FLOCI_DOCKER_DOCKER_HOST` | `unix:///var/run/docker.sock` | Docker daemon socket path or TCP address |
| `FLOCI_DOCKER_DOCKER_CONFIG_PATH` | _(none)_ | Path to a directory containing Docker's `config.json` for registry auth |
-| `FLOCI_DOCKER_MAX_CONNECTIONS` | `1024` | Connection pool size for Floci's Docker client. Each live Lambda container holds two connections (log stream and exit watcher), so about half this many can run at once. See [Docker Configuration](docker.md#connection-pool) |
+| `FLOCI_DOCKER_MAX_CONNECTIONS` | `1024` | Connection pool size for Floci's Docker client. Each live Lambda container holds one connection (its exit watcher) plus one per extension; log streams use the streaming pool. See [Docker Configuration](docker.md#connection-pool) |
+| `FLOCI_DOCKER_STREAMING_MAX_CONNECTIONS` | `512` | Connection pool size for container log-follow streams, kept apart from the main pool so they never starve create, start, stop and remove. See [Docker Configuration](docker.md#connection-pool) |
| `FLOCI_DOCKER_IMAGE_REGISTRY_BASE` | _(none)_ | Optional registry/repository base for every Docker image Floci launches. When set, `postgres:16-alpine` resolves as `<base>/postgres:16-alpine` and `public.ecr.aws/docker/library/ubuntu:24.04` resolves as `<base>/public.ecr.aws/docker/library/ubuntu:24.04` |
| `FLOCI_DOCKER_LOG_MAX_SIZE` | `10m` | Log rotation max size for spawned containers (e.g. `10m`, `1g`) |
| `FLOCI_DOCKER_LOG_MAX_FILE` | `3` | Number of rotated log files to keep for spawned containers |
diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
index 7b749edb00..cb02f7ddd2 100644
--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
@@ -3462,10 +3462,10 @@ public interface EmulatorConfig {
/**
* Size of the connection pool behind the shared Docker client. Every Docker call leases a
* connection from it, and some hold one for as long as a container runs: a Lambda
- * container holds two (its followed log stream and its exit watcher) plus one per
- * extension, and other container-backed services hold one for their log stream. Once
- * those fill the pool, create, start, stop and remove wait for a free connection,
- * including the calls that would release one, so the emulator stalls.
+ * container holds one (its exit watcher) plus one per extension. Log-follow streams
+ * live in the separate streaming pool. Once the long-lived connections fill this pool,
+ * create, start, stop and remove wait for a free connection, including the calls that
+ * would release one, so the emulator stalls.
*
* <p>1024 rather than the former hard-coded 100, which capped live Lambda containers at
* about 50: a tenth of the 500 that the runtime API port range
@@ -3475,6 +3475,17 @@ public interface EmulatorConfig {
@WithDefault("1024")
int maxConnections();
+ /**
+ * Connection pool size for the {@code @StreamingDocker} DockerClient, sized for the
+ * long-lived container log-follow streams that occupy a slot for a container's entire
+ * lifetime. Exec-output and container-wait streams stay on the main pool. Total live
+ * containers is unbounded across distinct functions (each {@code WarmPool} caps only
+ * per-function), so this pool is sized generously rather than tied to any single
+ * function's warm-pool cap.
+ */
+ @WithDefault("512")
+ int streamingMaxConnections();
+
/**
* Optional namespace inserted into Floci-managed child container and volume names.
* Useful when multiple Floci processes share one Docker daemon.
diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java
index 427936304d..84ea66b552 100644
--- a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java
+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java
@@ -7,8 +7,10 @@ import com.github.dockerjava.api.command.CreateContainerResponse;
import com.github.dockerjava.api.command.InspectContainerResponse;
import com.github.dockerjava.api.command.InspectVolumeResponse;
import com.github.dockerjava.api.command.ListVolumesResponse;
+import com.github.dockerjava.api.exception.ConflictException;
import com.github.dockerjava.api.exception.DockerException;
import com.github.dockerjava.api.exception.NotFoundException;
+import com.github.dockerjava.api.exception.NotModifiedException;
import com.github.dockerjava.api.model.Bind;
import com.github.dockerjava.api.model.Capability;
import com.github.dockerjava.api.model.Container;
@@ -42,6 +44,7 @@ import java.util.Map;
import java.util.Optional;
import java.util.OptionalInt;
import java.util.Set;
+import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
@@ -56,6 +59,13 @@ public class ContainerLifecycleManager {
private static final Logger LOG = Logger.getLogger(ContainerLifecycleManager.class);
+ /**
+ * Label carrying a fresh id per {@code create()} invocation, used to tell this call's own
+ * replayed create apart from a genuinely concurrent caller's, see
+ * {@link #createWithConflictRecovery}.
+ */
+ static final String CREATE_ATTEMPT_LABEL = "floci_create_attempt_id";
+
// Matches crun/runc's "join keyctl '<id>': Disk quota exceeded" error, which the OCI runtime
// reports for the kernel session-keyring quota (kernel.keys.maxkeys), not actual disk space.
// Under high concurrent container counts (podman machine's Fedora CoreOS default is 200 keys
@@ -217,14 +227,57 @@ public class ContainerLifecycleManager {
.toArray(ExposedPort[]::new);
createCmd.withExposedPorts(exposed);
}
- createCmd.withLabels(mergedLabels(spec.labels()));
+ // A fresh id per create() call, not per spec: two calls can carry identical image, name,
+ // and labels (a second, genuinely concurrent caller racing on the same fixed name), and
+ // matchesSpec must be able to tell that apart from a replay of THIS call's own create
+ // hitting a lost-response conflict, see createWithConflictRecovery.
+ String createAttemptId = UUID.randomUUID().toString();
+ Map<String, String> labels = mergedLabels(spec.labels());
+ labels.put(CREATE_ATTEMPT_LABEL, createAttemptId);
+ createCmd.withLabels(labels);
- CreateContainerResponse response = createCmd.exec();
- String containerId = response.getId();
+ String containerId = createWithConflictRecovery(createCmd, spec, createAttemptId);
LOG.infov("Created container {0} (name={1}, not yet started)", containerId, spec.name());
return containerId;
}
+ /**
+ * Executes the create, recovering the container when a replay of it lost the race with its own
+ * earlier attempt.
+ *
+ * <p>{@link RetryingDockerHttpClient} replays {@code POST /containers/create}, it has a
+ * {@code bodyBytes} body, no hijacked input, and no {@code /exec} in its path, so it is
+ * replayable by that decorator's rules. That replay is what makes this recovery necessary: a
+ * create is not idempotent, so if the daemon accepts a named create but the response is lost to
+ * a transient socket drop, the replay hits a 409 name conflict instead of the transient error
+ * the retry exists to absorb. {@code DockerRetry} correctly classifies that 409 as
+ * non-transient and rethrows it immediately, so without this it surfaces as a spurious build
+ * failure, and leaks the container the first attempt actually created, untracked.
+ *
+ * <p>Recovery only fires for a named spec, and only adopts a candidate whose image, labels, and
+ * this exact call's {@link #CREATE_ATTEMPT_LABEL} all match. Image and labels alone cannot tell
+ * this call's own lost-response replay apart from a second, genuinely concurrent
+ * {@code create()} racing on the same fixed name/image/labels, which would otherwise be adopted
+ * as if it were this invocation's container.
+ */
+ private String createWithConflictRecovery(CreateContainerCmd createCmd, ContainerSpec spec,
+ String createAttemptId) {
+ try {
+ return createCmd.exec().getId();
+ } catch (ConflictException e) {
+ if (spec.name() != null) {
+ Optional<Container> existing = findByName(spec.name());
+ if (existing.isPresent() && matchesSpec(existing.get(), spec, createAttemptId)) {
+ LOG.infov("Create for {0} hit a name conflict; an earlier attempt's response "
+ + "was lost but the container exists, adopting {1}",
+ spec.name(), existing.get().getId());
+ return existing.get().getId();
+ }
+ }
+ throw e;
+ }
+ }
+
/**
* Applies endpoint IPAM while the container is still in CREATED state. docker-java does not
* expose Docker's {@code NetworkingConfig} on create, but it does expose the equivalent
@@ -295,6 +348,27 @@ public class ContainerLifecycleManager {
}
}
+ /**
+ * Confirms a name-conflicting container is the one THIS create() call produced, not a stale or
+ * differently-owned container, and not one from a genuinely concurrent create() sharing the same
+ * fixed name, image, and spec labels. It must carry every label this call would have applied,
+ * including a {@link #CREATE_ATTEMPT_LABEL} equal to this call's {@code createAttemptId}: the
+ * one part of the label set a different concurrent call could never coincidentally reproduce,
+ * and on its own sufficient proof of ownership.
+ *
+ * <p>Deliberately does NOT compare the image. The list API reports {@code getImage()} as an
+ * image ID/digest rather than the tag once that tag has been re-pulled, retagged, or removed, so
+ * a tag comparison would reject this call's own container and leak it untracked, the failure
+ * this recovery exists to prevent, arising precisely under the daemon churn that makes a lost
+ * create response likely. The attempt-id match already excludes every foreign container.
+ */
+ private boolean matchesSpec(Container existing, ContainerSpec spec, String createAttemptId) {
+ Map<String, String> expectedLabels = mergedLabels(spec.labels());
+ expectedLabels.put(CREATE_ATTEMPT_LABEL, createAttemptId);
+ Map<String, String> actualLabels = existing.getLabels();
+ return actualLabels != null && actualLabels.entrySet().containsAll(expectedLabels.entrySet());
+ }
+
/**
* Starts a previously created container and resolves its endpoints.
*
@@ -341,14 +415,29 @@ public class ContainerLifecycleManager {
}
/**
- * Starts a container, translating crun/runc's kernel-keyring-quota error into a message that
- * names the actual cause and a fix, instead of the misleading "Disk quota exceeded" the OCI
- * runtime reports. Package-private so tests can verify a given call site routes through this
- * translation rather than a raw {@code dockerClient.startContainerCmd(...)} call.
+ * Starts {@code containerId}, treating "already running" as success and translating
+ * crun/runc's kernel-keyring-quota error into a message that names the actual cause and a fix,
+ * instead of the misleading "Disk quota exceeded" the OCI runtime reports. Transient socket
+ * blips are retried below this call, at the transport seam
+ * ({@link RetryingDockerHttpClient}); no retry may live here, where it would compound backoff
+ * on the transport's already-exhausted budget.
+ *
+ * <p>The 304 handling is what makes the transport's replay safe: when the daemon honoured a
+ * start whose response was lost to a broken pipe, the replayed start meets HTTP 304, a
+ * successful response at transport level, which docker-java converts to
+ * {@link NotModifiedException} above it. That reports the outcome we wanted and is swallowed.
+ * Letting it escape would turn a recovered blip into a hard launch failure, the exact bug
+ * the retry exists to remove.
+ *
+ * <p>Package-private so tests can verify a given call site routes through this translation
+ * rather than a raw {@code dockerClient.startContainerCmd(...)} call.
*/
void startContainer(String containerId) {
try {
dockerClient.startContainerCmd(containerId).exec();
+ } catch (NotModifiedException alreadyRunning) {
+ LOG.debugv("Container {0} was already running (304), treating start as done",
+ containerId);
} catch (DockerException e) {
String message = e.getMessage();
if (message != null && KEYRING_QUOTA_PATTERN.matcher(message).find()) {
@@ -477,6 +566,13 @@ public class ContainerLifecycleManager {
* {@code floci_emulator=floci-aws} (plus {@code floci_namespace} when configured) so both
* {@code docker volume prune --filter label=floci=true} (all emulators) and
* {@code --filter label=floci_emulator=floci-aws} (this emulator only) work.
+ *
+ * <p>Transient socket blips on both daemon calls here, the existence check and the create,
+ * are retried at the transport seam ({@link RetryingDockerHttpClient}). The existence guard
+ * is what keeps the transport's replay safe: {@code POST /volumes/create} with the same name
+ * is itself idempotent, and when the daemon created the volume but the response was lost to
+ * a broken pipe, the replayed create finds it already there while later calls see it exists
+ * and do nothing, the volume analogue of start treating an HTTP 304 as success.
*/
public void ensureVolume(String volumeName) {
createVolumeIfAbsent(volumeName);
diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLogStreamer.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLogStreamer.java
index b57cd4f64c..16a64d8958 100644
--- a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLogStreamer.java
+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLogStreamer.java
@@ -47,8 +47,12 @@ public class ContainerLogStreamer {
private final DockerClient dockerClient;
private final CloudWatchLogsService cloudWatchLogsService;
+ // Log-follow attaches hold their connection open for the whole container lifetime, so
+ // this class is the primary consumer of the streaming pool: sharing the control-plane
+ // pool is what starved short create/start/stop calls during an LZA Logging fan-out
+ // (see DockerClientProducer / StreamingDocker).
@Inject
- public ContainerLogStreamer(DockerClient dockerClient, CloudWatchLogsService cloudWatchLogsService) {
+ public ContainerLogStreamer(@StreamingDocker DockerClient dockerClient, CloudWatchLogsService cloudWatchLogsService) {
this.dockerClient = dockerClient;
this.cloudWatchLogsService = cloudWatchLogsService;
}
diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/DockerClientProducer.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/DockerClientProducer.java
index 6f812d3f36..02f0be1132 100644
--- a/src/main/java/io/github/hectorvent/floci/core/common/docker/DockerClientProducer.java
+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/DockerClientProducer.java
@@ -10,6 +10,7 @@ import com.github.dockerjava.core.DockerClientImpl;
import com.github.dockerjava.core.SSLConfig;
import com.github.dockerjava.core.util.CertificateUtils;
import com.github.dockerjava.httpclient5.ApacheDockerHttpClient;
+import com.github.dockerjava.transport.DockerHttpClient;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.enterprise.inject.Produces;
import jakarta.inject.Inject;
@@ -33,7 +34,7 @@ import java.util.Optional;
import java.util.function.Function;
/**
- * CDI producer for the DockerClient singleton bean.
+ * CDI producer for the DockerClient beans.
*/
@ApplicationScoped
public class DockerClientProducer {
@@ -408,13 +409,55 @@ public class DockerClientProducer {
return maxConnections;
}
+ /** Same check as {@link #validateMaxConnections(int)}, for the streaming pool's setting. */
+ static int validateStreamingMaxConnections(int streamingMaxConnections) {
+ if (streamingMaxConnections < 1) {
+ throw new IllegalArgumentException("floci.docker.streaming-max-connections "
+ + "(FLOCI_DOCKER_STREAMING_MAX_CONNECTIONS) must be at least 1, got: "
+ + streamingMaxConnections);
+ }
+ return streamingMaxConnections;
+ }
+
+ /**
+ * Control-plane DockerClient: create/start/stop/remove/copyArchive and every other
+ * short-lived call in Floci. Kept on its own connection pool, separate from the
+ * {@link StreamingDocker} bean, so long-lived log-follow streams can never occupy every
+ * lease in this pool and starve these short calls until httpclient5's connection-request
+ * timeout fires. Exec-output and container-wait streams (CodeBuild phases, Lambda
+ * extensions and exit watchers, EKS audit followers) still use this pool.
+ */
@Produces
@ApplicationScoped
public DockerClient dockerClient() {
- int maxConnections = validateMaxConnections(config.docker().maxConnections());
+ return buildDockerClient(buildClientConfig(),
+ validateMaxConnections(config.docker().maxConnections()), "control-plane");
+ }
+
+ /**
+ * Streaming DockerClient: container log-follow ({@link ContainerLogStreamer}). Each
+ * stream occupies a connection pool slot for its entire lifetime, so sharing the
+ * control-plane pool meant a fan-out of many concurrent streams exhausted it and blocked
+ * control-plane calls. Sized separately because {@code WarmPool}'s cap is per-function,
+ * so total live streams across distinct functions is unbounded.
+ */
+ @Produces
+ @StreamingDocker
+ @ApplicationScoped
+ public DockerClient streamingDockerClient() {
+ return buildDockerClient(buildClientConfig(),
+ validateStreamingMaxConnections(config.docker().streamingMaxConnections()), "streaming");
+ }
+
+ /**
+ * Builds the {@link DefaultDockerClientConfig} shared by both DockerClient beans (host and
+ * Docker-context resolution, optional Docker config path, and any TLS material the active
+ * context carries). The two beans differ only in connection pool size, never in how they
+ * reach the daemon.
+ */
+ private DefaultDockerClientConfig buildClientConfig() {
ResolvedDockerConnection connection = resolveDockerConnection(config.docker(), System::getenv);
String dockerHost = connection.host();
- LOG.infov("Creating DockerClient for host: {0} (max connections: {1})", dockerHost, maxConnections);
// createDefaultConfigBuilder() reads DOCKER_HOST directly from System.getenv() and passes
// it to withDockerHost(), which calls URI.create() immediately. If DOCKER_HOST is set
@@ -439,7 +482,12 @@ public class DockerClientProducer {
builder.withDockerCertPath(path.toString());
}
});
- DefaultDockerClientConfig clientConfig = builder.build();
+ return builder.build();
+ }
+
+ private DockerClient buildDockerClient(DefaultDockerClientConfig clientConfig, int maxConnections, String role) {
+ LOG.infov("Creating {0} DockerClient pool (maxConnections={1}) for host: {2}",
+ role, maxConnections, clientConfig.getDockerHost());
ApacheDockerHttpClient httpClient = new ApacheDockerHttpClient.Builder()
.dockerHost(clientConfig.getDockerHost())
@@ -448,7 +496,22 @@ public class DockerClientProducer {
.responseTimeout(Duration.ofMinutes(5))
.build();
- return DockerClientImpl.getInstance(clientConfig, httpClient);
+ return DockerClientImpl.getInstance(clientConfig, wrapForRole(httpClient, role));
+ }
+
+ /**
+ * Wraps the control-plane transport in {@link RetryingDockerHttpClient} so every short-lived
+ * docker call survives a transient socket drop; the streaming transport stays unwrapped,
+ * its requests (log-follow, exec output) hold hijacked or long-lived streams that must never
+ * be replayed, and its exec-start calls are the one request retrying could turn into a
+ * re-run command.
+ */
+ static DockerHttpClient wrapForRole(DockerHttpClient transport, String role) {
+ if (!"control-plane".equals(role)) {
+ return transport;
+ }
+ LOG.info("Wrapping control-plane docker transport in transient-I/O retry");
+ return new RetryingDockerHttpClient(transport);
}
/**
diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/DockerRetry.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/DockerRetry.java
new file mode 100644
index 0000000000..fdb53754b9
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/DockerRetry.java
@@ -0,0 +1,178 @@
+package io.github.hectorvent.floci.core.common.docker;
+
+import org.jboss.logging.Logger;
+
+import java.io.IOException;
+import java.io.InterruptedIOException;
+import java.net.SocketTimeoutException;
+import java.util.Locale;
+
+/**
+ * Retry policy for docker daemon calls. Every docker command travels over the single shared
+ * socket; when many builds drive the daemon at once (e.g. an LZA Bootstrap stage fanning out
+ * ~15 CodeBuild actions) it intermittently drops a connection mid-call with a transient I/O
+ * error, {@code Broken pipe} or {@code Connection reset}, that docker-java surfaces wrapped
+ * in a {@link RuntimeException}. These clear on a retry; a genuine daemon rejection (a 4xx, a
+ * name conflict) does not and must surface immediately. Callers wrap idempotent docker calls
+ * (create container, copy archive in) so a transient blip does not fail the whole build.
+ */
+public final class DockerRetry {
+
+ private static final Logger LOG = Logger.getLogger(DockerRetry.class);
+
+ /**
+ * Upper bound on any single backoff sleep. Socket saturation from a fan-out build wave can
+ * persist for several seconds, so the backoff grows exponentially toward this cap to ride it
+ * out rather than exhausting a handful of sub-second retries against a still-busy socket.
+ */
+ public static final long BACKOFF_CAP_MS = 8000L;
+
+ private DockerRetry() {
+ }
+
+ /**
+ * Exponential backoff for attempt {@code n} (1-based): {@code base * 2^(n-1)}, clamped to
+ * {@code cap}. A zero base yields zero (no sleep); the shift is bounded so it cannot overflow.
+ */
+ public static long backoffDelay(int attempt, long baseMillis, long capMillis) {
+ if (baseMillis <= 0) {
+ return 0L;
+ }
+ int shift = Math.min(attempt - 1, 30);
+ long delay = baseMillis << shift;
+ if (delay < 0 || delay > capMillis) {
+ return capMillis;
+ }
+ return delay;
+ }
+
+ /**
+ * A failure raised on this host while preparing a docker call's payload (for example a tar
+ * writer that cannot read its source file), as opposed to one raised by the daemon
+ * transport. It is never retried, even when its cause is an {@link IOException}, because
+ * rebuilding the payload would fail the same way.
+ */
+ public static final class LocalFailure extends RuntimeException {
+ public LocalFailure(String message, Throwable cause) {
+ super(message, cause);
+ }
+ }
+
+ /** A docker call that returns a value and may throw. */
+ @FunctionalInterface
+ public interface DockerCall<T> {
+ T run() throws Exception;
+ }
+
+ /** A docker call that returns nothing and may throw. */
+ @FunctionalInterface
+ public interface DockerOp {
+ void run() throws Exception;
+ }
+
+ /**
+ * True when {@code t} (or any cause in its chain) is a transient docker I/O failure worth
+ * retrying, an {@link IOException} such as {@code Broken pipe} / {@code Connection reset},
+ * possibly wrapped by docker-java in a {@link RuntimeException}.
+ */
+ public static boolean isTransientIo(Throwable t) {
+ // InterruptedIOException covers both a genuine thread interruption (masking it would
+ // delay shutdown) and httpclient5's ConnectionRequestTimeoutException, the pool has no
+ // free connection lease after waiting the full request timeout, meaning the pool is
+ // exhausted rather than a socket blip. Retrying either just re-enters another wait and
+ // adds pressure to an already-starved pool, so both must be excluded from the general
+ // "any IOException is transient" rule below, checked over the WHOLE chain first, since
+ // an outer wrapper IOException would otherwise short-circuit the walk before it ever
+ // reaches an inner InterruptedIOException cause. java.net.SocketTimeoutException is ALSO
+ // an InterruptedIOException subtype but means something different (the daemon just took
+ // too long to answer, not that the pool is exhausted or the thread was interrupted) and
+ // must stay retryable, so it's excluded from this exclusion.
+ for (Throwable c = t; c != null; c = c.getCause()) {
+ if (c instanceof InterruptedIOException && !(c instanceof SocketTimeoutException)) {
+ return false;
+ }
+ if (c instanceof LocalFailure) {
+ return false;
+ }
+ if (c.getCause() == c) {
+ break;
+ }
+ }
+ for (Throwable c = t; c != null; c = c.getCause()) {
+ if (c instanceof IOException) {
+ return true;
+ }
+ String m = c.getMessage();
+ if (m != null) {
+ // Locale.ROOT: the default locale lowercases per the host's language, and a
+ // Turkish default turns "I" into a dotless i, so "Broken pipe" stops matching on
+ // exactly the hosts least likely to be tested. Matching both messages
+ // case-insensitively also covers daemons that vary the capitalisation.
+ String lower = m.toLowerCase(Locale.ROOT);
+ if (lower.contains("broken pipe") || lower.contains("connection reset")) {
+ return true;
+ }
+ }
+ if (c.getCause() == c) {
+ break;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Runs {@code call}, retrying up to {@code maxAttempts} times on a transient docker I/O
+ * error ({@link #isTransientIo}) with a capped exponential backoff between attempts
+ * ({@link #backoffDelay}), and returns its value. A non-transient failure is rethrown
+ * immediately. A {@link RuntimeException} is rethrown as-is; a checked exception is wrapped.
+ */
+ public static <T> T call(int maxAttempts, long backoffMillis, DockerCall<T> call) {
+ if (maxAttempts < 1) {
+ // Zero or negative would run the call zero times and fall out of the loop returning
+ // nothing, turning a misconfiguration into a silent no-op rather than a failure.
+ throw new IllegalArgumentException("maxAttempts must be >= 1, got " + maxAttempts);
+ }
+ int attempt = 0;
+ while (true) {
+ attempt++;
+ try {
+ return call.run();
+ } catch (Exception e) {
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ throw asUnchecked(e);
+ }
+ if (attempt >= maxAttempts || !isTransientIo(e)) {
+ throw asUnchecked(e);
+ }
+ LOG.warnv("Transient docker I/O error on attempt {0}/{1}, retrying: {2}",
+ attempt, maxAttempts, e.getMessage());
+ sleep(backoffDelay(attempt, backoffMillis, BACKOFF_CAP_MS));
+ }
+ }
+ }
+
+ /** Void variant of {@link #call}. */
+ public static void run(int maxAttempts, long backoffMillis, DockerOp op) {
+ call(maxAttempts, backoffMillis, () -> {
+ op.run();
+ return null;
+ });
+ }
+
+ private static RuntimeException asUnchecked(Exception e) {
+ return e instanceof RuntimeException re ? re : new RuntimeException(e);
+ }
+
+ private static void sleep(long millis) {
+ if (millis <= 0) {
+ return;
+ }
+ try {
+ Thread.sleep(millis);
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException("Interrupted while backing off before a docker retry", ie);
+ }
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/RetryingDockerHttpClient.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/RetryingDockerHttpClient.java
new file mode 100644
index 0000000000..ddd9c793c4
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/RetryingDockerHttpClient.java
@@ -0,0 +1,183 @@
+package io.github.hectorvent.floci.core.common.docker;
+
+import com.github.dockerjava.transport.DockerHttpClient;
+
+import java.io.IOException;
+import java.util.LinkedHashMap;
+import java.util.Map;
+
+/**
+ * Decorates a {@link DockerHttpClient} with the {@link DockerRetry} policy so every docker call
+ * travelling over the shared daemon socket survives a transient mid-call drop ({@code Broken
+ * pipe}, {@code Connection reset}) without each call site having to wrap itself.
+ *
+ * <p>The daemon drops connections under fan-out load (an LZA Bootstrap stage firing ~15 CodeBuild
+ * actions, a Prepare stage at six live containers), and docker-java surfaces the blip as
+ * {@code RuntimeException(IOException)} from inside {@code execute()}, which is exactly where
+ * this decorator sits. Guarding call sites one at a time was fixed three separate times
+ * ({@code createContainerCmd}, {@code startContainerCmd}, {@code ensureVolume}) while ~60 other
+ * docker call sites stayed bare; one guard at the transport boundary covers them all, including
+ * every call added later.
+ *
+ * <p>A request is only retried when replaying it cannot change semantics:
+ * <ul>
+ * <li>Its body, if any, must live in {@code bodyBytes()}, a plain {@code byte[]} the transport
+ * re-reads from scratch on every attempt. A request whose body is only available as a
+ * one-shot {@code InputStream} (the tar upload of {@code copyArchiveToContainerCmd}) may
+ * have been partially consumed by the failed attempt and is never replayed.</li>
+ * <li>It must not carry {@code hijackedInput()} (bidirectional attach streams).</li>
+ * <li>Its path must not contain {@code /exec}: {@code POST /exec/{id}/start} re-runs a shell
+ * command whose first run may have executed before the socket died, a worse bug than the
+ * one retrying fixes. The {@code contains} spelling (not {@code startsWith}) also covers
+ * exec-create ({@code POST /containers/{id}/exec}), where a replay would merely leak an
+ * unused exec ID; that conservative breadth costs nothing.</li>
+ * <li>It must not be an unnamed container create, a network create, a container restart or a
+ * network connect/disconnect: replaying those after a lost response duplicates a
+ * container, loses a network to a swallowed 409, restarts twice, or hits a spurious
+ * conflict.</li>
+ * </ul>
+ *
+ * <p>Idempotency of what a replay <em>means</em> stays the call site's job: docker answers a
+ * replayed start of an already-running container with HTTP 304 (raised above this transport as
+ * {@code NotModifiedException} and treated as success by the caller), and {@code ensureVolume}
+ * re-checks existence so a replayed create finds the volume the lost response landed.
+ */
+public final class RetryingDockerHttpClient implements DockerHttpClient {
+
+ static final String CONNECTION_HEADER = "Connection";
+ static final String CONNECTION_CLOSE = "close";
+
+ /**
+ * Mirrors the per-call-site policy this decorator replaces: six attempts at 500ms
+ * exponential backoff (capped by {@link DockerRetry#BACKOFF_CAP_MS}) rides out the
+ * several-second socket saturation a fan-out build wave causes.
+ */
+ static final int MAX_ATTEMPTS = 6;
+ static final long BACKOFF_MS = 500L;
+
+ private final DockerHttpClient delegate;
+ private final int maxAttempts;
+ private final long backoffMillis;
+
+ public RetryingDockerHttpClient(DockerHttpClient delegate) {
+ this(delegate, MAX_ATTEMPTS, BACKOFF_MS);
+ }
+
+ RetryingDockerHttpClient(DockerHttpClient delegate, int maxAttempts, long backoffMillis) {
+ this.delegate = delegate;
+ this.maxAttempts = maxAttempts;
+ this.backoffMillis = backoffMillis;
+ }
+
+ @Override
+ public Response execute(Request request) {
+ Request effectiveRequest = withConnectionClose(request);
+ if (!isReplayable(effectiveRequest)) {
+ return delegate.execute(effectiveRequest);
+ }
+ return DockerRetry.call(maxAttempts, backoffMillis, () -> delegate.execute(effectiveRequest));
+ }
+
+ /**
+ * Retires every non-hijacked control-plane connection after its response instead of returning
+ * it to docker-java's pool. The upstream Apache transport disables stale-connection validation,
+ * so a Podman-closed pooled socket can otherwise be leased again and fail on its next write.
+ *
+ * <p>This is independent of retry eligibility: one-shot archive uploads and {@code /exec}
+ * requests must not be replayed, but their successfully completed connections must not be
+ * returned to the pool either. Hijacked requests remain unchanged because docker-java upgrades
+ * those connections and manages their lifetime as streams.
+ */
+ static Request withConnectionClose(Request request) {
+ if (request.hijackedInput() != null) {
+ return request;
+ }
+
+ Map<String, String> headers = new LinkedHashMap<>();
+ request.headers().forEach((name, value) -> {
+ if (!CONNECTION_HEADER.equalsIgnoreCase(name)) {
+ headers.put(name, value);
+ }
+ });
+ headers.put(CONNECTION_HEADER, CONNECTION_CLOSE);
+
+ Request.Builder builder = Request.builder()
+ .method(request.method())
+ .path(request.path())
+ .headers(headers);
+ if (request.bodyBytes() != null) {
+ builder.bodyBytes(request.bodyBytes());
+ } else if (request.body() != null) {
+ builder.body(request.body());
+ }
+ return builder.build();
+ }
+
+ /**
+ * Whether re-executing {@code request} after a transient failure is semantically safe.
+ * {@code bodyBytes()} takes precedence over {@code body()} exactly as it does in the
+ * transport itself: when both are set the transport sends the bytes and never touches the
+ * stream, so the request replays cleanly.
+ */
+ static boolean isReplayable(Request request) {
+ if (request.hijackedInput() != null) {
+ return false;
+ }
+ if (request.path().contains("/exec")) {
+ return false;
+ }
+ if (isNonIdempotentCreate(request) || isNonIdempotentAction(request)) {
+ return false;
+ }
+ return request.bodyBytes() != null || request.body() == null;
+ }
+
+ /**
+ * Bodyless POST actions whose replay after a lost response changes the outcome. A replayed
+ * {@code containers/{id}/restart} restarts the container a second time (extra downtime), and
+ * a replayed {@code networks/{id}/connect} or {@code disconnect} gets a conflict or not-found
+ * the caller does not expect for an operation that in fact succeeded.
+ */
+ private static boolean isNonIdempotentAction(Request request) {
+ if (!"POST".equals(request.method())) {
+ return false;
+ }
+ String path = request.path();
+ int queryStart = path.indexOf('?');
+ String route = queryStart < 0 ? path : path.substring(0, queryStart);
+ return route.endsWith("/restart") || route.endsWith("/connect") || route.endsWith("/disconnect");
+ }
+
+ /**
+ * Creates whose replay after a lost response is harmful. A replayed {@code networks/create}
+ * gets a 409 that callers swallow, silently losing the network. A replayed unnamed
+ * {@code containers/create} makes a second, orphaned container; a named one is safe because
+ * the caller adopts the existing container on a 409.
+ */
+ private static boolean isNonIdempotentCreate(Request request) {
+ if (!"POST".equals(request.method())) {
+ return false;
+ }
+ String path = request.path();
+ int queryStart = path.indexOf('?');
+ String route = queryStart < 0 ? path : path.substring(0, queryStart);
+ if (route.endsWith("/networks/create")) {
+ return true;
+ }
+ if (route.endsWith("/containers/create")) {
+ String query = queryStart < 0 ? "" : path.substring(queryStart + 1);
+ for (String param : query.split("&")) {
+ if (param.startsWith("name=")) {
+ return false;
+ }
+ }
+ return true;
+ }
+ return false;
+ }
+
+ @Override
+ public void close() throws IOException {
+ delegate.close();
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/RetryingTarCopier.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/RetryingTarCopier.java
new file mode 100644
index 0000000000..1f44874586
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/RetryingTarCopier.java
@@ -0,0 +1,203 @@
+package io.github.hectorvent.floci.core.common.docker;
+
+import com.github.dockerjava.api.DockerClient;
+import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
+import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
+import org.jboss.logging.Logger;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.PipedInputStream;
+import java.io.PipedOutputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Retryable tar-copies into containers, shared by every service that pushes content into a
+ * container over the Docker API.
+ *
+ * <p>The transport-level retry ({@link RetryingDockerHttpClient}) must refuse to replay a
+ * one-shot {@code InputStream} body, the streamed tar bytes are gone after the first attempt,
+ * so copies retry here instead, where the whole operation (tar bytes or pipe, streamer thread,
+ * request) is rebuilt per attempt. Tar extract over the same path is idempotent, so a replay is
+ * safe; non-transient failures (daemon rejections) surface after a single attempt, unwrapped.
+ */
+public final class RetryingTarCopier {
+
+ private static final Logger LOG = Logger.getLogger(RetryingTarCopier.class);
+
+ static final int DEFAULT_MAX_ATTEMPTS = 6;
+ static final long DEFAULT_BACKOFF_MS = 500L;
+
+ /**
+ * Buffer for the tar-streaming pipe. The default {@link PipedInputStream} buffer is only 1KB,
+ * which forces a writer/reader thread hand-off (wait/notify) every 1KB. Streaming a ~90MB
+ * node_modules through that ran at ~0.5MB/s (≈3 min per cold start), pure synchronization
+ * thrash, not I/O. A large buffer lets the tar writer stream ahead so throughput is bound by
+ * the Docker daemon, not the pipe.
+ */
+ private static final int TAR_PIPE_BUFFER_BYTES = 16 * 1024 * 1024;
+
+ private RetryingTarCopier() {
+ }
+
+ /** Writes one attempt's tar content; invoked freshly per attempt. */
+ @FunctionalInterface
+ public interface TarWriter {
+ void write(OutputStream out) throws IOException;
+ }
+
+ /** Copies an in-memory payload as a single-entry tar. */
+ public static void copyBytes(DockerClient docker, String containerId, String remotePath,
+ String entryName, byte[] content, int mode) {
+ copyBytes(docker, containerId, remotePath, entryName, content, mode,
+ DEFAULT_MAX_ATTEMPTS, DEFAULT_BACKOFF_MS);
+ }
+
+ public static void copyBytes(DockerClient docker, String containerId, String remotePath,
+ String entryName, byte[] content, int mode,
+ int maxAttempts, long backoffMillis) {
+ byte[] tar = singleFileTar(entryName, content, mode);
+ retry(entryName, containerId, maxAttempts, backoffMillis, () ->
+ docker.copyArchiveToContainerCmd(containerId)
+ .withRemotePath(remotePath)
+ .withTarInputStream(new ByteArrayInputStream(tar))
+ .exec());
+ }
+
+ /** Streams a file from disk as a single-entry tar, without buffering it in memory. */
+ public static void copyFile(DockerClient docker, String containerId, String remotePath,
+ String entryName, Path sourceFile, int mode) {
+ copyFile(docker, containerId, remotePath, entryName, sourceFile, mode,
+ DEFAULT_MAX_ATTEMPTS, DEFAULT_BACKOFF_MS);
+ }
+
+ public static void copyFile(DockerClient docker, String containerId, String remotePath,
+ String entryName, Path sourceFile, int mode,
+ int maxAttempts, long backoffMillis) {
+ copyStreamed(docker, containerId, remotePath, sourceFile.toString(), out -> {
+ try (TarArchiveOutputStream tar = newTarStream(out)) {
+ TarArchiveEntry entry = new TarArchiveEntry(entryName);
+ entry.setSize(Files.size(sourceFile));
+ entry.setMode(mode);
+ tar.putArchiveEntry(entry);
+ try (InputStream fis =Files.newInputStream(sourceFile)) {
+ fis.transferTo(tar);
+ }
+ tar.closeArchiveEntry();
+ }
+ }, maxAttempts, backoffMillis);
+ }
+
+ /**
+ * Streams arbitrary tar content produced by {@code tarWriter} through a fresh pipe and
+ * streamer thread per attempt. {@code label} names the payload in errors and thread names.
+ */
+ public static void copyStreamed(DockerClient docker, String containerId, String remotePath,
+ String label, TarWriter tarWriter) {
+ copyStreamed(docker, containerId, remotePath, label, tarWriter,
+ DEFAULT_MAX_ATTEMPTS, DEFAULT_BACKOFF_MS);
+ }
+
+ public static void copyStreamed(DockerClient docker, String containerId, String remotePath,
+ String label, TarWriter tarWriter,
+ int maxAttempts, long backoffMillis) {
+ retry(label, containerId, maxAttempts, backoffMillis, () -> {
+ AtomicReference<Throwable> writerFailure = new AtomicReference<>();
+ try (PipedOutputStream pos = new PipedOutputStream();
+ PipedInputStream pis = new PipedInputStream(pos, TAR_PIPE_BUFFER_BYTES)) {
+ Thread streamer = new Thread(() -> {
+ try (pos) {
+ tarWriter.write(pos);
+ } catch (Throwable e) {
+ // Recorded and rechecked below instead of just logged: closing the pipe
+ // here without producing output looks like a clean, empty tar to the
+ // reader, the daemon accepts it and exec() below returns normally,
+ // so a writer failure (e.g. the source file went missing) would otherwise
+ // report success while nothing (or a truncated file) was actually copied.
+ writerFailure.set(e);
+ LOG.debugv("Tar streamer for {0} ended early: {1}", label, e.getMessage());
+ }
+ }, "tar-streamer-" + label);
+ streamer.start();
+ docker.copyArchiveToContainerCmd(containerId)
+ .withRemotePath(remotePath)
+ .withTarInputStream(pis)
+ .exec();
+ // Joined before the try-with-resources closes pos/pis on the way out: a real
+ // exec() only returns after consuming the tar to EOF, which the streamer causes
+ // by closing pos once it's done, so this never blocks in production, but closing
+ // pos out from under a still-writing streamer (as a mocked/short-circuited exec()
+ // would) turns its own write into a spurious "Pipe closed" failure.
+ streamer.join();
+ }
+ Throwable failure = writerFailure.get();
+ if (failure != null) {
+ // exec() already returned, so the daemon transport was fine: this is a local
+ // writer failure that rebuilding the copy would only repeat.
+ throw new DockerRetry.LocalFailure("Tar streamer for " + label + " failed: "
+ + failure.getMessage(), failure);
+ }
+ });
+ }
+
+ /** Copies a host path, letting docker-java build the tar; rebuilt per attempt. */
+ public static void copyHostResource(DockerClient docker, String containerId, String remotePath,
+ String hostPath) {
+ copyHostResource(docker, containerId, remotePath, hostPath,
+ DEFAULT_MAX_ATTEMPTS, DEFAULT_BACKOFF_MS);
+ }
+
+ public static void copyHostResource(DockerClient docker, String containerId, String remotePath,
+ String hostPath, int maxAttempts, long backoffMillis) {
+ retry(hostPath, containerId, maxAttempts, backoffMillis, () ->
+ docker.copyArchiveToContainerCmd(containerId)
+ .withHostResource(hostPath)
+ .withRemotePath(remotePath)
+ .exec());
+ }
+
+ /** Single-entry tar bytes, reusable across attempts and callers. */
+ public static byte[] singleFileTar(String entryName, byte[] content, int mode) {
+ try {
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ try (TarArchiveOutputStream tar = newTarStream(out)) {
+ TarArchiveEntry entry = new TarArchiveEntry(entryName);
+ entry.setSize(content.length);
+ entry.setMode(mode);
+ tar.putArchiveEntry(entry);
+ tar.write(content);
+ tar.closeArchiveEntry();
+ }
+ return out.toByteArray();
+ } catch (IOException e) {
+ throw new IllegalStateException("Could not build in-memory tar for " + entryName, e);
+ }
+ }
+
+ /** GNU long names + star big numbers so any path length and file size survive the archive. */
+ public static TarArchiveOutputStream newTarStream(OutputStream out) {
+ TarArchiveOutputStream tar = new TarArchiveOutputStream(out);
+ tar.setLongFileMode(TarArchiveOutputStream.LONGFILE_GNU);
+ tar.setBigNumberMode(TarArchiveOutputStream.BIGNUMBER_STAR);
+ return tar;
+ }
+
+ private interface CopyAttempt {
+ void run() throws Exception;
+ }
+
+ private static void retry(String label, String containerId, int maxAttempts, long backoffMillis,
+ CopyAttempt attempt) {
+ try {
+ DockerRetry.run(maxAttempts, backoffMillis, attempt::run);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to copy " + label + " into container "
+ + containerId + ": " + e.getMessage(), e);
+ }
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/StreamingDocker.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/StreamingDocker.java
new file mode 100644
index 0000000000..9b1edcea49
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/StreamingDocker.java
@@ -0,0 +1,24 @@
+package io.github.hectorvent.floci.core.common.docker;
+
+import jakarta.inject.Qualifier;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * CDI qualifier for the DockerClient dedicated to long-lived streaming operations,
+ * container log-follow ({@link ContainerLogStreamer}). Exec-output and container-wait
+ * streams still use the control-plane client. These calls occupy a connection pool
+ * slot for a container's entire lifetime, so they must not share a pool
+ * with short-lived control-plane calls (create/start/stop/remove/copyArchive): a fan-out
+ * of many long-lived streams would otherwise starve the control-plane calls out of a
+ * lease until httpclient5's connection-request timeout fires. See
+ * {@link DockerClientProducer} for the two beans this qualifier distinguishes.
+ */
+@Qualifier
+@Retention(RetentionPolicy.RUNTIME)
+@Target({ElementType.FIELD, ElementType.PARAMETER, ElementType.METHOD})
+public @interface StreamingDocker {
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2ContainerManager.java b/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2ContainerManager.java
index 25ec90a183..ab2d2fc6fd 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2ContainerManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2ContainerManager.java
@@ -12,6 +12,7 @@ import io.github.hectorvent.floci.core.common.docker.ContainerSpec;
import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;
import io.github.hectorvent.floci.core.common.docker.DockerHostResolver;
import io.github.hectorvent.floci.core.common.docker.PortAllocator;
+import io.github.hectorvent.floci.core.common.docker.RetryingTarCopier;
import io.github.hectorvent.floci.core.common.docker.UserDataPipeline;
import io.github.hectorvent.floci.services.ec2.Ec2InstanceTypeCatalog.CatalogInstanceType;
import io.github.hectorvent.floci.services.ec2.model.Instance;
@@ -32,11 +33,8 @@ import com.github.dockerjava.api.model.MountType;
import jakarta.annotation.PreDestroy;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
-import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
-import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
import org.jboss.logging.Logger;
-import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.Closeable;
import java.io.IOException;
@@ -1494,11 +1492,8 @@ public class Ec2ContainerManager {
// Copy authorized_keys via docker cp
String keyContent = publicKey.trim() + "\n";
- byte[] tar = buildSingleFileTar("authorized_keys", keyContent.getBytes(StandardCharsets.UTF_8), 0600);
- dockerClient.copyArchiveToContainerCmd(containerId)
- .withRemotePath("/root/.ssh")
- .withTarInputStream(new ByteArrayInputStream(tar))
- .exec();
+ RetryingTarCopier.copyBytes(dockerClient, containerId, "/root/.ssh",
+ "authorized_keys", keyContent.getBytes(StandardCharsets.UTF_8), 0600);
execInContainer(containerId, new String[]{"chmod", "600", "/root/.ssh/authorized_keys"}, 5);
LOG.infov("Injected SSH public key into container {0}", containerId);
@@ -1764,8 +1759,4 @@ public class Ec2ContainerManager {
static Optional<String> preferredMetadataSourceIp(Map<String, ContainerNetwork> networks) {
return Ec2MetadataProxy.preferredMetadataSourceIp(networks);
}
-
- private byte[] buildSingleFileTar(String filename, byte[] content, int mode) throws IOException {
- return UserDataPipeline.buildSingleFileTar(filename, content, mode);
- }
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/eks/EksClusterManager.java b/src/main/java/io/github/hectorvent/floci/services/eks/EksClusterManager.java
index 82aca346f4..7d36f7f02e 100644
--- a/src/main/java/io/github/hectorvent/floci/services/eks/EksClusterManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/eks/EksClusterManager.java
@@ -26,6 +26,7 @@ import io.github.hectorvent.floci.core.common.docker.ContainerSpec;
import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;
import io.github.hectorvent.floci.core.common.docker.DockerHostResolver;
import io.github.hectorvent.floci.core.common.docker.PortAllocator;
+import io.github.hectorvent.floci.core.common.docker.RetryingTarCopier;
import io.github.hectorvent.floci.core.common.docker.UserDataPipeline;
import io.github.hectorvent.floci.services.ec2.ClusterNodeInstanceProvider;
import io.github.hectorvent.floci.services.ec2.Ec2InstanceTypeCatalog;
@@ -1419,11 +1420,8 @@ public class EksClusterManager
*/
private void copyWebhookIntoContainer(String containerId, String localFile, String clusterName) {
try {
- lifecycleManager.getDockerClient()
- .copyArchiveToContainerCmd(containerId)
- .withHostResource(localFile)
- .withRemotePath(WEBHOOK_CONFIG_DIR)
- .exec();
+ RetryingTarCopier.copyHostResource(lifecycleManager.getDockerClient(), containerId,
+ WEBHOOK_CONFIG_DIR, localFile);
} catch (Exception e) {
LOG.warnv("EKS token-webhook may not authenticate for cluster {0}: could not copy kubeconfig "
+ "into the k3s container: {1}", clusterName, e.getMessage());
@@ -1851,11 +1849,8 @@ public class EksClusterManager
writeLocalCopy(Paths.get(config.services().eks().dataPath(), "registries", clusterName,
"registries.yaml"), content, clusterName);
try {
- lifecycleManager.getDockerClient()
- .copyArchiveToContainerCmd(containerId)
- .withTarInputStream(new ByteArrayInputStream(tarSingleFile(REGISTRIES_TAR_ENTRY, content)))
- .withRemotePath("/etc")
- .exec();
+ RetryingTarCopier.copyBytes(lifecycleManager.getDockerClient(), containerId, "/etc",
+ REGISTRIES_TAR_ENTRY, content.getBytes(StandardCharsets.UTF_8), 0644);
LOG.infov("Injected ECR registry mirror ({0}) into k3s cluster {1}", endpoint, clusterName);
} catch (Exception e) {
LOG.warnv("EKS cluster {0} gets no ECR registry mirror: could not copy registries.yaml "
diff --git a/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/ContainerLauncher.java b/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/ContainerLauncher.java
index e120e9d516..fce3841ae1 100644
--- a/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/ContainerLauncher.java
+++ b/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/ContainerLauncher.java
@@ -8,6 +8,7 @@ import io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;
import io.github.hectorvent.floci.core.common.docker.ContainerSpec;
import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;
import io.github.hectorvent.floci.core.common.docker.DockerHostResolver;
+import io.github.hectorvent.floci.core.common.docker.RetryingTarCopier;
import io.github.hectorvent.floci.core.common.docker.LaunchedContainerAwsEnv;
import io.github.hectorvent.floci.services.ecr.registry.EcrRegistryManager;
import io.github.hectorvent.floci.services.iam.model.SessionCreds;
@@ -1229,20 +1230,19 @@ public class ContainerLauncher implements LambdaRuntimeLauncher {
}
}
- /**
- * Buffer for the tar-streaming pipe. The default {@link java.io.PipedInputStream} buffer is
- * only 1KB, which forces a writer/reader thread hand-off (wait/notify) every 1KB. Streaming a
- * ~90MB node_modules through that ran at ~0.5MB/s (≈3 min per cold start) — pure synchronization
- * thrash, not I/O. A large buffer lets the tar writer stream ahead so throughput is bound by the
- * Docker daemon, not the pipe.
- */
- private static final int TAR_PIPE_BUFFER_BYTES = 16 * 1024 * 1024;
-
@FunctionalInterface
interface DirectoryTarWriter {
void write(Path sourceDir, OutputStream out) throws IOException;
}
+ /**
+ * See {@link RetryingTarCopier}: these copies retry at the call site because the transport
+ * seam cannot replay a one-shot {@code InputStream} body. A failure throws so launch() cleans
+ * up the half-built container instead of leaking it.
+ */
+ static final int COPY_MAX_ATTEMPTS = 6;
+ static final long COPY_RETRY_BACKOFF_MS = 500L;
+
private void copyDirToContainer(DockerClient dockerClient, String containerId,
Path sourceDir, String remotePath, String functionName) {
copyDirToContainer(dockerClient, containerId, sourceDir, remotePath, functionName,
@@ -1253,7 +1253,14 @@ public class ContainerLauncher implements LambdaRuntimeLauncher {
Path sourceDir, String remotePath, String functionName,
DirectoryTarWriter tarWriter) {
copyDirToContainer(dockerClient, containerId, sourceDir, remotePath, functionName,
- tarWriter, false);
+ tarWriter, false, COPY_MAX_ATTEMPTS, COPY_RETRY_BACKOFF_MS);
+ }
+
+ void copyDirToContainer(DockerClient dockerClient, String containerId,
+ Path sourceDir, String remotePath, String functionName,
+ int maxAttempts, long backoffMillis) {
+ copyDirToContainer(dockerClient, containerId, sourceDir, remotePath, functionName,
+ ContainerLauncher::createTarFromDir, false, maxAttempts, backoffMillis);
}
private void copyDirToContainerStrict(DockerClient dockerClient, String containerId,
@@ -1266,93 +1273,79 @@ public class ContainerLauncher implements LambdaRuntimeLauncher {
Path sourceDir, String remotePath, String functionName,
DirectoryTarWriter tarWriter) {
copyDirToContainer(dockerClient, containerId, sourceDir, remotePath, functionName,
- tarWriter, true);
+ tarWriter, true, COPY_MAX_ATTEMPTS, COPY_RETRY_BACKOFF_MS);
}
private static void copyDirToContainer(DockerClient dockerClient, String containerId,
Path sourceDir, String remotePath, String functionName,
- DirectoryTarWriter tarWriter, boolean failOnTarFailure) {
+ DirectoryTarWriter tarWriter, boolean failOnTarFailure,
+ int maxAttempts, long backoffMillis) {
// No per-copy gating here: the heavy /var/task populate for large code already holds a
// populateSemaphore permit; small-code direct copies and layer copies are light enough
// to run unthrottled.
- try (java.io.PipedOutputStream pos = new java.io.PipedOutputStream();
- java.io.PipedInputStream pis = new java.io.PipedInputStream(pos, TAR_PIPE_BUFFER_BYTES)) {
-
- AtomicReference<IOException> tarFailure = new AtomicReference<>();
- Thread tarStreamer = new Thread(() -> {
- try (pos) {
- tarWriter.write(sourceDir, pos);
- } catch (IOException e) {
- if (failOnTarFailure) {
- tarFailure.set(e);
- } else {
- LOG.errorv("Failed to stream directory tar for function {0}: {1}",
- functionName, e.getMessage());
- }
- }
- }, "tar-streamer-dir-" + functionName);
- tarStreamer.start();
-
- dockerClient.copyArchiveToContainerCmd(containerId)
- .withRemotePath(remotePath)
- .withTarInputStream(pis)
- .exec();
- if (failOnTarFailure) {
- waitForTarStreamer(tarStreamer, tarFailure, functionName, sourceDir);
+ AtomicReference<IOException> tarFailure = new AtomicReference<>();
+ try {
+ RetryingTarCopier.copyStreamed(dockerClient, containerId, remotePath,
+ "dir-" + functionName, out -> {
+ tarFailure.set(null);
+ try {
+ tarWriter.write(sourceDir, out);
+ } catch (IOException e) {
+ tarFailure.set(e);
+ if (!failOnTarFailure) {
+ LOG.errorv("Failed to stream directory tar for function {0}: {1}",
+ functionName, e.getMessage());
+ }
+ throw e;
+ }
+ },
+ maxAttempts, backoffMillis);
+ } catch (RuntimeException e) {
+ // RetryingTarCopier always surfaces a tar-producer failure it captured (so a writer
+ // failure the daemon happened to accept doesn't masquerade as success), wrapped in its
+ // own generic message. A distinct docker-level failure (the daemon itself rejecting the
+ // stream, not our own writer) isn't ours to reinterpret and must propagate as-is.
+ IOException producerFailure = tarFailure.get();
+ if (!isCausedBy(e, producerFailure)) {
+ throw e;
}
- LOG.debugv("Copied directory {0} into container {1} at {2}", sourceDir, containerId, remotePath);
- } catch (Exception e) {
- // Fail loudly so launch() cleans up the half-built container instead of leaking it.
+ if (!failOnTarFailure) {
+ LOG.debugv("Ignoring tolerated tar-producer failure for function {0}", functionName);
+ return;
+ }
+ IOException cause = new IOException("Failed to stream tar for function " + functionName
+ + " from " + sourceDir, producerFailure);
throw new RuntimeException("Failed to copy directory " + sourceDir + " into container "
- + containerId + " for function " + functionName + ": " + e.getMessage(), e);
+ + containerId + " for function " + functionName + ": " + cause.getMessage(), cause);
}
+ LOG.debugv("Copied directory {0} into container {1} at {2}", sourceDir, containerId, remotePath);
+ }
+
+ /** True if {@code target} appears by reference in {@code thrown}'s cause chain. */
+ private static boolean isCausedBy(Throwable thrown, Throwable target) {
+ if (target == null) {
+ return false;
+ }
+ for (Throwable t = thrown; t != null; t = t.getCause()) {
+ if (t == target) {
+ return true;
+ }
+ }
+ return false;
}
private void copyFileToContainer(DockerClient dockerClient, String containerId,
Path sourceFile, String remotePath, String entryName, String functionName) {
- try (java.io.PipedOutputStream pos = new java.io.PipedOutputStream();
- java.io.PipedInputStream pis = new java.io.PipedInputStream(pos, TAR_PIPE_BUFFER_BYTES)) {
-
- new Thread(() -> {
- try (TarArchiveOutputStream tar = newTarStream(pos)) {
- TarArchiveEntry entry = new TarArchiveEntry(entryName);
- entry.setSize(Files.size(sourceFile));
- entry.setMode(0755);
- tar.putArchiveEntry(entry);
- try (var fis = Files.newInputStream(sourceFile)) {
- fis.transferTo(tar);
- }
- tar.closeArchiveEntry();
- } catch (IOException e) {
- LOG.errorv("Failed to stream file tar for function {0}: {1}", functionName, e.getMessage());
- }
- }, "tar-streamer-file-" + functionName).start();
-
- dockerClient.copyArchiveToContainerCmd(containerId)
- .withRemotePath(remotePath)
- .withTarInputStream(pis)
- .exec();
- LOG.debugv("Copied file {0} as {1} into container {2} at {3}", sourceFile, entryName, containerId, remotePath);
- } catch (Exception e) {
- throw new RuntimeException("Failed to copy file " + sourceFile + " into container "
- + containerId + " for function " + functionName + ": " + e.getMessage(), e);
- }
+ copyFileToContainer(dockerClient, containerId, sourceFile, remotePath, entryName, functionName,
+ COPY_MAX_ATTEMPTS, COPY_RETRY_BACKOFF_MS);
}
- private static void waitForTarStreamer(Thread tarStreamer, AtomicReference<IOException> tarFailure,
- String functionName, Path sourcePath) throws IOException {
- try {
- tarStreamer.join();
- } catch (InterruptedException e) {
- Thread.currentThread().interrupt();
- throw new IOException("Interrupted while streaming tar for function " + functionName
- + " from " + sourcePath, e);
- }
- IOException failure = tarFailure.get();
- if (failure != null) {
- throw new IOException("Failed to stream tar for function " + functionName
- + " from " + sourcePath, failure);
- }
+ void copyFileToContainer(DockerClient dockerClient, String containerId,
+ Path sourceFile, String remotePath, String entryName, String functionName,
+ int maxAttempts, long backoffMillis) {
+ RetryingTarCopier.copyFile(dockerClient, containerId, remotePath, entryName, sourceFile,
+ 0755, maxAttempts, backoffMillis);
+ LOG.debugv("Copied file {0} as {1} into container {2} at {3}", sourceFile, entryName, containerId, remotePath);
}
private static boolean isProvidedRuntime(String runtime) {
@@ -1530,7 +1523,7 @@ public class ContainerLauncher implements LambdaRuntimeLauncher {
* are preserved without truncation.
*/
private static void createTarFromDir(Path sourceDir, OutputStream out) throws IOException {
- try (TarArchiveOutputStream tar = newTarStream(out);
+ try (TarArchiveOutputStream tar = RetryingTarCopier.newTarStream(out);
var stream = Files.walk(sourceDir)) {
for (Path path : (Iterable<Path>) stream::iterator) {
if (Files.isDirectory(path)) {
@@ -1548,11 +1541,4 @@ public class ContainerLauncher implements LambdaRuntimeLauncher {
}
}
}
-
- private static TarArchiveOutputStream newTarStream(OutputStream out) {
- TarArchiveOutputStream tar = new TarArchiveOutputStream(out);
- tar.setLongFileMode(TarArchiveOutputStream.LONGFILE_GNU);
- tar.setBigNumberMode(TarArchiveOutputStream.BIGNUMBER_STAR);
- return tar;
- }
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/mwaa/MwaaEnvironmentManager.java b/src/main/java/io/github/hectorvent/floci/services/mwaa/MwaaEnvironmentManager.java
index 333ac13854..fd8d01f868 100644
--- a/src/main/java/io/github/hectorvent/floci/services/mwaa/MwaaEnvironmentManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/mwaa/MwaaEnvironmentManager.java
@@ -10,17 +10,15 @@ import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager.C
import io.github.hectorvent.floci.core.common.docker.ContainerSpec;
import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;
import io.github.hectorvent.floci.core.common.docker.LaunchedContainerAwsEnv;
+import io.github.hectorvent.floci.core.common.docker.RetryingTarCopier;
import io.github.hectorvent.floci.services.mwaa.model.Environment;
import com.github.dockerjava.api.command.InspectContainerResponse;
import com.github.dockerjava.api.exception.NotFoundException;
import com.github.dockerjava.api.model.ContainerNetwork;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
-import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
-import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
import org.jboss.logging.Logger;
-import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.net.HttpURLConnection;
@@ -693,11 +691,8 @@ public class MwaaEnvironmentManager {
return false;
}
try {
- lifecycleManager.getDockerClient()
- .copyArchiveToContainerCmd(containerId)
- .withTarInputStream(new ByteArrayInputStream(tarSingleFile(relativePath, content)))
- .withRemotePath(remoteDir)
- .exec();
+ RetryingTarCopier.copyBytes(lifecycleManager.getDockerClient(), containerId, remoteDir,
+ relativePath, content, 0644);
return true;
} catch (Exception e) {
LOG.warnv("Could not copy {0} into MWAA container {1}: {2}", relativePath, containerId, e.getMessage());
@@ -705,23 +700,6 @@ public class MwaaEnvironmentManager {
}
}
- private static byte[] tarSingleFile(String entryName, byte[] content) {
- try {
- ByteArrayOutputStream out = new ByteArrayOutputStream();
- try (TarArchiveOutputStream tar = new TarArchiveOutputStream(out)) {
- TarArchiveEntry entry = new TarArchiveEntry(entryName);
- entry.setSize(content.length);
- entry.setMode(0644);
- tar.putArchiveEntry(entry);
- tar.write(content);
- tar.closeArchiveEntry();
- }
- return out.toByteArray();
- } catch (IOException e) {
- throw new IllegalStateException("Could not build in-memory tar for " + entryName, e);
- }
- }
-
ContainerExec.Result execInContainer(String containerId, String[] cmd) throws Exception {
return ContainerExec.run(lifecycleManager.getDockerClient(), containerId, cmd, 30).throwIfTimedOut(containerId);
}
diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml
index f2e43b0114..daba00bcf8 100644
--- a/src/main/resources/application.yml
+++ b/src/main/resources/application.yml
@@ -308,12 +308,16 @@ floci:
log-max-file: "3"
docker-host: unix:///var/run/docker.sock
# FLOCI_DOCKER_MAX_CONNECTIONS: connection pool size for every Docker call Floci makes. Each
- # live Lambda container holds 2 connections for its whole lifetime (log follow + exit watcher),
- # so the warm-container ceiling is about max-connections / 2; past it, create/start/stop/remove
- # wait for a free connection and the emulator stalls. 1024 covers the 500 containers the
- # runtime API port range allows. Keep in step with the @WithDefault on
+ # live Lambda container holds 1 connection for its whole lifetime (its exit watcher) plus one
+ # per extension; log follow uses the streaming pool below. Past the limit,
+ # create/start/stop/remove wait for a free connection and the emulator stalls. 1024 covers
+ # the 500 containers the runtime API port range allows. Keep in step with the @WithDefault on
# EmulatorConfig.DockerConfig.maxConnections: a value set here WINS over the annotation.
max-connections: 1024
+ # FLOCI_DOCKER_STREAMING_MAX_CONNECTIONS: separate pool for container log-follow streams, so
+ # they never starve create/start/stop/remove. Keep in step with the @WithDefault on
+ # EmulatorConfig.DockerConfig.streamingMaxConnections: a value set here WINS over the annotation.
+ streaming-max-connections: 512
resource-namespace: "" # FLOCI_DOCKER_RESOURCE_NAMESPACE; scopes child container/volume names
docker-config-path: ""
# extra-labels: # Extra Docker labels applied to every Floci-created container and volume
diff --git a/src/test/java/io/github/hectorvent/floci/config/ApplicationDefaultsTest.java b/src/test/java/io/github/hectorvent/floci/config/ApplicationDefaultsTest.java
index e5d5a8fc1b..8b9a10bfd5 100644
--- a/src/test/java/io/github/hectorvent/floci/config/ApplicationDefaultsTest.java
+++ b/src/test/java/io/github/hectorvent/floci/config/ApplicationDefaultsTest.java
@@ -132,4 +132,24 @@ class ApplicationDefaultsTest {
assertEquals(String.valueOf(maxConnections), fallback.value(),
"the @WithDefault must match application.yml, which wins over it");
}
+
+ // Catches: streaming-max-connections declared and documented but absent from the shipped
+ // application.yml, or a yml value that silently disagrees with the @WithDefault fallback.
+ @Test
+ void productionConfigDeclaresTheStreamingDockerPoolAtItsDefault() throws IOException, NoSuchMethodException {
+ JsonNode streamingMaxConnections = new YAMLMapper()
+ .readTree(Path.of("src/main/resources/application.yml").toFile())
+ .path("floci").path("docker").path("streaming-max-connections");
+
+ assertFalse(streamingMaxConnections.isMissingNode(),
+ "streaming-max-connections should be declared in application.yml");
+ assertEquals(512, streamingMaxConnections.asInt());
+
+ WithDefault fallback = EmulatorConfig.DockerConfig.class
+ .getMethod("streamingMaxConnections")
+ .getAnnotation(WithDefault.class);
+ assertNotNull(fallback, "streamingMaxConnections should declare a fallback default");
+ assertEquals("512", fallback.value(),
+ "the @WithDefault must match application.yml, which wins over it");
+ }
}
diff --git a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerCreateConflictRecoveryTest.java b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerCreateConflictRecoveryTest.java
new file mode 100644
index 0000000000..9fce8f85f3
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerCreateConflictRecoveryTest.java
@@ -0,0 +1,226 @@
+package io.github.hectorvent.floci.core.common.docker;
+
+import com.github.dockerjava.api.DockerClient;
+import com.github.dockerjava.api.command.CreateContainerCmd;
+import com.github.dockerjava.api.command.ListContainersCmd;
+import com.github.dockerjava.api.exception.ConflictException;
+import com.github.dockerjava.api.model.Container;
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.services.lambda.launcher.ImageCacheService;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.Mockito.RETURNS_SELF;
+import static org.mockito.Mockito.lenient;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * A create retry can hit the daemon twice for the same logical create: the first attempt's
+ * response is lost to a transient socket error ({@link DockerRetry}), but the daemon already
+ * created the named container, so the retry's {@code createCmd.exec()} fails with a 409 name
+ * conflict instead of a transient I/O error. Without recovery this surfaces as a create
+ * failure while a live, untracked container sits on the daemon.
+ */
+@ExtendWith(MockitoExtension.class)
+class ContainerLifecycleManagerCreateConflictRecoveryTest {
+
+ @Mock
+ DockerClient dockerClient;
+
+ @Mock
+ ImageCacheService imageCacheService;
+
+ @Mock
+ ContainerDetector containerDetector;
+
+ @Mock
+ PortAllocator portAllocator;
+
+ @Mock
+ EmulatorConfig config;
+
+ @Mock
+ EmulatorConfig.DockerConfig dockerConfig;
+
+ @Mock
+ EmulatorConfig.TlsConfig tlsConfig;
+
+ @BeforeEach
+ void setUp() {
+ lenient().when(config.docker()).thenReturn(dockerConfig);
+ lenient().when(config.tls()).thenReturn(tlsConfig);
+ // create() now resolves the image through the cache before building the create command.
+ lenient().when(imageCacheService.ensureImageExists("busybox:stable")).thenReturn("busybox:stable");
+ lenient().when(dockerConfig.resourceNamespace()).thenReturn(Optional.empty());
+ }
+
+ @Test
+ void createAdoptsContainerThatWonTheRaceOnRetryConflict() {
+ CreateContainerCmd createCmd = mock(CreateContainerCmd.class, RETURNS_SELF);
+ when(dockerClient.createContainerCmd("busybox:stable")).thenReturn(createCmd);
+ // The container that actually got created on the daemon carries exactly the labels this
+ // create() call applied (including the per-call attempt-id), captured here rather than
+ // hardcoded, since createWithConflictRecovery only adopts a container proven to be THIS
+ // call's own lost-response attempt, not merely one with matching image/spec labels.
+ @SuppressWarnings("unchecked")
+ ArgumentCaptor<Map<String, String>> labelsCaptor =
+ ArgumentCaptor.forClass(Map.class);
+ when(createCmd.withLabels(labelsCaptor.capture())).thenReturn(createCmd);
+ when(createCmd.exec()).thenThrow(new ConflictException("named container already exists"));
+
+ Container existing = mock(Container.class);
+ when(existing.getId()).thenReturn("winning-container-id");
+ when(existing.getNames()).thenReturn(new String[] {"/emulator-fixed-name"});
+ when(existing.getLabels()).thenAnswer(inv -> labelsCaptor.getValue());
+
+ ListContainersCmd listCmd = mock(ListContainersCmd.class, RETURNS_SELF);
+ when(dockerClient.listContainersCmd()).thenReturn(listCmd);
+ when(listCmd.exec()).thenReturn(List.of(existing));
+
+ ContainerSpec spec = new ContainerSpec(
+ "busybox:stable", "emulator-fixed-name", List.of(), null, null, null, Map.of(),
+ List.of(), null, List.of(), List.of(), List.of(), Map.of(), null, false, null,
+ List.of(), null, null, List.of());
+
+ String containerId = manager().create(spec);
+
+ assertEquals("winning-container-id", containerId);
+ }
+
+ /**
+ * Image and label matching alone cannot tell "my own create's lost response" apart from a
+ * second, genuinely concurrent {@code create()} call racing on the same fixed name, image,
+ * and spec labels (e.g. a duplicate CreateBroker request retried by an AWS SDK client while
+ * the original is still in flight), both produce a container satisfying {@code matchesSpec}.
+ * Adopting the other call's container means starting, modifying, or removing a container this
+ * invocation doesn't own. Each {@code create()} call tags its own createCmd with a fresh
+ * per-call attempt id; a name-conflicting container missing (or mismatching) this exact id
+ * belongs to a different call and must not be adopted.
+ */
+ @Test
+ void createRethrowsConflictWhenExistingContainerIsFromADifferentConcurrentCreateCall() {
+ CreateContainerCmd createCmd = mock(CreateContainerCmd.class, RETURNS_SELF);
+ when(dockerClient.createContainerCmd("busybox:stable")).thenReturn(createCmd);
+ when(createCmd.exec()).thenThrow(new ConflictException("named container already exists"));
+
+ Container existing = mock(Container.class);
+ when(existing.getNames()).thenReturn(new String[] {"/emulator-fixed-name"});
+ // Same image and default/spec labels as this call would apply, but a different
+ // create() invocation's attempt id, simulating a genuinely concurrent racing caller,
+ // not this call's own lost-response retry.
+ Map<String, String> foreignLabels = new HashMap<>(
+ Map.of("floci", "true", "floci_emulator", "floci-aws"));
+ foreignLabels.put(ContainerLifecycleManager.CREATE_ATTEMPT_LABEL, "some-other-callers-attempt-id");
+ when(existing.getLabels()).thenReturn(foreignLabels);
+
+ ListContainersCmd listCmd = mock(ListContainersCmd.class, RETURNS_SELF);
+ when(dockerClient.listContainersCmd()).thenReturn(listCmd);
+ when(listCmd.exec()).thenReturn(List.of(existing));
+
+ ContainerSpec spec = new ContainerSpec(
+ "busybox:stable", "emulator-fixed-name", List.of(), null, null, null, Map.of(),
+ List.of(), null, List.of(), List.of(), List.of(), Map.of(), null, false, null,
+ List.of(), null, null, List.of());
+
+ assertThrows(ConflictException.class, () -> manager().create(spec));
+ }
+
+ /**
+ * The list API reports {@code Container.getImage()} as an image ID/digest rather than the tag
+ * whenever the tag has been re-pulled, retagged, or removed since the container was created,
+ * routine on a long-lived host. Comparing it against the spec's tag would then reject the
+ * container THIS call just created, rethrow the conflict, and leak that container untracked:
+ * exactly the failure conflict recovery exists to prevent, and only under the daemon churn that
+ * makes a lost create response likely in the first place. The per-call attempt id is already
+ * proof of ownership no other call can reproduce, so adoption must not hinge on the image
+ * string.
+ */
+ @Test
+ void createAdoptsOwnContainerWhenListApiReportsImageAsADigest() {
+ CreateContainerCmd createCmd = mock(CreateContainerCmd.class, RETURNS_SELF);
+ when(dockerClient.createContainerCmd("busybox:stable")).thenReturn(createCmd);
+ @SuppressWarnings("unchecked")
+ ArgumentCaptor<Map<String, String>> labelsCaptor =
+ ArgumentCaptor.forClass(Map.class);
+ when(createCmd.withLabels(labelsCaptor.capture())).thenReturn(createCmd);
+ when(createCmd.exec()).thenThrow(new ConflictException("named container already exists"));
+
+ Container existing = mock(Container.class);
+ when(existing.getId()).thenReturn("winning-container-id");
+ when(existing.getNames()).thenReturn(new String[] {"/emulator-fixed-name"});
+ // lenient: matchesSpec must NOT read the image at all. Stubbing it anyway is the regression
+ // guard, reintroduce a tag comparison and this digest makes the adoption fail.
+ lenient().when(existing.getImage()).thenReturn(
+ "sha256:1f2e3d4c5b6a798877665544332211009988776655443322110099887766554433");
+ when(existing.getLabels()).thenAnswer(inv -> labelsCaptor.getValue());
+
+ ListContainersCmd listCmd = mock(ListContainersCmd.class, RETURNS_SELF);
+ when(dockerClient.listContainersCmd()).thenReturn(listCmd);
+ when(listCmd.exec()).thenReturn(List.of(existing));
+
+ ContainerSpec spec = new ContainerSpec(
+ "busybox:stable", "emulator-fixed-name", List.of(), null, null, null, Map.of(),
+ List.of(), null, List.of(), List.of(), List.of(), Map.of(), null, false, null,
+ List.of(), null, null, List.of());
+
+ assertEquals("winning-container-id", manager().create(spec));
+ }
+
+ @Test
+ void createRethrowsConflictWhenExistingContainerLabelsDoNotMatch() {
+ CreateContainerCmd createCmd = mock(CreateContainerCmd.class, RETURNS_SELF);
+ when(dockerClient.createContainerCmd("busybox:stable")).thenReturn(createCmd);
+ when(createCmd.exec()).thenThrow(new ConflictException("named container already exists"));
+
+ Container existing = mock(Container.class);
+ when(existing.getNames()).thenReturn(new String[] {"/emulator-fixed-name"});
+ when(existing.getLabels()).thenReturn(
+ Map.of("floci", "true", "floci_emulator", "floci-lambda"));
+
+ ListContainersCmd listCmd = mock(ListContainersCmd.class, RETURNS_SELF);
+ when(dockerClient.listContainersCmd()).thenReturn(listCmd);
+ when(listCmd.exec()).thenReturn(List.of(existing));
+
+ ContainerSpec spec = new ContainerSpec(
+ "busybox:stable", "emulator-fixed-name", List.of(), null, null, null, Map.of(),
+ List.of(), null, List.of(), List.of(), List.of(), Map.of(), null, false, null,
+ List.of(), null, null, List.of());
+
+ assertThrows(ConflictException.class, () -> manager().create(spec));
+ }
+
+ @Test
+ void createRethrowsConflictWhenNoNamedContainerExists() {
+ CreateContainerCmd createCmd = mock(CreateContainerCmd.class, RETURNS_SELF);
+ when(dockerClient.createContainerCmd("busybox:stable")).thenReturn(createCmd);
+ when(createCmd.exec()).thenThrow(new ConflictException("named container already exists"));
+
+ ListContainersCmd listCmd = mock(ListContainersCmd.class, RETURNS_SELF);
+ when(dockerClient.listContainersCmd()).thenReturn(listCmd);
+ when(listCmd.exec()).thenReturn(List.of());
+
+ ContainerSpec spec = new ContainerSpec(
+ "busybox:stable", "emulator-fixed-name", List.of(), null, null, null, Map.of(),
+ List.of(), null, List.of(), List.of(), List.of(), Map.of(), null, false, null,
+ List.of(), null, null, List.of());
+
+ assertThrows(ConflictException.class, () -> manager().create(spec));
+ }
+
+ private ContainerLifecycleManager manager() {
+ return new ContainerLifecycleManager(
+ dockerClient, imageCacheService, containerDetector, portAllocator, config);
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerLabelsTest.java b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerLabelsTest.java
index 746851b2f7..ebaa9e94c0 100644
--- a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerLabelsTest.java
+++ b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerLabelsTest.java
@@ -22,6 +22,7 @@ import org.mockito.ArgumentCaptor;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
+import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
@@ -314,10 +315,17 @@ class ContainerLifecycleManagerLabelsTest {
return createCmd;
}
+ /**
+ * Strips the per-call {@code ContainerLifecycleManager.CREATE_ATTEMPT_LABEL}, a random id
+ * generated fresh on every {@code create()} call for conflict-recovery adoption safety,
+ * orthogonal to the default-label behavior this test file covers.
+ */
private Map<String, String> capturedLabels(CreateContainerCmd createCmd) {
ArgumentCaptor<Map<String, String>> labels = labelsCaptor();
verify(createCmd).withLabels(labels.capture());
- return labels.getValue();
+ Map<String, String> captured = new HashMap<>(labels.getValue());
+ captured.remove(ContainerLifecycleManager.CREATE_ATTEMPT_LABEL);
+ return captured;
}
@SuppressWarnings("unchecked")
diff --git a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerStartTest.java b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerStartTest.java
new file mode 100644
index 0000000000..48244904ad
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerStartTest.java
@@ -0,0 +1,154 @@
+package io.github.hectorvent.floci.core.common.docker;
+
+import com.github.dockerjava.api.DockerClient;
+import com.github.dockerjava.api.command.InspectContainerCmd;
+import com.github.dockerjava.api.command.InspectContainerResponse;
+import com.github.dockerjava.api.command.StartContainerCmd;
+import com.github.dockerjava.api.exception.NotModifiedException;
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.services.lambda.launcher.ImageCacheService;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.IntConsumer;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Start semantics that must hold with retries living at the transport seam
+ * ({@link RetryingDockerHttpClient}), not at this call site.
+ *
+ * <p>The load-bearing piece the transport cannot provide: when the daemon honoured a start whose
+ * response was lost to a broken pipe, the transport's replay meets HTTP 304, a perfectly
+ * successful response at transport level, which docker-java converts to
+ * {@link NotModifiedException} <em>above</em> the transport. The call site must treat it as
+ * success, or a recovered blip becomes a hard launch failure.
+ *
+ * <p>Equally load-bearing in the other direction: this manager sees the docker API through a
+ * client whose transport has already spent the full retry budget, so it must not loop again
+ * itself, an outer loop would compound backoff on an already-exhausted inner one (~90s+ worst
+ * case for a single call) and multiply daemon pressure exactly when the socket is saturated.
+ */
+@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.LENIENT)
+class ContainerLifecycleManagerStartTest {
+
+ @Mock
+ private ImageCacheService imageCacheService;
+
+ @Mock
+ private ContainerDetector containerDetector;
+
+ @Mock
+ private PortAllocator portAllocator;
+
+ @Mock
+ private EmulatorConfig config;
+
+ private final ContainerSpec spec = new ContainerSpec("busybox:latest");
+
+ /**
+ * A client whose {@code startContainerCmd(...).exec()} defers to {@code behaviour}, which is
+ * handed the 1-based attempt number and may throw. Attempts are counted in {@code execCalls}.
+ */
+ private DockerClient dockerWhoseStartExec(AtomicInteger execCalls, IntConsumer behaviour) {
+ StartContainerCmd cmd = mock(StartContainerCmd.class);
+ when(cmd.exec()).thenAnswer(inv -> {
+ behaviour.accept(execCalls.incrementAndGet());
+ return null;
+ });
+ DockerClient docker = mock(DockerClient.class, RETURNS_DEEP_STUBS);
+ when(docker.startContainerCmd("container-abc")).thenReturn(cmd);
+ return docker;
+ }
+
+ private ContainerLifecycleManager manager(DockerClient docker) {
+ return new ContainerLifecycleManager(
+ docker, imageCacheService, containerDetector, portAllocator, config);
+ }
+
+ @Test
+ void startCreatedTreatsNotModifiedAsSuccess() {
+ AtomicInteger execCalls = new AtomicInteger();
+ DockerClient docker = dockerWhoseStartExec(execCalls, attempt -> {
+ // The transport's retry replayed a start the daemon had honoured before the socket
+ // died; docker answered the replay with HTTP 304, raised here as NotModifiedException.
+ throw new NotModifiedException("Container already started");
+ });
+
+ ContainerLifecycleManager.ContainerInfo info = assertDoesNotThrow(
+ () -> manager(docker).startCreated("container-abc", spec),
+ "a 304 means the container is running, that is success, not failure");
+
+ assertEquals("container-abc", info.containerId());
+ assertEquals(1, execCalls.get());
+ }
+
+ @Test
+ void startCreatedDoesNotRetryOnTopOfTheTransport() {
+ AtomicInteger execCalls = new AtomicInteger();
+ RuntimeException transportGaveUp = new RuntimeException(new IOException("Broken pipe"));
+ DockerClient docker = dockerWhoseStartExec(execCalls, attempt -> {
+ throw transportGaveUp;
+ });
+
+ RuntimeException thrown = assertThrows(RuntimeException.class,
+ () -> manager(docker).startCreated("container-abc", spec));
+
+ assertSame(transportGaveUp, thrown,
+ "a transient error surfacing here means the transport already spent the retry"
+ + " budget; it must surface unchanged");
+ assertEquals(1, execCalls.get(),
+ "no call-site loop: an outer retry would compound backoff on an exhausted"
+ + " inner one and multiply pressure on a saturated socket");
+ }
+
+ @Test
+ void startCreatedDoesNotSwallowNonTransientFailure() {
+ AtomicInteger execCalls = new AtomicInteger();
+ RuntimeException portConflict =
+ new RuntimeException("Bind for 0.0.0.0:80 failed: port is already allocated");
+ DockerClient docker = dockerWhoseStartExec(execCalls, attempt -> {
+ throw portConflict;
+ });
+
+ RuntimeException thrown = assertThrows(RuntimeException.class,
+ () -> manager(docker).startCreated("container-abc", spec));
+
+ assertSame(portConflict, thrown, "a genuine daemon rejection must surface unchanged");
+ assertEquals(1, execCalls.get());
+ }
+
+ @Test
+ void adoptTreatsNotModifiedAsSuccess() {
+ AtomicInteger execCalls = new AtomicInteger();
+ DockerClient docker = dockerWhoseStartExec(execCalls, attempt -> {
+ throw new NotModifiedException("Container already started");
+ });
+
+ // adopt() inspects first, sees a stopped container, and starts it.
+ InspectContainerResponse inspect = mock(InspectContainerResponse.class, RETURNS_DEEP_STUBS);
+ when(inspect.getState().getRunning()).thenReturn(Boolean.FALSE);
+ InspectContainerCmd inspectCmd = mock(InspectContainerCmd.class);
+ when(inspectCmd.exec()).thenReturn(inspect);
+ when(docker.inspectContainerCmd("container-abc")).thenReturn(inspectCmd);
+
+ assertDoesNotThrow(() -> manager(docker).adopt("container-abc", List.of()),
+ "adopting a container that turns out to be running already is success");
+
+ assertEquals(1, execCalls.get());
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerVolumeTest.java b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerVolumeTest.java
index 2210157670..b31719c847 100644
--- a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerVolumeTest.java
+++ b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerVolumeTest.java
@@ -4,6 +4,7 @@ import com.github.dockerjava.api.DockerClient;
import com.github.dockerjava.api.command.CreateContainerCmd;
import com.github.dockerjava.api.command.CreateContainerResponse;
import com.github.dockerjava.api.command.CreateVolumeCmd;
+import com.github.dockerjava.api.command.CreateVolumeResponse;
import com.github.dockerjava.api.command.InspectVolumeCmd;
import com.github.dockerjava.api.command.InspectVolumeResponse;
import com.github.dockerjava.api.command.ListVolumesCmd;
@@ -24,11 +25,15 @@ import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
+import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Optional;
import java.util.OptionalInt;
import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.IntConsumer;
+import java.util.function.IntFunction;
import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.Mockito.*;
@@ -431,6 +436,95 @@ class ContainerLifecycleManagerVolumeTest {
verify(dockerClient, times(2)).createContainerCmd("busybox:stable");
}
+ /** Stubs {@code inspectVolumeCmd("vol").exec()} to defer to {@code behaviour}, counting calls. */
+ private AtomicInteger inspectVolumeBehaving(IntFunction<InspectVolumeResponse> behaviour) {
+ AtomicInteger calls = new AtomicInteger();
+ InspectVolumeCmd cmd = mock(InspectVolumeCmd.class);
+ when(dockerClient.inspectVolumeCmd("vol")).thenReturn(cmd);
+ when(cmd.exec()).thenAnswer(inv -> behaviour.apply(calls.incrementAndGet()));
+ return calls;
+ }
+
+ /** Stubs {@code createVolumeCmd()...exec()} to defer to {@code behaviour}, counting calls. */
+ private AtomicInteger createVolumeBehaving(IntConsumer behaviour) {
+ AtomicInteger calls = new AtomicInteger();
+ CreateVolumeCmd cmd = mock(CreateVolumeCmd.class, RETURNS_SELF);
+ when(dockerClient.createVolumeCmd()).thenReturn(cmd);
+ when(cmd.exec()).thenAnswer(inv -> {
+ behaviour.accept(calls.incrementAndGet());
+ return mock(CreateVolumeResponse.class);
+ });
+ return calls;
+ }
+
+ @Test
+ void ensureVolume_createsMissingVolumeExactlyOnce() {
+ AtomicInteger inspects = inspectVolumeBehaving(attempt -> {
+ throw new NotFoundException("No such volume: vol");
+ });
+ AtomicInteger creates = createVolumeBehaving(attempt -> {
+ });
+
+ manager.ensureVolume("vol");
+
+ assertEquals(1, inspects.get());
+ assertEquals(1, creates.get(), "a missing volume must be created");
+ }
+
+ @Test
+ void ensureVolume_doesNotCreateWhenVolumeAlreadyExists() {
+ // This existence guard is what keeps ensureVolume idempotent now that transient-I/O
+ // retries live at the transport: when the daemon created the volume but the response was
+ // lost to a broken pipe, the transport's replayed create finds the volume already there,
+ // POST /volumes/create with the same name is itself idempotent, and later ensureVolume
+ // calls see it exists and do nothing.
+ AtomicInteger inspects = inspectVolumeBehaving(attempt ->
+ mock(InspectVolumeResponse.class));
+
+ manager.ensureVolume("vol");
+
+ assertEquals(1, inspects.get());
+ verify(dockerClient, never()).createVolumeCmd();
+ }
+
+ @Test
+ void ensureVolume_doesNotRetryOnTopOfTheTransport() {
+ inspectVolumeBehaving(attempt -> {
+ throw new NotFoundException("No such volume: vol");
+ });
+ RuntimeException transportGaveUp = new RuntimeException(new IOException("Broken pipe"));
+ AtomicInteger creates = createVolumeBehaving(attempt -> {
+ throw transportGaveUp;
+ });
+
+ RuntimeException thrown =
+ assertThrows(RuntimeException.class, () -> manager.ensureVolume("vol"));
+
+ assertSame(transportGaveUp, thrown,
+ "a transient error surfacing here means the transport already spent the retry"
+ + " budget; it must surface unchanged");
+ assertEquals(1, creates.get(),
+ "no call-site loop: an outer retry would compound backoff on an exhausted"
+ + " inner one");
+ }
+
+ @Test
+ void ensureVolume_doesNotSwallowDaemonRejection() {
+ inspectVolumeBehaving(attempt -> {
+ throw new NotFoundException("No such volume: vol");
+ });
+ DockerException rejected = new DockerException("volume name already in use", 409);
+ AtomicInteger creates = createVolumeBehaving(attempt -> {
+ throw rejected;
+ });
+
+ DockerException thrown =
+ assertThrows(DockerException.class, () -> manager.ensureVolume("vol"));
+
+ assertSame(rejected, thrown, "a genuine daemon rejection must surface unchanged");
+ assertEquals(1, creates.get(), "a name conflict never clears on a retry");
+ }
+
@Test
void ensureSharedVolume_startsHelperThroughTheTranslatingStartContainer() {
// github.com/floci-io/floci/issues/2243 (follow-up, PR #2797 review): the shared-volume
diff --git a/src/test/java/io/github/hectorvent/floci/core/common/docker/DockerClientProducerTest.java b/src/test/java/io/github/hectorvent/floci/core/common/docker/DockerClientProducerTest.java
index f3cf1dea0c..cc7756cde1 100644
--- a/src/test/java/io/github/hectorvent/floci/core/common/docker/DockerClientProducerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/core/common/docker/DockerClientProducerTest.java
@@ -255,6 +255,34 @@ class DockerClientProducerTest {
"An explicitly-configured docker-host should take priority over the Windows named-pipe fallback");
}
+ // Control-plane calls (create/start/stop/remove/copyArchive) and long-lived streaming
+ // calls (log-follow, exec output streams) must not share one connection pool: a
+ // container's whole-lifetime streams would otherwise occupy pool slots that
+ // short-lived control-plane calls need, starving them into
+ // ConnectionRequestTimeoutException. The @Default and @StreamingDocker beans must
+ // therefore be genuinely separate DockerClient instances (and thus separate pools),
+ // not the same instance handed out under two qualifiers.
+ @Test
+ void dockerClientAndStreamingDockerClient_areDistinctInstances() {
+ EmulatorConfig config = mock(EmulatorConfig.class);
+ EmulatorConfig.DockerConfig docker = mock(EmulatorConfig.DockerConfig.class);
+ when(config.docker()).thenReturn(docker);
+ when(docker.dockerHost()).thenReturn("unix:///var/run/docker.sock");
+ when(docker.dockerConfigPath()).thenReturn(Optional.empty());
+ when(docker.maxConnections()).thenReturn(100);
+ when(docker.streamingMaxConnections()).thenReturn(512);
+
+ DockerClientProducer producer = new DockerClientProducer(config);
+
+ DockerClient controlPlaneClient = producer.dockerClient();
+ DockerClient streamingClient = producer.streamingDockerClient();
+
+ assertNotNull(controlPlaneClient);
+ assertNotNull(streamingClient);
+ assertNotSame(controlPlaneClient, streamingClient,
+ "Control-plane and streaming DockerClient beans must use separate connection pools");
+ }
+
private static void writeContextFixture(Path dockerConfigDir, String contextName, String host)
throws IOException {
writeContextFixture(dockerConfigDir, contextName, host, false);
diff --git a/src/test/java/io/github/hectorvent/floci/core/common/docker/DockerRetryTest.java b/src/test/java/io/github/hectorvent/floci/core/common/docker/DockerRetryTest.java
new file mode 100644
index 0000000000..0c436335ba
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/core/common/docker/DockerRetryTest.java
@@ -0,0 +1,177 @@
+package io.github.hectorvent.floci.core.common.docker;
+
+import org.apache.hc.core5.http.ConnectionRequestTimeoutException;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.io.InterruptedIOException;
+import java.net.SocketTimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Docker daemon calls made concurrently over the single shared socket intermittently fail with
+ * a transient I/O error ({@code Broken pipe} / {@code Connection reset}) that docker-java wraps
+ * in a {@link RuntimeException}. Such failures clear on a retry; anything else must surface
+ * immediately. {@link DockerRetry} centralises that policy for every docker call site.
+ */
+class DockerRetryTest {
+
+ @Test
+ void brokenPipeNestedInRuntimeExceptionIsTransient() {
+ assertTrue(DockerRetry.isTransientIo(new RuntimeException(new IOException("Broken pipe"))));
+ }
+
+ @Test
+ void connectionResetIsTransient() {
+ assertTrue(DockerRetry.isTransientIo(new RuntimeException("Connection reset by peer")));
+ }
+
+ @Test
+ void directIoExceptionIsTransient() {
+ assertTrue(DockerRetry.isTransientIo(new IOException("Broken pipe")));
+ }
+
+ @Test
+ void plainRuntimeExceptionIsNotTransient() {
+ assertFalse(DockerRetry.isTransientIo(new RuntimeException("400 Bad Request")));
+ }
+
+ // ConnectionRequestTimeoutException extends InterruptedIOException extends IOException, so the
+ // old "any IOException in the cause chain is transient" rule wrongly retried it. It means the
+ // httpclient5 pool has no free connection/lease after waiting the full request timeout, the
+ // pool is exhausted, not the socket blipping. Retrying just re-enters another full wait and
+ // adds more pressure to an already-starved pool, so this must NOT be classified as transient.
+ @Test
+ void connectionRequestTimeoutIsNotTransient() {
+ assertFalse(DockerRetry.isTransientIo(new ConnectionRequestTimeoutException("Timeout waiting for connection from pool")));
+ }
+
+ // A plain InterruptedIOException (e.g. from Thread.interrupt() during a blocking read) signals
+ // cancellation/shutdown, not a socket blip. Retrying it masks the interrupt and delays
+ // shutdown, so it must not be classified as transient either, same rule as its
+ // ConnectionRequestTimeoutException subtype above, just not pool-exhaustion-specific.
+ @Test
+ void plainInterruptedIoExceptionIsNotTransient() {
+ assertFalse(DockerRetry.isTransientIo(new InterruptedIOException("interrupted")));
+ }
+
+ // A single forward walk returns true as soon as it hits the outer IOException, never
+ // reaching the InterruptedIOException wrapped inside it, the exclusion must be checked
+ // over the whole chain before the general "any IOException is transient" rule applies.
+ @Test
+ void interruptedIoExceptionWrappedInAnotherIOExceptionIsStillNotTransient() {
+ assertFalse(DockerRetry.isTransientIo(
+ new IOException("wrapper", new InterruptedIOException("interrupted"))));
+ }
+
+ // java.net.SocketTimeoutException extends InterruptedIOException, but a slow daemon
+ // response is a socket blip like Broken pipe/Connection reset, not a genuine interrupt or
+ // pool exhaustion -- it must stay retryable.
+ @Test
+ void socketTimeoutExceptionIsStillTransientDespiteExtendingInterruptedIOException() {
+ assertTrue(DockerRetry.isTransientIo(new SocketTimeoutException("Read timed out")));
+ }
+
+ @Test
+ void recoversAfterTransientFailures() {
+ AtomicInteger calls = new AtomicInteger();
+ DockerRetry.run(4, 0L, () -> {
+ if (calls.incrementAndGet() < 3) {
+ throw new RuntimeException(new IOException("Broken pipe"));
+ }
+ });
+ assertEquals(3, calls.get());
+ }
+
+ @Test
+ void returnsValueFromSupplierVariant() {
+ AtomicInteger calls = new AtomicInteger();
+ String result = DockerRetry.call(3, 0L, () -> {
+ if (calls.incrementAndGet() < 2) {
+ throw new RuntimeException(new IOException("Broken pipe"));
+ }
+ return "ok";
+ });
+ assertEquals("ok", result);
+ assertEquals(2, calls.get());
+ }
+
+ @Test
+ void exhaustsThenRethrowsLastTransient() {
+ AtomicInteger calls = new AtomicInteger();
+ RuntimeException boom = assertThrows(RuntimeException.class,
+ () -> DockerRetry.run(3, 0L, () -> {
+ calls.incrementAndGet();
+ throw new RuntimeException(new IOException("Broken pipe"));
+ }));
+ assertEquals(3, calls.get());
+ assertTrue(DockerRetry.isTransientIo(boom));
+ }
+
+ @Test
+ void backoffIsExponentialAndCapped() {
+ assertEquals(500L, DockerRetry.backoffDelay(1, 500L, 8000L));
+ assertEquals(1000L, DockerRetry.backoffDelay(2, 500L, 8000L));
+ assertEquals(2000L, DockerRetry.backoffDelay(3, 500L, 8000L));
+ assertEquals(4000L, DockerRetry.backoffDelay(4, 500L, 8000L));
+ assertEquals(8000L, DockerRetry.backoffDelay(5, 500L, 8000L)); // 8000 hits cap
+ assertEquals(8000L, DockerRetry.backoffDelay(9, 500L, 8000L)); // capped, no overflow
+ assertEquals(0L, DockerRetry.backoffDelay(3, 0L, 8000L)); // zero base stays zero
+ }
+
+ @Test
+ void doesNotRetryNonTransient() {
+ AtomicInteger calls = new AtomicInteger();
+ IllegalStateException expected = new IllegalStateException("nope");
+ RuntimeException thrown = assertThrows(RuntimeException.class,
+ () -> DockerRetry.run(5, 0L, () -> {
+ calls.incrementAndGet();
+ throw expected;
+ }));
+ assertEquals(1, calls.get());
+ assertSame(expected, thrown);
+ }
+
+ @Test
+ void interruptedExceptionFromOperationRestoresFlagAndIsNotRetried() {
+ // Catches: wrapping an InterruptedException from the operation without restoring the
+ // thread's interrupt flag, or retrying past a cancellation.
+ AtomicInteger calls = new AtomicInteger();
+ try {
+ assertThrows(RuntimeException.class, () -> DockerRetry.call(5, 0L, () -> {
+ calls.incrementAndGet();
+ throw new InterruptedException("stop");
+ }));
+ assertTrue(Thread.currentThread().isInterrupted());
+ assertEquals(1, calls.get());
+ } finally {
+ Thread.interrupted();
+ }
+ }
+
+ @Test
+ void rejectsMaxAttemptsBelowOne() {
+ AtomicInteger calls = new AtomicInteger();
+ assertThrows(IllegalArgumentException.class,
+ () -> DockerRetry.run(0, 0L, calls::incrementAndGet));
+ assertThrows(IllegalArgumentException.class,
+ () -> DockerRetry.run(-1, 0L, calls::incrementAndGet));
+ // The misconfiguration must fail loudly, not silently skip the call entirely.
+ assertEquals(0, calls.get());
+ }
+
+ @Test
+ void brokenPipeMatchesRegardlessOfCase() {
+ // The daemon's capitalisation is not a contract, and the default locale must not decide
+ // whether a transient drop is recognised.
+ assertTrue(DockerRetry.isTransientIo(new RuntimeException("broken pipe")));
+ assertTrue(DockerRetry.isTransientIo(new RuntimeException("BROKEN PIPE")));
+ assertTrue(DockerRetry.isTransientIo(new RuntimeException("Broken Pipe")));
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/core/common/docker/RetryingDockerHttpClientTest.java b/src/test/java/io/github/hectorvent/floci/core/common/docker/RetryingDockerHttpClientTest.java
new file mode 100644
index 0000000000..1eec5ab0bf
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/core/common/docker/RetryingDockerHttpClientTest.java
@@ -0,0 +1,358 @@
+package io.github.hectorvent.floci.core.common.docker;
+
+import com.github.dockerjava.transport.DockerHttpClient;
+import com.github.dockerjava.transport.DockerHttpClient.Request;
+import com.github.dockerjava.transport.DockerHttpClient.Response;
+import org.apache.hc.core5.http.ConnectionRequestTimeoutException;
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.IntFunction;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.Mockito.mock;
+
+/**
+ * The docker daemon drops connections mid-call under fan-out load, and docker-java surfaces the
+ * blip as {@code RuntimeException(IOException("Broken pipe"))} from inside
+ * {@code DockerHttpClient.execute()}. Retrying per call site has now been fixed three separate
+ * times ({@code createContainerCmd}, {@code startContainerCmd}, {@code ensureVolume}) while ~60
+ * other docker call sites stayed bare. {@link RetryingDockerHttpClient} moves the retry to the
+ * transport seam so every call site, present and future, is covered once.
+ *
+ * <p>The retry may only fire when replaying the request cannot change semantics: the request must
+ * be replayable (no one-shot body stream, no hijacked stdin) and must not be an exec-start, which
+ * would re-run a shell command whose first run may have executed.
+ */
+class RetryingDockerHttpClientTest {
+
+ private static final int MAX_ATTEMPTS = 3;
+
+ /** A delegate whose {@code execute} defers to {@code behaviour}, handed the 1-based attempt. */
+ private static final class FakeTransport implements DockerHttpClient {
+ final AtomicInteger calls = new AtomicInteger();
+ final List<Request> seenRequests = new ArrayList<>();
+ final IntFunction<Response> behaviour;
+
+ FakeTransport(IntFunction<Response> behaviour) {
+ this.behaviour = behaviour;
+ }
+
+ @Override
+ public Response execute(Request request) {
+ seenRequests.add(request);
+ return behaviour.apply(calls.incrementAndGet());
+ }
+
+ @Override
+ public void close() {
+ }
+ }
+
+ private static RuntimeException brokenPipe() {
+ return new RuntimeException(new IOException("Broken pipe"));
+ }
+
+ @Test
+ void retriesTransientBrokenPipeOnBodylessRequestThenSucceeds() {
+ Response ok = mock(Response.class);
+ FakeTransport delegate = new FakeTransport(attempt -> {
+ if (attempt < 3) {
+ throw brokenPipe();
+ }
+ return ok;
+ });
+ RetryingDockerHttpClient client = new RetryingDockerHttpClient(delegate, MAX_ATTEMPTS, 0L);
+
+ Request ping = Request.builder().method(Request.Method.GET).path("/_ping").build();
+
+ assertSame(ok, client.execute(ping));
+ assertEquals(3, delegate.calls.get(),
+ "a transient Broken pipe on a bodyless GET must be retried at the transport");
+ }
+
+ @Test
+ void retriesByteArrayBodiedRequestReplayingIdenticalBytes() {
+ Response ok = mock(Response.class);
+ FakeTransport delegate = new FakeTransport(attempt -> {
+ if (attempt == 1) {
+ throw brokenPipe();
+ }
+ return ok;
+ });
+ RetryingDockerHttpClient client = new RetryingDockerHttpClient(delegate, MAX_ATTEMPTS, 0L);
+
+ byte[] body = "{\"Image\":\"busybox\"}".getBytes(StandardCharsets.UTF_8);
+ Request create = Request.builder()
+ .method(Request.Method.POST)
+ .path("/containers/create?name=x")
+ .bodyBytes(body)
+ .build();
+
+ assertSame(ok, client.execute(create));
+ assertEquals(2, delegate.calls.get(),
+ "a POST whose body is a plain byte[] is replayable and must be retried");
+ // This is the replay-safety proof, not a formality: the transport builds a fresh entity
+ // from bodyBytes() on every execute(), so both attempts must see the very same bytes.
+ assertEquals(2, delegate.seenRequests.size());
+ assertEquals(Arrays.hashCode(delegate.seenRequests.get(0).bodyBytes()),
+ Arrays.hashCode(delegate.seenRequests.get(1).bodyBytes()),
+ "the retried attempt must carry byte-identical body content");
+ assertEquals("/containers/create?name=x", delegate.seenRequests.get(1).path());
+ }
+
+ @Test
+ void injectsCanonicalCloseWhilePreservingRequestDataAndHeaders() {
+ Response ok = mock(Response.class);
+ FakeTransport delegate = new FakeTransport(attempt -> ok);
+ RetryingDockerHttpClient client = new RetryingDockerHttpClient(delegate, MAX_ATTEMPTS, 0L);
+
+ byte[] body = "{\"Image\":\"busybox\"}".getBytes(StandardCharsets.UTF_8);
+ Request create = Request.builder()
+ .method(Request.Method.POST)
+ .path("/containers/create")
+ .bodyBytes(body)
+ .putHeader("Content-Type", "application/json")
+ .putHeader("connection", "keep-alive")
+ .build();
+
+ assertSame(ok, client.execute(create));
+ Request effective = delegate.seenRequests.get(0);
+ assertEquals(create.method(), effective.method());
+ assertEquals(create.path(), effective.path());
+ assertSame(body, effective.bodyBytes());
+ assertEquals("application/json", effective.headers().get("Content-Type"));
+ assertEquals("close", effective.headers().get("Connection"));
+ assertEquals(1, effective.headers().entrySet().stream()
+ .filter(entry -> "Connection".equalsIgnoreCase(entry.getKey()))
+ .count(), "the transport must receive one unambiguous Connection header");
+ }
+
+ @Test
+ void doesNotRetryOneShotStreamBody() {
+ FakeTransport delegate = new FakeTransport(attempt -> {
+ throw brokenPipe();
+ });
+ RetryingDockerHttpClient client = new RetryingDockerHttpClient(delegate, MAX_ATTEMPTS, 0L);
+
+ // The tar upload of copyArchiveToContainerCmd: bodyBytes is null, body() is a one-shot
+ // stream that the first (failed) attempt may have partially consumed. Replaying it would
+ // send a truncated archive, so the failure must surface after exactly one attempt.
+ InputStream tar = new ByteArrayInputStream(new byte[]{1, 2, 3});
+ Request putArchive = Request.builder()
+ .method(Request.Method.PUT)
+ .path("/containers/abc/archive")
+ .body(tar)
+ .build();
+
+ RuntimeException thrown = assertThrows(RuntimeException.class, () -> client.execute(putArchive));
+ assertEquals("Broken pipe", thrown.getCause().getMessage());
+ assertEquals(1, delegate.calls.get(),
+ "a one-shot stream body cannot be replayed; the transport must not retry it");
+ assertSame(tar, delegate.seenRequests.get(0).body(),
+ "adding Connection: close must preserve the one-shot stream object");
+ assertEquals("close", delegate.seenRequests.get(0).headers().get("Connection"));
+ }
+
+ @Test
+ void doesNotRetryHijackedRequest() {
+ FakeTransport delegate = new FakeTransport(attempt -> {
+ throw brokenPipe();
+ });
+ RetryingDockerHttpClient client = new RetryingDockerHttpClient(delegate, MAX_ATTEMPTS, 0L);
+
+ Request attach = Request.builder()
+ .method(Request.Method.POST)
+ .path("/containers/abc/attach")
+ .hijackedInput(new ByteArrayInputStream(new byte[0]))
+ .build();
+
+ assertThrows(RuntimeException.class, () -> client.execute(attach));
+ assertEquals(1, delegate.calls.get(),
+ "a hijacked (bidirectional stdin) request must never be replayed");
+ assertSame(attach, delegate.seenRequests.get(0),
+ "hijacked requests must reach docker-java unchanged for Connection: Upgrade");
+ assertEquals(Map.of(), delegate.seenRequests.get(0).headers());
+ }
+
+ @Test
+ void doesNotRetryExecPathsEvenWhenReplayable() {
+ // POST /exec/{id}/start carries a small JSON body, so by the replayable-body rule alone it
+ // would be retried, re-running a shell command whose first run may have executed. Exec
+ // paths are excluded outright.
+ FakeTransport startDelegate = new FakeTransport(attempt -> {
+ throw brokenPipe();
+ });
+ RetryingDockerHttpClient startClient =
+ new RetryingDockerHttpClient(startDelegate, MAX_ATTEMPTS, 0L);
+ Request execStart = Request.builder()
+ .method(Request.Method.POST)
+ .path("/exec/deadbeef/start")
+ .bodyBytes("{}".getBytes(StandardCharsets.UTF_8))
+ .build();
+
+ assertThrows(RuntimeException.class, () -> startClient.execute(execStart));
+ assertEquals(1, startDelegate.calls.get(),
+ "exec-start re-runs the command if replayed; it must surface after one attempt");
+ assertEquals("close", startDelegate.seenRequests.get(0).headers().get("Connection"),
+ "non-replayable exec control calls must still retire their connection");
+
+ // The exclusion is contains("/exec"), not startsWith: exec-create
+ // (POST /containers/{id}/exec) is also excluded, and this pins that breadth so a later
+ // refactor to startsWith cannot pass CI.
+ FakeTransport createDelegate = new FakeTransport(attempt -> {
+ throw brokenPipe();
+ });
+ RetryingDockerHttpClient createClient =
+ new RetryingDockerHttpClient(createDelegate, MAX_ATTEMPTS, 0L);
+ Request execCreate = Request.builder()
+ .method(Request.Method.POST)
+ .path("/containers/abc/exec")
+ .bodyBytes("{}".getBytes(StandardCharsets.UTF_8))
+ .build();
+
+ assertThrows(RuntimeException.class, () -> createClient.execute(execCreate));
+ assertEquals(1, createDelegate.calls.get(),
+ "exec-create must be excluded too, the /exec exclusion covers both spellings");
+ }
+
+ @Test
+ void doesNotRetryUnnamedContainerCreate() {
+ // Catches: replaying an unnamed containers/create whose first attempt landed, which
+ // creates a second orphaned container nobody can adopt.
+ FakeTransport delegate = new FakeTransport(attempt -> {
+ throw brokenPipe();
+ });
+ RetryingDockerHttpClient client = new RetryingDockerHttpClient(delegate, MAX_ATTEMPTS, 0L);
+ Request create = Request.builder()
+ .method(Request.Method.POST)
+ .path("/containers/create")
+ .bodyBytes("{}".getBytes(StandardCharsets.UTF_8))
+ .build();
+
+ RuntimeException thrown = assertThrows(RuntimeException.class, () -> client.execute(create));
+ assertEquals("Broken pipe", thrown.getCause().getMessage());
+ assertEquals(1, delegate.calls.get());
+ }
+
+ @Test
+ void doesNotRetryNetworkCreate() {
+ // Catches: replaying networks/create, where the replay gets a 409 that the VPC manager
+ // swallows, so the VPC silently loses its network.
+ FakeTransport delegate = new FakeTransport(attempt -> {
+ throw brokenPipe();
+ });
+ RetryingDockerHttpClient client = new RetryingDockerHttpClient(delegate, MAX_ATTEMPTS, 0L);
+ Request create = Request.builder()
+ .method(Request.Method.POST)
+ .path("/networks/create")
+ .bodyBytes("{}".getBytes(StandardCharsets.UTF_8))
+ .build();
+
+ RuntimeException thrown = assertThrows(RuntimeException.class, () -> client.execute(create));
+ assertEquals("Broken pipe", thrown.getCause().getMessage());
+ assertEquals(1, delegate.calls.get());
+ }
+
+ @Test
+ void doesNotRetryRestartOrNetworkConnectAndDisconnect() {
+ // Catches: replaying a bodyless restart whose first attempt landed, which restarts the
+ // container a second time, and replaying a network connect/disconnect, which the daemon
+ // answers with a conflict or not-found the caller never expected.
+ for (String path : List.of("/containers/abc/restart?t=10", "/networks/net1/connect",
+ "/networks/net1/disconnect")) {
+ FakeTransport delegate = new FakeTransport(attempt -> {
+ throw brokenPipe();
+ });
+ RetryingDockerHttpClient client = new RetryingDockerHttpClient(delegate, MAX_ATTEMPTS, 0L);
+ Request request = Request.builder()
+ .method(Request.Method.POST)
+ .path(path)
+ .build();
+
+ RuntimeException thrown = assertThrows(RuntimeException.class, () -> client.execute(request));
+ assertEquals("Broken pipe", thrown.getCause().getMessage());
+ assertEquals(1, delegate.calls.get(), path);
+ }
+ }
+
+ @Test
+ void retriesNamedContainerCreate() {
+ // Catches: over-broad exclusion that stops replaying named creates, which are safe
+ // because the caller adopts the container on a 409.
+ Response ok = mock(Response.class);
+ FakeTransport delegate = new FakeTransport(attempt -> {
+ if (attempt == 1) {
+ throw brokenPipe();
+ }
+ return ok;
+ });
+ RetryingDockerHttpClient client = new RetryingDockerHttpClient(delegate, MAX_ATTEMPTS, 0L);
+ Request create = Request.builder()
+ .method(Request.Method.POST)
+ .path("/containers/create?platform=linux%2Famd64&name=floci-x")
+ .bodyBytes("{}".getBytes(StandardCharsets.UTF_8))
+ .build();
+
+ assertSame(ok, client.execute(create));
+ assertEquals(2, delegate.calls.get());
+ }
+
+ @Test
+ void doesNotRetryNonTransientFailures() {
+ // Pool exhaustion: retrying re-enters another full connection-request wait and adds
+ // pressure to an already-starved pool. isTransientIo deliberately excludes it.
+ RuntimeException poolExhausted =
+ new RuntimeException(new ConnectionRequestTimeoutException("no free lease"));
+ FakeTransport exhaustedDelegate = new FakeTransport(attempt -> {
+ throw poolExhausted;
+ });
+ RetryingDockerHttpClient exhaustedClient =
+ new RetryingDockerHttpClient(exhaustedDelegate, MAX_ATTEMPTS, 0L);
+ Request ping = Request.builder().method(Request.Method.GET).path("/_ping").build();
+
+ RuntimeException thrown =
+ assertThrows(RuntimeException.class, () -> exhaustedClient.execute(ping));
+ assertSame(poolExhausted, thrown, "pool exhaustion must surface unchanged");
+ assertEquals(1, exhaustedDelegate.calls.get(), "pool exhaustion must not be retried");
+
+ // A genuine daemon rejection (e.g. a 409 name conflict raised above the transport as a
+ // DockerException) never clears on a retry either.
+ IllegalStateException rejection = new IllegalStateException("Conflict: name already in use");
+ FakeTransport rejectedDelegate = new FakeTransport(attempt -> {
+ throw rejection;
+ });
+ RetryingDockerHttpClient rejectedClient =
+ new RetryingDockerHttpClient(rejectedDelegate, MAX_ATTEMPTS, 0L);
+
+ IllegalStateException surfaced =
+ assertThrows(IllegalStateException.class, () -> rejectedClient.execute(ping));
+ assertSame(rejection, surfaced, "a daemon rejection must surface unchanged");
+ assertEquals(1, rejectedDelegate.calls.get());
+ }
+
+ @Test
+ void producerWrapsControlPlaneTransportOnly() {
+ DockerHttpClient raw = new FakeTransport(attempt -> null);
+
+ DockerHttpClient controlPlane = DockerClientProducer.wrapForRole(raw, "control-plane");
+ DockerHttpClient streaming = DockerClientProducer.wrapForRole(raw, "streaming");
+
+ assertSame(RetryingDockerHttpClient.class, controlPlane.getClass(),
+ "the control-plane transport must be wrapped in the retry decorator");
+ assertNotSame(raw, controlPlane);
+ assertSame(raw, streaming,
+ "the streaming transport (log-follow, exec output) must stay unwrapped");
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/core/common/docker/RetryingTarCopierTest.java b/src/test/java/io/github/hectorvent/floci/core/common/docker/RetryingTarCopierTest.java
new file mode 100644
index 0000000000..ef69eeb126
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/core/common/docker/RetryingTarCopierTest.java
@@ -0,0 +1,204 @@
+package io.github.hectorvent.floci.core.common.docker;
+
+import com.github.dockerjava.api.DockerClient;
+import com.github.dockerjava.api.command.CopyArchiveToContainerCmd;
+import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
+import org.apache.commons.compress.archivers.tar.TarArchiveInputStream;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.mockito.ArgumentCaptor;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.Path;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.RETURNS_SELF;
+import static org.mockito.Mockito.atLeastOnce;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+/**
+ * The transport-level retry ({@code RetryingDockerHttpClient}) must refuse to replay a one-shot
+ * {@code InputStream} body, the streamed tar bytes are gone after the first attempt. Every
+ * tar-copy into a container therefore needs its retry at the call site, rebuilding the whole
+ * operation (tar bytes or pipe, streamer thread, request) per attempt. This module is that call
+ * site, shared by every service that copies content into containers; tar extract over the same
+ * path is idempotent, so replay is safe.
+ */
+class RetryingTarCopierTest {
+
+ private static final int MAX_ATTEMPTS = 3;
+
+ private static RuntimeException brokenPipe() {
+ return new RuntimeException(new IOException("Broken pipe"));
+ }
+
+ private static CopyArchiveToContainerCmd failingOnce(DockerClient docker, AtomicInteger calls) {
+ CopyArchiveToContainerCmd cmd = mock(CopyArchiveToContainerCmd.class, RETURNS_SELF);
+ when(cmd.exec()).thenAnswer(inv -> {
+ if (calls.incrementAndGet() == 1) {
+ throw brokenPipe();
+ }
+ return null;
+ });
+ when(docker.copyArchiveToContainerCmd(any())).thenReturn(cmd);
+ return cmd;
+ }
+
+ @Test
+ void copyBytesRetriesWithAFreshStreamPerAttempt() {
+ DockerClient docker = mock(DockerClient.class);
+ AtomicInteger calls = new AtomicInteger();
+ CopyArchiveToContainerCmd cmd = failingOnce(docker, calls);
+
+ RetryingTarCopier.copyBytes(docker, "c-1", "/etc", "registries.yaml",
+ "mirrors: {}".getBytes(StandardCharsets.UTF_8), 0644, MAX_ATTEMPTS, 0L);
+
+ assertEquals(2, calls.get(), "a broken pipe on a byte copy is a socket blip, not a failure");
+ ArgumentCaptor<InputStream> streams = ArgumentCaptor.forClass(InputStream.class);
+ verify(cmd, atLeastOnce()).withTarInputStream(streams.capture());
+ assertEquals(2, streams.getAllValues().size(), "each attempt must send its own stream");
+ assertNotSame(streams.getAllValues().get(0), streams.getAllValues().get(1),
+ "replaying a consumed stream would upload zero bytes");
+ }
+
+ @Test
+ void copyBytesDoesNotRetryDaemonRejection() {
+ DockerClient docker = mock(DockerClient.class);
+ CopyArchiveToContainerCmd cmd = mock(CopyArchiveToContainerCmd.class, RETURNS_SELF);
+ RuntimeException noSuchContainer = new RuntimeException("No such container: c-1");
+ AtomicInteger calls = new AtomicInteger();
+ when(cmd.exec()).thenAnswer(inv -> {
+ calls.incrementAndGet();
+ throw noSuchContainer;
+ });
+ when(docker.copyArchiveToContainerCmd(any())).thenReturn(cmd);
+
+ RuntimeException thrown = assertThrows(RuntimeException.class,
+ () -> RetryingTarCopier.copyBytes(docker, "c-1", "/etc", "f", new byte[]{1},
+ 0644, MAX_ATTEMPTS, 0L));
+
+ assertSame(noSuchContainer, thrown.getCause(),
+ "a genuine daemon rejection must surface after a single attempt");
+ assertEquals(1, calls.get());
+ }
+
+ // copyFile streams tarWriter through a pipe on its own thread; if the writer fails before
+ // producing any bytes (e.g. the source file vanished), closing the pipe with nothing written
+ // looks like a clean, empty tar to a reader that isn't checking for that, the daemon accepts
+ // it and exec() returns normally, so the caller would otherwise see a false success instead
+ // of the real failure.
+ @Test
+ void copyFileFailsTheAttemptWhenTheSourceFileIsMissingInsteadOfReportingSuccess(@TempDir Path tempDir) {
+ DockerClient docker = mock(DockerClient.class);
+ CopyArchiveToContainerCmd cmd = mock(CopyArchiveToContainerCmd.class, RETURNS_SELF);
+ when(cmd.exec()).thenAnswer(inv -> null);
+ when(docker.copyArchiveToContainerCmd(any())).thenReturn(cmd);
+ Path missing = tempDir.resolve("does-not-exist");
+
+ RuntimeException thrown = assertThrows(RuntimeException.class,
+ () -> RetryingTarCopier.copyFile(docker, "c-1", "/etc", "f", missing, 0644, 1, 0L));
+
+ Throwable cause = thrown.getCause();
+ while (cause != null && !(cause instanceof NoSuchFileException)) {
+ cause = cause.getCause();
+ }
+ assertEquals(NoSuchFileException.class, cause == null ? null : cause.getClass(),
+ "the streamer's real failure must surface, not a phantom success");
+ }
+
+ @Test
+ void copyStreamedDoesNotRetryALocalWriterFailure() {
+ // Catches: a local tar-writer IOException (unreadable source file) being retried as a
+ // transient daemon blip, rebuilding the copy several times with backoff.
+ DockerClient docker = mock(DockerClient.class);
+ CopyArchiveToContainerCmd cmd = mock(CopyArchiveToContainerCmd.class, RETURNS_SELF);
+ when(cmd.exec()).thenAnswer(inv -> null);
+ when(docker.copyArchiveToContainerCmd(any())).thenReturn(cmd);
+ AtomicInteger writes = new AtomicInteger();
+
+ RuntimeException thrown = assertThrows(RuntimeException.class,
+ () -> RetryingTarCopier.copyStreamed(docker, "c-1", "/var/task", "fn-code",
+ out -> {
+ writes.incrementAndGet();
+ throw new IOException("Permission denied");
+ },
+ MAX_ATTEMPTS, 0L));
+
+ assertEquals(1, writes.get());
+ assertTrue(thrown.getMessage().contains("Permission denied"), thrown.getMessage());
+ }
+
+ @Test
+ void copyFileRebuildsThePipePerAttempt(@TempDir Path dir) throws IOException {
+ Path file = dir.resolve("bootstrap");
+ Files.writeString(file, "#!/bin/sh\n");
+ DockerClient docker = mock(DockerClient.class);
+ AtomicInteger calls = new AtomicInteger();
+ failingOnce(docker, calls);
+
+ RetryingTarCopier.copyFile(docker, "c-1", "/var/runtime", "bootstrap", file, 0755,
+ MAX_ATTEMPTS, 0L);
+
+ assertEquals(2, calls.get());
+ }
+
+ @Test
+ void copyStreamedInvokesTheWriterFreshlyPerAttempt() {
+ DockerClient docker = mock(DockerClient.class);
+ AtomicInteger calls = new AtomicInteger();
+ failingOnce(docker, calls);
+ AtomicInteger writes = new AtomicInteger();
+
+ RetryingTarCopier.copyStreamed(docker, "c-1", "/var/task", "fn-code",
+ out -> {
+ writes.incrementAndGet();
+ out.write("x".getBytes(StandardCharsets.UTF_8));
+ },
+ MAX_ATTEMPTS, 0L);
+
+ assertEquals(2, calls.get());
+ assertEquals(2, writes.get(), "the tar content must be regenerated for every attempt");
+ }
+
+ @Test
+ void copyHostResourceRetriesTransientBrokenPipe(@TempDir Path dir) throws IOException {
+ Path file = dir.resolve("kubeconfig.yaml");
+ Files.writeString(file, "apiVersion: v1");
+ DockerClient docker = mock(DockerClient.class);
+ AtomicInteger calls = new AtomicInteger();
+ failingOnce(docker, calls);
+
+ RetryingTarCopier.copyHostResource(docker, "c-1", "/var/lib/rancher", file.toString(),
+ MAX_ATTEMPTS, 0L);
+
+ assertEquals(2, calls.get());
+ }
+
+ @Test
+ void singleFileTarPreservesNameContentAndMode() throws IOException {
+ byte[] content = "ssh-ed25519 AAAA".getBytes(StandardCharsets.UTF_8);
+
+ byte[] tar = RetryingTarCopier.singleFileTar("authorized_keys", content, 0600);
+
+ try (TarArchiveInputStream in = new TarArchiveInputStream(new ByteArrayInputStream(tar))) {
+ TarArchiveEntry entry = in.getNextEntry();
+ assertEquals("authorized_keys", entry.getName());
+ assertEquals(0600, entry.getMode() & 07777, "an ssh key copied world-readable is a defect");
+ assertEquals(new String(content, StandardCharsets.UTF_8),
+ new String(in.readAllBytes(), StandardCharsets.UTF_8));
+ }
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambda/launcher/ContainerLauncherCopyRetryTest.java b/src/test/java/io/github/hectorvent/floci/services/lambda/launcher/ContainerLauncherCopyRetryTest.java
new file mode 100644
index 0000000000..559d772fe5
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/lambda/launcher/ContainerLauncherCopyRetryTest.java
@@ -0,0 +1,115 @@
+package io.github.hectorvent.floci.services.lambda.launcher;
+
+import com.github.dockerjava.api.DockerClient;
+import com.github.dockerjava.api.command.CopyArchiveToContainerCmd;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import io.github.hectorvent.floci.config.EmulatorConfig;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.RETURNS_SELF;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * The transport-level retry ({@code RetryingDockerHttpClient}) must refuse to replay a one-shot
+ * {@code InputStream} body, after the first attempt the streamed tar bytes are gone. That leaves
+ * the Lambda launcher's tar-copies (function code, layers, and the TLS CA cert) exposed to
+ * transient broken pipes: one blip while copying {@code floci-selfsigned.crt} failed an entire
+ * LZA Accounts stage with {@code Lambda.InitError}. The copy is idempotent (tar extract over the
+ * same path), and the whole operation, fresh pipe, fresh streamer thread, fresh request, can be
+ * rebuilt per attempt, so the retry lives at the call site.
+ */
+class ContainerLauncherCopyRetryTest {
+
+ private static final int MAX_ATTEMPTS = 3;
+
+ private static RuntimeException brokenPipe() {
+ return new RuntimeException(new IOException("Broken pipe"));
+ }
+
+ private ContainerLauncher launcher() {
+ // The constructor sizes its code-volume populate semaphore off the config, so a bare null
+ // no longer builds; the copy paths under test read nothing else from it.
+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
+ when(config.services().lambda().codeVolumePopulateConcurrency()).thenReturn(Optional.empty());
+ return new ContainerLauncher(null, null, null, null, null, null, config, null, null, null, null);
+ }
+
+ @Test
+ void copyFileToContainerRetriesTransientBrokenPipe(@TempDir Path dir) throws IOException {
+ Path cert = dir.resolve("floci-selfsigned.crt");
+ Files.writeString(cert, "-----BEGIN CERTIFICATE-----");
+ DockerClient docker = mock(DockerClient.class);
+ CopyArchiveToContainerCmd cmd = mock(CopyArchiveToContainerCmd.class, RETURNS_SELF);
+ AtomicInteger calls = new AtomicInteger();
+ when(cmd.exec()).thenAnswer(inv -> {
+ if (calls.incrementAndGet() == 1) {
+ throw brokenPipe();
+ }
+ return null;
+ });
+ when(docker.copyArchiveToContainerCmd(any())).thenReturn(cmd);
+
+ launcher().copyFileToContainer(docker, "container-1", cert, "/etc", "floci-ca.crt", "fn",
+ MAX_ATTEMPTS, 0L);
+
+ assertEquals(2, calls.get(),
+ "a broken pipe on the cert copy is a socket blip, not a launch failure");
+ }
+
+ @Test
+ void copyFileToContainerDoesNotRetryDaemonRejection(@TempDir Path dir) throws IOException {
+ Path cert = dir.resolve("floci-selfsigned.crt");
+ Files.writeString(cert, "-----BEGIN CERTIFICATE-----");
+ DockerClient docker = mock(DockerClient.class);
+ CopyArchiveToContainerCmd cmd = mock(CopyArchiveToContainerCmd.class, RETURNS_SELF);
+ RuntimeException noSuchContainer = new RuntimeException("No such container: container-1");
+ AtomicInteger calls = new AtomicInteger();
+ when(cmd.exec()).thenAnswer(inv -> {
+ calls.incrementAndGet();
+ throw noSuchContainer;
+ });
+ when(docker.copyArchiveToContainerCmd(any())).thenReturn(cmd);
+
+ RuntimeException thrown = assertThrows(RuntimeException.class,
+ () -> launcher().copyFileToContainer(docker, "container-1", cert, "/etc",
+ "floci-ca.crt", "fn", MAX_ATTEMPTS, 0L));
+
+ assertSame(noSuchContainer, thrown.getCause(),
+ "a genuine daemon rejection must surface after a single attempt");
+ assertEquals(1, calls.get());
+ }
+
+ @Test
+ void copyDirToContainerRetriesTransientBrokenPipe(@TempDir Path dir) throws IOException {
+ Files.writeString(dir.resolve("index.js"), "exports.handler = () => {};");
+ DockerClient docker = mock(DockerClient.class);
+ CopyArchiveToContainerCmd cmd = mock(CopyArchiveToContainerCmd.class, RETURNS_SELF);
+ AtomicInteger calls = new AtomicInteger();
+ when(cmd.exec()).thenAnswer(inv -> {
+ if (calls.incrementAndGet() == 1) {
+ throw brokenPipe();
+ }
+ return null;
+ });
+ when(docker.copyArchiveToContainerCmd(any())).thenReturn(cmd);
+
+ launcher().copyDirToContainer(docker, "container-1", dir, "/var/task", "fn",
+ MAX_ATTEMPTS, 0L);
+
+ assertEquals(2, calls.get(),
+ "each attempt rebuilds the pipe and streamer thread, so the dir copy may retry");
+ }
+}