Light, fluffy, and always free - The AWS Local Emulator alternative
feat(docker): named-volume storage for child containers (#743)
Hector Ventura01dbc76fc853
diff --git a/docker-compose.yml b/docker-compose.ymlindex d20f64816e..265e46d098 100644--- a/docker-compose.yml+++ b/docker-compose.yml@@ -14,7 +14,9 @@ services:environment:FLOCI_SERVICES_DOCKER_NETWORK: floci_defaultFLOCI_SERVICES_RDS_PROXY_BASE_PORT: "7001"- FLOCI_STORAGE_HOST_PERSISTENT_PATH: ${PWD}/data+ # FLOCI_STORAGE_HOST_PERSISTENT_PATH is no longer needed.+ # Floci now uses named Docker volumes by default.+ # If you were using this env var for persistence, migrate: docker volume ls -f label=floci=trueFLOCI_HOSTNAME: flociFLOCI_BASE_URL: http://floci:4566FLOCI_SERVICES_LAMBDA_HOT_RELOAD_ENABLED: "true"diff --git a/docs/configuration/docker-compose.md b/docs/configuration/docker-compose.mdindex a97b6809cd..a546dda116 100644--- a/docs/configuration/docker-compose.md+++ b/docs/configuration/docker-compose.md@@ -178,6 +178,8 @@ All `application.yml` options can be overridden via environment variables using| `FLOCI_DEFAULT_ACCOUNT_ID` | `000000000000` | AWS account ID used in ARNs || `FLOCI_STORAGE_MODE` | `memory` | Global storage mode (`memory`, `persistent`, `hybrid`, `wal`) || `FLOCI_STORAGE_PERSISTENT_PATH` | `./data` | Directory for persistent storage |+| `FLOCI_STORAGE_PRUNE_VOLUMES_ON_DELETE` | `false` | Remove named volumes immediately on resource delete (`true` in memory mode always) |+| `FLOCI_STORAGE_HOST_PERSISTENT_PATH` | _(none)_ | Absolute host path for container bind mounts (RDS, OpenSearch, MSK, ECR). When unset, Floci uses named Docker volumes automatically. || `FLOCI_DOCKER_DOCKER_HOST` | `unix:///var/run/docker.sock` | Docker daemon socket (shared by Lambda, RDS, ElastiCache) || `FLOCI_DOCKER_DOCKER_CONFIG_PATH` | `` | Path to dir with Docker's config.json (e.g. `/root/.docker`) || `FLOCI_DOCKER_REGISTRY_CREDENTIALS_0__SERVER` | `` | Registry hostname for explicit credential entry 0 |diff --git a/docs/configuration/storage.md b/docs/configuration/storage.mdindex 96b341a52a..6b859ea423 100644--- a/docs/configuration/storage.md+++ b/docs/configuration/storage.md@@ -70,9 +70,68 @@ Override the global mode for individual services via environment variables. When| `FLOCI_STORAGE_SERVICES_RDS_MODE` | global default | RDS storage mode (see note below) |!!! note "RDS storage mode"- For RDS, `FLOCI_STORAGE_SERVICES_RDS_MODE` controls whether DB containers get a persistent Docker volume.- When set to `memory` (or when the global mode is `memory`), no volume is created and data is lost when the container stops.- Any other mode (`hybrid`, `persistent`, `wal`) causes a per-instance named volume (`floci-rds-<id>`) to be created and removed when the instance is deleted.+ `FLOCI_STORAGE_SERVICES_RDS_MODE` controls Floci's own metadata persistence for RDS, not the+ DB container volumes. In all modes, each DB instance or cluster gets a named Docker volume+ (`floci-rds-{volumeId}`). In `memory` mode the volume is automatically removed when the+ instance is deleted. In other modes the volume is retained unless+ `FLOCI_STORAGE_PRUNE_VOLUMES_ON_DELETE=true`.++## Container Storage (RDS, OpenSearch, MSK, ECR)++Services that spawn Docker containers (RDS, OpenSearch, MSK, ECR registry) need a volume for their+data. Floci manages this automatically using **named Docker volumes** — no extra configuration+required.++### How it works++Each resource gets a `volumeId` (a 6-character hex string, e.g. `a1b2c3`) generated at creation+time and stored in the resource model. The container name and volume name both use this suffix:++```+floci-rds-a1b2c3 # RDS instance container and volume+floci-opensearch-b4c5d6 # OpenSearch domain container and volume+floci-msk-e7f8a9 # MSK cluster container and volume+floci-ecr-registry-data # ECR shared registry volume (singleton)+```++Volumes are labelled `floci=true` so you can manage them with standard Docker commands:++```bash+# List all Floci-managed volumes+docker volume ls --filter label=floci=true++# Remove all Floci-managed volumes (destructive)+docker volume prune --filter label=floci=true+```++### Volume lifecycle++By default volumes survive resource deletion, matching real AWS behavior.++| `storage.mode` | Volume on resource delete |+|---|---|+| `memory` | **Always removed** — memory mode implies no persistence across restarts |+| `persistent` / `hybrid` / `wal` | Retained (default) — remove with `FLOCI_STORAGE_PRUNE_VOLUMES_ON_DELETE=true` |++```bash+# Remove named volumes immediately when a resource is deleted (useful in CI with persistent mode)+FLOCI_STORAGE_PRUNE_VOLUMES_ON_DELETE=true+```++### Host-path mode (advanced)++Set `FLOCI_STORAGE_HOST_PERSISTENT_PATH` to an **absolute host path** to use bind mounts instead+of named volumes. This is only needed when you must access the container data directly from the+host filesystem.++```bash+FLOCI_STORAGE_HOST_PERSISTENT_PATH=/absolute/host/path/data+```++!!! warning+ `FLOCI_STORAGE_HOST_PERSISTENT_PATH` must be an absolute path (starting with `/`). Volume+ names and relative paths are not supported and will be silently ignored, falling back to+ named-volume mode.## Environment Variable Overridediff --git a/docs/services/msk.md b/docs/services/msk.mdindex dff7050ae6..a0866abe26 100644--- a/docs/services/msk.md+++ b/docs/services/msk.md@@ -34,7 +34,7 @@ floci:When `mock` is set to `false` (default), Floci uses the Docker API to start a Redpanda container for each created cluster. For Docker socket setup, private registry authentication, and other Docker settings see [Docker Configuration](../configuration/docker.md).- **Port Mapping**: The Kafka API (9092) is mapped to a dynamic host port.-- **Persistence**: Data is stored in the Floci persistent path under `msk/<cluster-name>`.+- **Persistence**: Each cluster gets a named Docker volume (`floci-msk-{volumeId}`). In memory mode the volume is removed on cluster delete; in persistent modes it is retained unless `FLOCI_STORAGE_PRUNE_VOLUMES_ON_DELETE=true`.- **Readiness**: The cluster state transitions to `ACTIVE` once the Redpanda `/ready` endpoint is reachable.## Examplesdiff --git a/docs/services/opensearch.md b/docs/services/opensearch.mdindex 8426738947..2dc66d36a5 100644--- a/docs/services/opensearch.md+++ b/docs/services/opensearch.md@@ -134,7 +134,7 @@ services:- **Engine version default:** `OpenSearch_2.11`- **Supported engine versions:** `OpenSearch_2.13`, `OpenSearch_2.11`, `OpenSearch_2.9`, `OpenSearch_2.7`, `OpenSearch_2.5`, `OpenSearch_2.3`, `OpenSearch_1.3`, `OpenSearch_1.2`, `Elasticsearch_7.10`, `Elasticsearch_7.9`, `Elasticsearch_7.8`- **Cluster defaults:** `m5.large.search`, 1 instance, EBS enabled with 10 GiB `gp2` volume.-- **Data path:** `{floci.storage.persistent-path}/opensearch/{domainName}` — follows the shared storage root.+- **Container storage:** each domain gets a named Docker volume (`floci-opensearch-{volumeId}`) created automatically. In memory mode the volume is removed on domain delete; in persistent modes it is retained unless `FLOCI_STORAGE_PRUNE_VOLUMES_ON_DELETE=true`.## Examplesdiff --git a/docs/services/rds.md b/docs/services/rds.mdindex ccd6bf532a..5a3e763f54 100644--- a/docs/services/rds.md+++ b/docs/services/rds.md@@ -107,21 +107,34 @@ Override the image per-instance with the `--engine-version` flag or globally via## Persistence-By default, each DB instance or cluster gets its own named Docker volume (`floci-rds-<id>`). The volume is created when the instance is created and removed when the instance is deleted.+Each DB instance and cluster gets its own named Docker volume (`floci-rds-{volumeId}`) created+automatically. No configuration is required.-Set `FLOCI_STORAGE_SERVICES_RDS_MODE=memory` (or set the global `FLOCI_STORAGE_MODE=memory`) to disable volume creation entirely — DB containers become ephemeral and data is lost on restart. This is the recommended setting for CI.+| Scenario | Volume behavior |+|---|---|+| `memory` mode (default) | Volume is removed automatically when the instance is deleted |+| `persistent` / `hybrid` / `wal` | Volume is retained after delete — data survives for manual recovery |```bash-# CI — no volumes, fastest startup-FLOCI_STORAGE_SERVICES_RDS_MODE=memory+# CI — ephemeral, volumes cleaned up on each delete+FLOCI_STORAGE_MODE=memory++# Local dev — retain DB data across Floci restarts+FLOCI_STORAGE_MODE=hybrid++# Local dev — also remove volumes immediately on delete+FLOCI_STORAGE_MODE=hybrid+FLOCI_STORAGE_PRUNE_VOLUMES_ON_DELETE=true+```++To use a host bind mount instead of a named volume (advanced), set an absolute path:-# Local dev — persist DB data across Floci restarts-FLOCI_STORAGE_SERVICES_RDS_MODE=hybrid+```bashFLOCI_STORAGE_HOST_PERSISTENT_PATH=/absolute/host/path/data```!!! note "Docker Desktop on macOS"- Floci uses named Docker volumes (not bind mounts) for RDS persistence. This works correctly on Docker Desktop for macOS where bind-mounting paths inside the Floci container is not supported.+ Named volumes work correctly on Docker Desktop for macOS. Bind mounts to paths inside the Floci container are not supported — use named volumes (the default).## Authenticationdiff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.javaindex 130eda17e9..f21bb1091a 100644--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java@@ -95,6 +95,14 @@ public interface EmulatorConfig {@WithDefault("${floci.storage.persistent-path}")String hostPersistentPath();+ /**+ * When {@code true}, named volumes are removed immediately after a child container stops+ * on resource delete. In {@code memory} storage mode volumes are always removed regardless+ * of this flag. Defaults to {@code false} to match real AWS behaviour (data survives delete).+ */+ @WithDefault("false")+ boolean pruneVolumesOnDelete();+WalConfig wal();ServiceStorageOverrides services();diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.javaindex 9e521a0ed3..2cdc59a4cb 100644--- a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java@@ -174,6 +174,21 @@ public class ContainerLifecycleManager {}}+ /**+ * Creates a named volume if it does not already exist. Idempotent — safe to call on every+ * container start. Labels the volume {@code floci=true} so+ * {@code docker volume prune --filter label=floci} works.+ */+ public void ensureVolume(String volumeName) {+ if (!volumeExists(volumeName)) {+ dockerClient.createVolumeCmd()+ .withName(volumeName)+ .withLabels(Map.of("floci", "true"))+ .exec();+ LOG.debugv("Created volume {0}", volumeName);+ }+ }+/*** Removes a named Docker volume, ignoring errors if it does not exist or is still in use.*/diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerStorageHelper.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerStorageHelper.javanew file mode 100644index 0000000000..83c23085c3--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerStorageHelper.java@@ -0,0 +1,97 @@+package io.github.hectorvent.floci.core.common.docker;++import io.github.hectorvent.floci.config.EmulatorConfig;+import org.jboss.logging.Logger;++import java.io.IOException;+import java.nio.file.Files;+import java.nio.file.Path;++/**+ * Central helper for child-container volume management across RDS, OpenSearch, MSK, and ECR.+ *+ * <p>Two modes:+ * <ul>+ * <li>Named-volume (default) — Floci manages per-resource Docker named volumes labelled+ * {@code floci=true}. Active when {@code FLOCI_STORAGE_HOST_PERSISTENT_PATH} is not set.</li>+ * <li>Host-path (legacy) — active when {@code FLOCI_STORAGE_HOST_PERSISTENT_PATH} is set;+ * callers fall through to their existing bind-mount logic.</li>+ * </ul>+ */+public final class ContainerStorageHelper {++ private static final Logger LOG = Logger.getLogger(ContainerStorageHelper.class);++ private ContainerStorageHelper() {}++ /**+ * Canonical container/volume name for a resource. Uses {@code volumeId} when set;+ * falls back to {@code fallbackId} (the resource name) for resources created before+ * this change.+ */+ public static String resourceName(String service, String volumeId, String fallbackId) {+ return "floci-" + service + "-" + (volumeId != null ? volumeId : fallbackId);+ }++ /**+ * Returns {@code true} when named-volume mode is active.+ * Returns {@code false} only when {@code FLOCI_STORAGE_HOST_PERSISTENT_PATH} is set to+ * an absolute path, indicating the caller should use a host bind-mount instead.+ * Volume names and relative paths are not supported in {@code host-persistent-path} —+ * they are treated as named-volume mode.+ */+ public static boolean isNamedVolumeMode(EmulatorConfig config) {+ return !config.storage().hostPersistentPath().startsWith("/");+ }++ /**+ * Ensures the named volume exists and mounts it to {@code internalMount} in the container.+ * Must only be called when {@link #isNamedVolumeMode} returns {@code true}.+ */+ public static void applyStorage(+ ContainerBuilder.Builder builder,+ ContainerLifecycleManager lifecycleManager,+ String service,+ String volumeId,+ String fallbackId,+ String internalMount) {+ String volumeName = resourceName(service, volumeId, fallbackId);+ lifecycleManager.ensureVolume(volumeName);+ builder.withNamedVolume(volumeName, internalMount);+ }++ /**+ * Removes the named volume on resource delete, honouring the configured prune policy.+ *+ * <ul>+ * <li>In {@code memory} storage mode: always removes (data cannot survive a restart anyway).</li>+ * <li>In persistent modes: removes only when {@code prune-volumes-on-delete: true}.</li>+ * </ul>+ */+ public static void removeStorage(+ EmulatorConfig config,+ ContainerLifecycleManager lifecycleManager,+ String service,+ String volumeId,+ String fallbackId) {+ String volumeName = resourceName(service, volumeId, fallbackId);+ boolean isMemory = "memory".equals(config.storage().mode());+ if (isMemory || config.storage().pruneVolumesOnDelete()) {+ lifecycleManager.removeVolume(volumeName);+ } else {+ LOG.infov("Retained Docker volume {0}. Remove manually: docker volume rm {0}", volumeName);+ }+ }++ /**+ * Ensures the host data directory exists for host-path mode (absolute paths only).+ * Called by managers in their legacy host-path code paths.+ */+ public static void ensureHostDir(String hostDataPath) {+ try {+ Files.createDirectories(Path.of(hostDataPath));+ } catch (IOException e) {+ LOG.errorv("Failed to create data directory {0}: {1}", hostDataPath, e.getMessage());+ }+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/ecr/registry/EcrRegistryManager.java b/src/main/java/io/github/hectorvent/floci/services/ecr/registry/EcrRegistryManager.javaindex 61c0670b5f..ebe1374c02 100644--- a/src/main/java/io/github/hectorvent/floci/services/ecr/registry/EcrRegistryManager.java+++ b/src/main/java/io/github/hectorvent/floci/services/ecr/registry/EcrRegistryManager.java@@ -8,6 +8,7 @@ import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager.ContainerInfo;import io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;import io.github.hectorvent.floci.core.common.docker.ContainerSpec;+import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;import io.github.hectorvent.floci.core.common.docker.PortAllocator;import com.github.dockerjava.api.DockerClient;import com.github.dockerjava.api.async.ResultCallback;@@ -178,37 +179,23 @@ public class EcrRegistryManager {}private void addPersistenceMounts(ContainerBuilder.Builder specBuilder, List<String> env) {- String hostPersistentPath = config.storage().hostPersistentPath();- boolean inContainer = containerDetector.isRunningInContainer();- boolean isExplicitVolume = lifecycleManager.volumeExists(hostPersistentPath);- boolean isRelativeDefault = hostPersistentPath.startsWith(".");-- if (inContainer && isRelativeDefault) {- // Zero-config fallback for in-container Floci+ if (ContainerStorageHelper.isNamedVolumeMode(config)) {+ lifecycleManager.ensureVolume(NAMED_VOLUME);specBuilder.withNamedVolume(NAMED_VOLUME, "/var/lib/registry");- LOG.infov("Floci in container with relative host-persistent-path ({0}); "- + "using named volume {1} for ECR registry data",- hostPersistentPath, NAMED_VOLUME);- } else if (isExplicitVolume) {- // User set hostPersistentPath to a Docker named-volume name- String internalMountPath = "/app/data";- specBuilder.withNamedVolume(hostPersistentPath, internalMountPath);- env.add("REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=" + internalMountPath + "/ecr/registry");- } else {- // Host path bind-mount.- // normalize() eliminates any "./" segments produced by toAbsolutePath() on a- // relative config path (e.g. "./data/ecr" → "/app/./data/ecr") so the- // replace() matches reliably.- String dataPath = Paths.get(config.services().ecr().dataPath(), "registry")- .toAbsolutePath().normalize().toString();- String persistentPath = Paths.get(config.storage().persistentPath())- .toAbsolutePath().normalize().toString();- String hostDataPath = dataPath.replace(persistentPath, hostPersistentPath);- if (!inContainer) {- ensureDataDir();- }- specBuilder.withBind(hostDataPath, "/var/lib/registry");+ return;+ }++ // Legacy host-path mode: host-persistent-path is an absolute path+ boolean inContainer = containerDetector.isRunningInContainer();+ String dataPath = Paths.get(config.services().ecr().dataPath(), "registry")+ .toAbsolutePath().normalize().toString();+ String persistentPath = Paths.get(config.storage().persistentPath())+ .toAbsolutePath().normalize().toString();+ String hostDataPath = dataPath.replace(persistentPath, config.storage().hostPersistentPath());+ if (!inContainer) {+ ensureDataDir();}+ specBuilder.withBind(hostDataPath, "/var/lib/registry");}private void attachLogStream() {diff --git a/src/main/java/io/github/hectorvent/floci/services/msk/MskService.java b/src/main/java/io/github/hectorvent/floci/services/msk/MskService.javaindex dce843a13a..2e1be3b39c 100644--- a/src/main/java/io/github/hectorvent/floci/services/msk/MskService.java+++ b/src/main/java/io/github/hectorvent/floci/services/msk/MskService.java@@ -14,6 +14,7 @@ import jakarta.enterprise.context.ApplicationScoped;import jakarta.inject.Inject;import org.jboss.logging.Logger;+import java.security.SecureRandom;import java.util.ArrayList;import java.util.List;import java.util.Map;@@ -61,6 +62,7 @@ public class MskService {String clusterArn = AwsArnUtils.Arn.of("kafka", "us-east-1", "000000000000", "cluster/" + clusterName + "/" + java.util.UUID.randomUUID()).toString();MskCluster cluster = new MskCluster(clusterArn, clusterName);+ cluster.setVolumeId(String.format("%06x", new SecureRandom().nextInt(0xFFFFFF)));if (config.services().msk().mock()) {cluster.setState(ClusterState.ACTIVE);@@ -89,6 +91,7 @@ public class MskService {cluster.setState(ClusterState.DELETING);if (!config.services().msk().mock()) {redpandaManager.stopContainer(cluster);+ redpandaManager.removeClusterStorage(cluster);}storage.delete(clusterArn);}diff --git a/src/main/java/io/github/hectorvent/floci/services/msk/RedpandaManager.java b/src/main/java/io/github/hectorvent/floci/services/msk/RedpandaManager.javaindex cfcdbecfac..d49fe2ef64 100644--- a/src/main/java/io/github/hectorvent/floci/services/msk/RedpandaManager.java+++ b/src/main/java/io/github/hectorvent/floci/services/msk/RedpandaManager.java@@ -9,16 +9,15 @@ import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager.Cimport io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager.EndpointInfo;import io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;import io.github.hectorvent.floci.core.common.docker.ContainerSpec;+import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;import io.github.hectorvent.floci.services.msk.model.MskCluster;import jakarta.enterprise.context.ApplicationScoped;import jakarta.inject.Inject;import org.jboss.logging.Logger;import java.io.Closeable;-import java.io.IOException;import java.net.HttpURLConnection;import java.net.URI;-import java.nio.file.Files;import java.nio.file.Path;import java.util.ArrayList;import java.util.List;@@ -61,14 +60,6 @@ public class RedpandaManager {String containerName = "floci-msk-" + cluster.getClusterName();- // Ensure internal persistence path exists- Path dataPath = Path.of(config.storage().persistentPath(), "msk", cluster.getClusterName());- try {- Files.createDirectories(dataPath);- } catch (IOException e) {- LOG.errorv("Failed to create MSK data directory: {0}", dataPath, e);- }-// Cleanup stale containerlifecycleManager.removeIfExists(containerName);@@ -92,19 +83,15 @@ public class RedpandaManager {}// Handle persistence mounting- String hostPersistentPath = config.storage().hostPersistentPath();- boolean isVolume = !hostPersistentPath.startsWith("/") && !hostPersistentPath.startsWith(".");-- if (isVolume) {- // Volume mode: mount the whole volume and point Redpanda to the subdirectory- String internalMountPath = "/app/data";- specBuilder.withBind(hostPersistentPath, internalMountPath);- cmd.add("--data-dir");- cmd.add(internalMountPath + "/msk/" + cluster.getClusterName());+ if (ContainerStorageHelper.isNamedVolumeMode(config)) {+ ContainerStorageHelper.applyStorage(specBuilder, lifecycleManager,+ "msk", cluster.getVolumeId(), cluster.getClusterName(),+ "/var/lib/redpanda/data");} else {- // Directory mode: mount the specific subdirectory directly- String hostDataPath = Path.of(hostPersistentPath, "msk", cluster.getClusterName())+ // Legacy host-path mode: host-persistent-path is an absolute path+ String hostDataPath = Path.of(config.storage().hostPersistentPath(), "msk", cluster.getClusterName()).toAbsolutePath().toString();+ ContainerStorageHelper.ensureHostDir(hostDataPath);specBuilder.withBind(hostDataPath, "/var/lib/redpanda/data");}@@ -181,4 +168,9 @@ public class RedpandaManager {lifecycleManager.stopAndRemove(cluster.getContainerId(), logHandle);LOG.infov("Redpanda container {0} stopped and removed", cluster.getContainerId());}++ public void removeClusterStorage(MskCluster cluster) {+ ContainerStorageHelper.removeStorage(config, lifecycleManager,+ "msk", cluster.getVolumeId(), cluster.getClusterName());+ }}diff --git a/src/main/java/io/github/hectorvent/floci/services/msk/model/MskCluster.java b/src/main/java/io/github/hectorvent/floci/services/msk/model/MskCluster.javaindex c2ccddd2e1..36835f694b 100644--- a/src/main/java/io/github/hectorvent/floci/services/msk/model/MskCluster.java+++ b/src/main/java/io/github/hectorvent/floci/services/msk/model/MskCluster.java@@ -38,6 +38,9 @@ public class MskCluster {// Docker container ID for mock=falseprivate String containerId;+ // 6-char hex generated once at creation for stable, collision-free volume/container naming+ private String volumeId;+public MskCluster() {}public MskCluster(String clusterArn, String clusterName) {@@ -79,4 +82,7 @@ public class MskCluster {public String getContainerId() { return containerId; }public void setContainerId(String containerId) { this.containerId = containerId; }++ public String getVolumeId() { return volumeId; }+ public void setVolumeId(String volumeId) { this.volumeId = volumeId; }}diff --git a/src/main/java/io/github/hectorvent/floci/services/opensearch/OpenSearchDomainManager.java b/src/main/java/io/github/hectorvent/floci/services/opensearch/OpenSearchDomainManager.javaindex df6415ca96..d2cb99f36b 100644--- a/src/main/java/io/github/hectorvent/floci/services/opensearch/OpenSearchDomainManager.java+++ b/src/main/java/io/github/hectorvent/floci/services/opensearch/OpenSearchDomainManager.java@@ -6,16 +6,15 @@ import io.github.hectorvent.floci.core.common.docker.ContainerDetector;import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager.ContainerInfo;import io.github.hectorvent.floci.core.common.docker.ContainerSpec;+import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;import io.github.hectorvent.floci.core.common.docker.PortAllocator;import io.github.hectorvent.floci.services.opensearch.model.Domain;import jakarta.enterprise.context.ApplicationScoped;import jakarta.inject.Inject;import org.jboss.logging.Logger;-import java.io.IOException;import java.net.HttpURLConnection;import java.net.URI;-import java.nio.file.Files;import java.nio.file.Path;/**@@ -58,34 +57,31 @@ public class OpenSearchDomainManager {config.services().opensearch().proxyBasePort(),config.services().opensearch().proxyMaxPort());- Path dataPath = Path.of(config.services().opensearch().dataPath(), domain.getDomainName());- try {- Files.createDirectories(dataPath);- } catch (IOException e) {- LOG.warnv("Could not create OpenSearch data directory {0}: {1}", dataPath, e.getMessage());- }-lifecycleManager.removeIfExists(containerName);- String hostDataPath;- String hostPersistentPath = config.storage().hostPersistentPath();- if (hostPersistentPath.startsWith("/")) {- String dataPathStr = dataPath.toAbsolutePath().normalize().toString();- String persistentPathStr = Path.of(config.storage().persistentPath()).toAbsolutePath().normalize().toString();- hostDataPath = dataPathStr.replace(persistentPathStr, hostPersistentPath);- } else {- hostDataPath = "floci-opensearch-" + domain.getDomainName();- }-- ContainerSpec spec = containerBuilder.newContainer(image)+ ContainerBuilder.Builder specBuilder = containerBuilder.newContainer(image).withName(containerName).withEnv("discovery.type", "single-node").withEnv("DISABLE_SECURITY_PLUGIN", "true").withPortBinding(OPENSEARCH_PORT, hostPort)- .withBind(hostDataPath, "/usr/share/opensearch/data").withDockerNetwork(config.services().dockerNetwork())- .withLogRotation()- .build();+ .withLogRotation();++ if (ContainerStorageHelper.isNamedVolumeMode(config)) {+ ContainerStorageHelper.applyStorage(specBuilder, lifecycleManager,+ "opensearch", domain.getVolumeId(), domain.getDomainName(),+ "/usr/share/opensearch/data");+ } else {+ // Legacy host-path mode: host-persistent-path is an absolute path+ Path dataPath = Path.of(config.services().opensearch().dataPath(), domain.getDomainName());+ ContainerStorageHelper.ensureHostDir(dataPath.toString());+ String dataPathStr = dataPath.toAbsolutePath().normalize().toString();+ String persistentPathStr = Path.of(config.storage().persistentPath()).toAbsolutePath().normalize().toString();+ String hostDataPath = dataPathStr.replace(persistentPathStr, config.storage().hostPersistentPath());+ specBuilder.withBind(hostDataPath, "/usr/share/opensearch/data");+ }++ ContainerSpec spec = specBuilder.build();ContainerInfo info = lifecycleManager.createAndStart(spec);domain.setContainerId(info.containerId());@@ -134,4 +130,9 @@ public class OpenSearchDomainManager {lifecycleManager.stopAndRemove(domain.getContainerId(), null);LOG.infov("Stopped OpenSearch container for domain {0}", domain.getDomainName());}++ public void removeDomainStorage(Domain domain) {+ ContainerStorageHelper.removeStorage(config, lifecycleManager,+ "opensearch", domain.getVolumeId(), domain.getDomainName());+ }}diff --git a/src/main/java/io/github/hectorvent/floci/services/opensearch/OpenSearchService.java b/src/main/java/io/github/hectorvent/floci/services/opensearch/OpenSearchService.javaindex 8510d45702..4df5894713 100644--- a/src/main/java/io/github/hectorvent/floci/services/opensearch/OpenSearchService.java+++ b/src/main/java/io/github/hectorvent/floci/services/opensearch/OpenSearchService.java@@ -15,6 +15,7 @@ import jakarta.enterprise.context.ApplicationScoped;import jakarta.inject.Inject;import org.jboss.logging.Logger;+import java.security.SecureRandom;import java.time.Instant;import java.util.List;import java.util.Map;@@ -86,6 +87,7 @@ public class OpenSearchService {domain.setDeleted(false);domain.setEndpoint("");domain.setCreatedAt(Instant.now());+ domain.setVolumeId(String.format("%06x", new SecureRandom().nextInt(0xFFFFFF)));if (clusterConfig != null) {domain.setClusterConfig(clusterConfig);@@ -170,6 +172,7 @@ public class OpenSearchService {domain.setDeleted(true);if (!config.services().opensearch().mock()) {domainManager.stopDomain(domain);+ domainManager.removeDomainStorage(domain);}domainStore.delete(domainName);LOG.infov("Deleted OpenSearch domain: {0}", domainName);diff --git a/src/main/java/io/github/hectorvent/floci/services/opensearch/model/Domain.java b/src/main/java/io/github/hectorvent/floci/services/opensearch/model/Domain.javaindex d3277be78e..64fafa51e7 100644--- a/src/main/java/io/github/hectorvent/floci/services/opensearch/model/Domain.java+++ b/src/main/java/io/github/hectorvent/floci/services/opensearch/model/Domain.java@@ -46,6 +46,9 @@ public class Domain {@JsonProperty("ContainerId")private String containerId;+ @JsonProperty("VolumeId")+ private String volumeId;+@JsonProperty("CreatedAt")@JsonFormat(shape = JsonFormat.Shape.NUMBER)private Instant createdAt;@@ -140,6 +143,14 @@ public class Domain {this.containerId = containerId;}+ public String getVolumeId() {+ return volumeId;+ }++ public void setVolumeId(String volumeId) {+ this.volumeId = volumeId;+ }+public Instant getCreatedAt() {return createdAt;}diff --git a/src/main/java/io/github/hectorvent/floci/services/rds/RdsService.java b/src/main/java/io/github/hectorvent/floci/services/rds/RdsService.javaindex 1bd1d3b37f..f9ac7e48ce 100644--- a/src/main/java/io/github/hectorvent/floci/services/rds/RdsService.java+++ b/src/main/java/io/github/hectorvent/floci/services/rds/RdsService.java@@ -3,7 +3,6 @@ package io.github.hectorvent.floci.services.rds;import io.github.hectorvent.floci.config.EmulatorConfig;import io.github.hectorvent.floci.core.common.AwsException;import io.github.hectorvent.floci.core.common.RegionResolver;-import io.github.hectorvent.floci.core.common.ServiceConfigAccess;import io.github.hectorvent.floci.core.storage.InMemoryStorage;import io.github.hectorvent.floci.core.storage.StorageBackend;import io.github.hectorvent.floci.services.rds.container.RdsContainerHandle;@@ -19,6 +18,7 @@ import jakarta.enterprise.context.ApplicationScoped;import jakarta.inject.Inject;import org.jboss.logging.Logger;+import java.security.SecureRandom;import java.time.Instant;import java.util.ArrayList;import java.util.Collection;@@ -42,20 +42,17 @@ public class RdsService {private final RdsProxyManager proxyManager;private final RegionResolver regionResolver;private final EmulatorConfig config;- private final ServiceConfigAccess serviceConfigAccess;private final Set<Integer> usedPorts = ConcurrentHashMap.newKeySet();@Injectpublic RdsService(RdsContainerManager containerManager,RdsProxyManager proxyManager,RegionResolver regionResolver,- EmulatorConfig config,- ServiceConfigAccess serviceConfigAccess) {+ EmulatorConfig config) {this.containerManager = containerManager;this.proxyManager = proxyManager;this.regionResolver = regionResolver;this.config = config;- this.serviceConfigAccess = serviceConfigAccess;this.instances = new InMemoryStorage<>();this.clusters = new InMemoryStorage<>();this.parameterGroups = new InMemoryStorage<>();@@ -81,6 +78,7 @@ public class RdsService {String containerId = null;String containerHost = null;int containerPort = 0;+ String instanceVolumeId = null;if (dbClusterIdentifier != null && !dbClusterIdentifier.isBlank()) {// Cluster member — share the cluster's container@@ -95,7 +93,8 @@ public class RdsService {} else {// Standalone instance — start its own containerString image = imageForEngine(engine);- RdsContainerHandle handle = containerManager.start(id, engine, image, masterUsername, masterPassword, dbName);+ instanceVolumeId = String.format("%06x", new SecureRandom().nextInt(0xFFFFFF));+ RdsContainerHandle handle = containerManager.start(id, instanceVolumeId, engine, image, masterUsername, masterPassword, dbName);backendHost = handle.getHost();backendPort = handle.getPort();containerId = handle.getContainerId();@@ -110,9 +109,7 @@ public class RdsService {instance.setContainerId(containerId);instance.setContainerHost(containerHost);instance.setContainerPort(containerPort);- if (dbClusterIdentifier == null || dbClusterIdentifier.isBlank()) {- instance.setDockerVolumeName(isInMemory() ? null : "floci-rds-" + id);- }+ instance.setVolumeId(instanceVolumeId);String region = regionResolver.getDefaultRegion();instance.setDbiResourceId("db-" + java.util.UUID.randomUUID().toString().replace("-", "").substring(0, 24).toUpperCase());@@ -180,7 +177,7 @@ public class RdsService {LOG.warnv("Error stopping container during reboot of {0}: {1}", id, e.getMessage());}String image = imageForEngine(instance.getEngine());- RdsContainerHandle handle = containerManager.start(id, instance.getEngine(), image,+ RdsContainerHandle handle = containerManager.start(id, instance.getVolumeId(), instance.getEngine(), image,instance.getMasterUsername(), instance.getMasterPassword(), instance.getDbName());instance.setContainerId(handle.getContainerId());instance.setContainerHost(handle.getHost());@@ -222,9 +219,7 @@ public class RdsService {if (instance.getContainerId() != null) {containerManager.stop(buildHandle(instance));}- if (instance.getDockerVolumeName() != null) {- containerManager.removeVolume(instance.getDbInstanceIdentifier());- }+ containerManager.removeVolume(instance.getDbInstanceIdentifier(), instance.getVolumeId());} else {// Cluster member — remove from cluster's member listDbCluster cluster = clusters.get(clusterId).orElse(null);@@ -253,8 +248,9 @@ public class RdsService {DatabaseEngine engine = resolveEngine(engineParam);int proxyPort = allocateProxyPort();String image = imageForEngine(engine);+ String clusterVolumeId = String.format("%06x", new SecureRandom().nextInt(0xFFFFFF));- RdsContainerHandle handle = containerManager.start(id, engine, image, masterUsername, masterPassword, databaseName);+ RdsContainerHandle handle = containerManager.start(id, clusterVolumeId, engine, image, masterUsername, masterPassword, databaseName);DbEndpoint endpoint = new DbEndpoint("localhost", proxyPort);DbCluster cluster = new DbCluster(id, engine, engineVersion, masterUsername, masterPassword,@@ -263,7 +259,7 @@ public class RdsService {cluster.setContainerId(handle.getContainerId());cluster.setContainerHost(handle.getHost());cluster.setContainerPort(handle.getPort());- cluster.setDockerVolumeName(isInMemory() ? null : "floci-rds-" + id);+ cluster.setVolumeId(clusterVolumeId);String region = regionResolver.getDefaultRegion();cluster.setDbClusterResourceId("cluster-" + java.util.UUID.randomUUID().toString().replace("-", "").substring(0, 24).toUpperCase());@@ -323,9 +319,7 @@ public class RdsService {if (cluster.getContainerId() != null) {containerManager.stop(buildClusterHandle(cluster));}- if (cluster.getDockerVolumeName() != null) {- containerManager.removeVolume(id);- }+ containerManager.removeVolume(id, cluster.getVolumeId());releaseProxyPort(cluster.getProxyPort());clusters.delete(id);@@ -401,10 +395,6 @@ public class RdsService {// ── Helpers ───────────────────────────────────────────────────────────────- private boolean isInMemory() {- return "memory".equals(serviceConfigAccess.storageMode("rds"));- }-private DatabaseEngine resolveEngine(String engineParam) {if (engineParam == null) {return DatabaseEngine.POSTGRES;diff --git a/src/main/java/io/github/hectorvent/floci/services/rds/container/RdsContainerManager.java b/src/main/java/io/github/hectorvent/floci/services/rds/container/RdsContainerManager.javaindex c33512a52a..28fe9bc4ff 100644--- a/src/main/java/io/github/hectorvent/floci/services/rds/container/RdsContainerManager.java+++ b/src/main/java/io/github/hectorvent/floci/services/rds/container/RdsContainerManager.java@@ -6,6 +6,7 @@ import io.github.hectorvent.floci.core.common.ServiceConfigAccess;import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;import io.github.hectorvent.floci.core.common.docker.ContainerDetector;import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;+import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager.ContainerInfo;import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager.EndpointInfo;import io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;@@ -59,7 +60,7 @@ public class RdsContainerManager {this.serviceConfigAccess = serviceConfigAccess;}- public RdsContainerHandle start(String instanceId, DatabaseEngine engine,+ public RdsContainerHandle start(String instanceId, String volumeId, DatabaseEngine engine,String image, String masterUsername,String masterPassword, String dbName) {LOG.infov("Starting RDS backend container for instance: {0} engine={1}", instanceId, engine);@@ -89,7 +90,7 @@ public class RdsContainerManager {}// Handle persistence mounting- addPersistenceMounts(specBuilder, instanceId, engine, envVars);+ addPersistenceMounts(specBuilder, instanceId, volumeId, engine, envVars);// Add engine-specific commandList<String> cmd = buildContainerCmd(engine);@@ -143,51 +144,18 @@ public class RdsContainerManager {}private void addPersistenceMounts(ContainerBuilder.Builder specBuilder, String instanceId,- DatabaseEngine engine, List<String> envVars) {- if ("memory".equals(serviceConfigAccess.storageMode("rds"))) {- return; // ephemeral mode — no mount needed- }-- String hostPersistentPath = config.storage().hostPersistentPath();-- if (!hostPersistentPath.startsWith("/") && !hostPersistentPath.startsWith(".")) {- // Explicit Docker named volume (e.g. "floci-data"): mount the shared volume- String internalMountPath = "/app/data";- specBuilder.withNamedVolume(hostPersistentPath, internalMountPath);- String dataPath = internalMountPath + "/rds/" + instanceId;- applyEngineDataPath(specBuilder, engine, envVars, instanceId, dataPath);-- } else if (hostPersistentPath.startsWith("/")) {- // Absolute bind-mount path: user explicitly configured this host path- String hostDataPath = Path.of(hostPersistentPath, "rds", instanceId).toString();- try {- Files.createDirectories(Path.of(hostDataPath));- } catch (IOException e) {- LOG.errorv("Failed to create RDS data directory: {0}", hostDataPath, e);- }- specBuilder.withBind(hostDataPath, engineDefaultDataPath(engine));-- } else {- // Default (relative path like ./data): use a per-instance Docker named volume.- // This works on all platforms including Docker Desktop on macOS, where- // bind-mounting paths inside the Floci container is not allowed.- String volumeName = "floci-rds-" + instanceId;- specBuilder.withNamedVolume(volumeName, engineDefaultDataPath(engine));- if (engine == DatabaseEngine.POSTGRES) {- envVars.add("PGDATA=/var/lib/postgresql/data");- }+ String volumeId, DatabaseEngine engine, List<String> envVars) {+ if (ContainerStorageHelper.isNamedVolumeMode(config)) {+ ContainerStorageHelper.applyStorage(+ specBuilder, lifecycleManager, "rds", volumeId, instanceId,+ engineDefaultDataPath(engine));+ return;}- }- private void applyEngineDataPath(ContainerBuilder.Builder specBuilder, DatabaseEngine engine,- List<String> envVars, String instanceId, String dataPath) {- if (engine == DatabaseEngine.POSTGRES) {- envVars.add("PGDATA=" + dataPath);- } else {- // MySQL / MariaDB: use a dedicated per-instance named volume- String volumeName = "floci-rds-" + instanceId;- specBuilder.withNamedVolume(volumeName, engineDefaultDataPath(engine));- }+ // Legacy host-path mode: host-persistent-path is an absolute path+ String hostDataPath = Path.of(config.storage().hostPersistentPath(), "rds", instanceId).toString();+ ContainerStorageHelper.ensureHostDir(hostDataPath);+ specBuilder.withBind(hostDataPath, engineDefaultDataPath(engine));}private static String engineDefaultDataPath(DatabaseEngine engine) {@@ -197,14 +165,11 @@ public class RdsContainerManager {};}- public void removeVolume(String instanceId) {- String volumeName = "floci-rds-" + instanceId;- try {- lifecycleManager.getDockerClient().removeVolumeCmd(volumeName).exec();- LOG.infov("Removed Docker volume {0} for RDS instance {1}", volumeName, instanceId);- } catch (Exception e) {- LOG.warnv("Could not remove Docker volume {0}: {1}", volumeName, e.getMessage());+ public void removeVolume(String instanceId, String volumeId) {+ if (ContainerStorageHelper.isNamedVolumeMode(config)) {+ ContainerStorageHelper.removeStorage(config, lifecycleManager, "rds", volumeId, instanceId);}+ // host-path mode: host directories are not removed automatically}private List<String> buildEnvVars(DatabaseEngine engine, String masterUsername,diff --git a/src/main/java/io/github/hectorvent/floci/services/rds/model/DbCluster.java b/src/main/java/io/github/hectorvent/floci/services/rds/model/DbCluster.javaindex 91cc5962b6..8d0cd7a101 100644--- a/src/main/java/io/github/hectorvent/floci/services/rds/model/DbCluster.java+++ b/src/main/java/io/github/hectorvent/floci/services/rds/model/DbCluster.java@@ -26,7 +26,8 @@ public class DbCluster {private Instant createdAt;private int proxyPort;- private String dockerVolumeName; // null in in-memory mode; "floci-rds-<id>" otherwise+ private String dockerVolumeName;+ private String volumeId;// Transient — not persistedprivate transient String containerId;@@ -109,6 +110,9 @@ public class DbCluster {public String getDockerVolumeName() { return dockerVolumeName; }public void setDockerVolumeName(String dockerVolumeName) { this.dockerVolumeName = dockerVolumeName; }+ public String getVolumeId() { return volumeId; }+ public void setVolumeId(String volumeId) { this.volumeId = volumeId; }+public String getContainerId() { return containerId; }public void setContainerId(String containerId) { this.containerId = containerId; }diff --git a/src/main/java/io/github/hectorvent/floci/services/rds/model/DbInstance.java b/src/main/java/io/github/hectorvent/floci/services/rds/model/DbInstance.javaindex 04377be0d0..d57e08aece 100644--- a/src/main/java/io/github/hectorvent/floci/services/rds/model/DbInstance.java+++ b/src/main/java/io/github/hectorvent/floci/services/rds/model/DbInstance.java@@ -26,6 +26,7 @@ public class DbInstance {private int proxyPort;private String dockerVolumeName; // null in in-memory mode; "floci-rds-<id>" otherwise+ private String volumeId; // 6-char hex, generated once at creation; null on older persisted resources// Transient — not persisted; restored on startup by re-launching containersprivate transient String containerId;@@ -113,6 +114,9 @@ public class DbInstance {public String getDockerVolumeName() { return dockerVolumeName; }public void setDockerVolumeName(String dockerVolumeName) { this.dockerVolumeName = dockerVolumeName; }+ public String getVolumeId() { return volumeId; }+ public void setVolumeId(String volumeId) { this.volumeId = volumeId; }+public String getContainerId() { return containerId; }public void setContainerId(String containerId) { this.containerId = containerId; }diff --git a/src/main/resources/application.yml b/src/main/resources/application.ymlindex 1022d66127..11c689fd8e 100644--- a/src/main/resources/application.yml+++ b/src/main/resources/application.yml@@ -52,6 +52,7 @@ floci:mode: memorypersistent-path: ./datahost-persistent-path: ./data+ prune-volumes-on-delete: falsewal:compaction-interval-ms: 30000services:diff --git a/src/test/java/io/github/hectorvent/floci/services/rds/RdsServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/rds/RdsServiceTest.javaindex d444685c5c..c81ba3d5dc 100644--- a/src/test/java/io/github/hectorvent/floci/services/rds/RdsServiceTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/rds/RdsServiceTest.java@@ -3,7 +3,6 @@ package io.github.hectorvent.floci.services.rds;import io.github.hectorvent.floci.config.EmulatorConfig;import io.github.hectorvent.floci.core.common.AwsException;import io.github.hectorvent.floci.core.common.RegionResolver;-import io.github.hectorvent.floci.core.common.ServiceConfigAccess;import io.github.hectorvent.floci.services.rds.model.DatabaseEngine;import io.github.hectorvent.floci.services.rds.model.DbCluster;import io.github.hectorvent.floci.services.rds.container.RdsContainerHandle;@@ -43,11 +42,9 @@ class RdsServiceTest {when(rdsConfig.proxyBasePort()).thenReturn(7000);when(rdsConfig.proxyMaxPort()).thenReturn(7099);- ServiceConfigAccess serviceConfigAccess = mock(ServiceConfigAccess.class);- when(serviceConfigAccess.storageMode("rds")).thenReturn("in-memory");- rdsService = new RdsService(containerManager, proxyManager, regionResolver, config, serviceConfigAccess);+ rdsService = new RdsService(containerManager, proxyManager, regionResolver, config);- when(containerManager.start(any(), any(), any(), any(), any(), any()))+ when(containerManager.start(any(), any(), any(), any(), any(), any(), any())).thenReturn(new RdsContainerHandle("cont-id", "id", "localhost", 5432));}diff --git a/src/test/resources/application.yml b/src/test/resources/application.ymlindex e4006591b4..b5802f2bd4 100644--- a/src/test/resources/application.yml+++ b/src/test/resources/application.yml@@ -12,6 +12,7 @@ floci:storage:mode: memorypersistent-path: ./target/test-data+ prune-volumes-on-delete: truewal:compaction-interval-ms: 60000services: