Light, fluffy, and always free - The AWS Local Emulator alternative
fix(cloudwatchlogs): journal log events instead of rewriting the store under persistent mode (#4187)
Niels Kootstraa0cb86cc133f
diff --git a/docs/configuration/advanced/application-yml.md b/docs/configuration/advanced/application-yml.mdindex ac6cf27628..c108c91494 100644--- a/docs/configuration/advanced/application-yml.md+++ b/docs/configuration/advanced/application-yml.md@@ -55,6 +55,8 @@ floci:# mount-user: "1001:1001" # PosixUser: run mounting containers as uid[:gid]# mount-group-add: 2000 # supplementary gid added to mounting containerswal:+ # Also the cadence at which journaled stores under persistent mode (CloudWatch Logs events)+ # fold their .wal file into the store's JSON file.compaction-interval-ms: 30000services:ssm:diff --git a/docs/configuration/environment-variables.md b/docs/configuration/environment-variables.mdindex c17becfdbd..742ed0073e 100644--- a/docs/configuration/environment-variables.md+++ b/docs/configuration/environment-variables.md@@ -81,7 +81,7 @@ See [TLS / HTTPS](./tls.md) for SDK configuration examples and WebSocket (`wss:/| `FLOCI_STORAGE_PERSISTENT_PATH` | `./data` | Container-side directory for persistent and hybrid storage || `FLOCI_STORAGE_HOST_PERSISTENT_PATH` | `./data` | Host-side path for Docker volume bind-mounts (RDS, OpenSearch, MSK, ECR data). When unset, Floci uses named Docker volumes || `FLOCI_STORAGE_PRUNE_VOLUMES_ON_DELETE` | `false` | Remove named Docker volumes immediately when the resource is deleted |-| `FLOCI_STORAGE_WAL_COMPACTION_INTERVAL_MS` | `30000` | How often (ms) the WAL compaction runs. Only applies when `FLOCI_STORAGE_MODE=wal` |+| `FLOCI_STORAGE_WAL_COMPACTION_INTERVAL_MS` | `30000` | How often (ms) the WAL compaction runs. Applies to `wal` mode and to the stores that are journaled under `persistent` mode (CloudWatch Logs events) |### Per-service storage overridesdiff --git a/docs/configuration/storage.md b/docs/configuration/storage.mdindex 240905da0f..88cb62d636 100644--- a/docs/configuration/storage.md+++ b/docs/configuration/storage.md@@ -7,7 +7,7 @@ Floci supports four storage backends. You can set a global default and override| Mode | Data survives restart | Write performance | Use case ||---|---|---|---|| `memory` | No | Fastest | Unit tests, CI pipelines |-| `persistent` | Yes | Synchronous disk write on every change | Development with durable state |+| `persistent` | Yes | Synchronous disk write on every change; append-heavy stores are journaled (see below) | Development with durable state || `hybrid` | Yes | In-memory reads, async flush to disk | General local development || `wal` | Yes | Append-only write-ahead log with compaction | High-write workloads |@@ -22,6 +22,18 @@ Floci supports four storage backends. You can set a global default and override!!! note "Code default vs shipped default"The Java `@WithDefault` for `storage.mode` is `hybrid`, but the published Docker image ships with `memory` set in `application.yml`. Running the stock image gives you `memory` unless you set `FLOCI_STORAGE_MODE`.+### Journaled stores under `persistent` mode++Most stores under `persistent` mode are rewritten in full on every change, which keeps the file+current after every call but makes the cost of one write grow with the size of the store.+Append-heavy stores are journaled instead: a change is appended to a `.wal` file next to the+store, and the store's JSON file is rewritten from memory on the `FLOCI_STORAGE_WAL_COMPACTION_INTERVAL_MS`+cadence and at shutdown, and only when something changed. Today this applies to CloudWatch Logs+events (`cwlogs-events.json` with `cwlogs-events.wal`). After a clean shutdown the JSON file holds+every event; while Floci runs it can be up to one compaction interval behind, and the journal is+replayed on the next start. An existing `cwlogs-events.json` from an older version is picked up as+the first snapshot without any migration.+## Per-Service OverrideWhen not set for a service, it inherits `FLOCI_STORAGE_MODE`. Only override when you need different behavior for a specific service.@@ -39,7 +51,7 @@ When not set for a service, it inherits `FLOCI_STORAGE_MODE`. Only override when| `FLOCI_STORAGE_SERVICES_LAMBDA_MODE` | global default | Lambda storage mode || `FLOCI_STORAGE_SERVICES_LAMBDA_FLUSH_INTERVAL_MS` | `5000` | Lambda flush interval (ms) || `FLOCI_STORAGE_SERVICES_CLOUDWATCHLOGS_MODE` | global default | CloudWatch Logs storage mode |-| `FLOCI_STORAGE_SERVICES_CLOUDWATCHLOGS_FLUSH_INTERVAL_MS` | `5000` | CloudWatch Logs flush interval (ms) |+| `FLOCI_STORAGE_SERVICES_CLOUDWATCHLOGS_FLUSH_INTERVAL_MS` | `15000` | CloudWatch Logs flush interval (ms) || `FLOCI_STORAGE_SERVICES_CLOUDWATCHMETRICS_MODE` | global default | CloudWatch Metrics storage mode || `FLOCI_STORAGE_SERVICES_CLOUDWATCHMETRICS_FLUSH_INTERVAL_MS` | `5000` | CloudWatch Metrics flush interval (ms) || `FLOCI_STORAGE_SERVICES_SECRETSMANAGER_MODE` | global default | Secrets Manager storage mode |diff --git a/docs/services/cloudwatch.md b/docs/services/cloudwatch.mdindex e9e1b61e56..10ed9dd3f1 100644--- a/docs/services/cloudwatch.md+++ b/docs/services/cloudwatch.md@@ -243,6 +243,18 @@ given rather than by omitting it, which is what its SDK paginators expect.| `FLOCI_SERVICES_CLOUDWATCHLOGS_MAX_EVENTS_PER_QUERY` | `10000` | Maximum events returned per `FilterLogEvents` / `GetLogEvents` call, and the upper bound for a Logs Insights `limit` || `FLOCI_SERVICES_CLOUDWATCHLOGS_QUERY_COMPLETION_DELAY_MS` | `0` | Artificial Logs Insights query delay. With `0` a query completes immediately. A positive value emulates the real asynchronous lifecycle (`Running` → `Complete` after the delay), which also makes `StopQuery` on a still-running query return `success=true` |+### Storage++Log groups, streams, filters and resource policies follow the configured storage mode as every+other service does. The event store is the one CloudWatch Logs store under steady append load, so+under `persistent` mode it is journaled instead of rewritten on every `PutLogEvents` call: a+batch is appended to `cwlogs-events.wal`, and `cwlogs-events.json` is rewritten from memory on+the `FLOCI_STORAGE_WAL_COMPACTION_INTERVAL_MS` cadence and at shutdown. The store keeps at most+20,000 events per account (`FLOCI_SERVICES_CLOUDWATCHLOGS_MAX_STORED_EVENTS`), which bounds both+the memory footprint and the size of each snapshot. `hybrid` remains the alternative when a+bounded delay for every CloudWatch Logs store is acceptable. See+[Storage Modes](../configuration/storage.md#journaled-stores-under-persistent-mode).+### Examples```bashdiff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.javaindex d9fea803d0..b4a6547ede 100644--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java@@ -1663,9 +1663,10 @@ public interface EmulatorConfig {int maxEventsPerQuery();/**- * Upper bound on log events kept across all groups. The store is one JSON document rewritten- * in full on every flush, so an unbounded store turns a chatty or retrying Lambda into a- * sustained multi-hundred-MB/s disk writer. Oldest events are evicted first once exceeded.+ * Upper bound on log events kept across all groups. The store is compacted into one JSON+ * document on every flush (under persistent mode it is journaled in between), so an+ * unbounded store turns a chatty or retrying Lambda into a sustained multi-hundred-MB/s+ * disk writer. Oldest events are evicted first once exceeded.*/@WithDefault("20000")int maxStoredEvents();diff --git a/src/main/java/io/github/hectorvent/floci/core/storage/StorageFactory.java b/src/main/java/io/github/hectorvent/floci/core/storage/StorageFactory.javaindex a8750030b3..5bbf043764 100644--- a/src/main/java/io/github/hectorvent/floci/core/storage/StorageFactory.java+++ b/src/main/java/io/github/hectorvent/floci/core/storage/StorageFactory.java@@ -63,6 +63,24 @@ public class StorageFactory {*/public synchronized <V> AccountAwareStorageBackend<V> create(String serviceName, String fileName,TypeReference<Map<String, V>> typeReference) {+ return create(serviceName, fileName, typeReference, WriteProfile.DEFAULT);+ }++ /**+ * Create an account-aware storage backend for the given service with a write profile.+ * The profile only matters under {@code persistent} mode: an {@link WriteProfile#APPEND_HEAVY}+ * store is journaled to {@code <file>.wal} and its JSON file becomes the snapshot that+ * compaction rewrites, instead of the file being rewritten on every mutation. The first+ * {@code create()} for a path decides the profile; repeat calls reuse that backend.+ *+ * @param serviceName the service name (ssm, sqs, s3, …)+ * @param fileName the JSON file name for persistent storage+ * @param typeReference Jackson type reference for deserialization+ * @param profile how the store is written+ */+ public synchronized <V> AccountAwareStorageBackend<V> create(String serviceName, String fileName,+ TypeReference<Map<String, V>> typeReference,+ WriteProfile profile) {String mode = resolveMode(serviceName);long flushInterval = resolveFlushInterval(serviceName);Path basePath = Path.of(config.storage().persistentPath());@@ -83,7 +101,21 @@ public class StorageFactory {StorageBackend<String, V> inner = switch (mode) {case "memory" -> new InMemoryStorage<>();- case "persistent" -> new PersistentStorage<>(filePath, typeReference);+ case "persistent" -> {+ if (profile != WriteProfile.APPEND_HEAVY) {+ yield new PersistentStorage<>(filePath, typeReference);+ }+ // The store file doubles as the WAL snapshot: a file written by the plain+ // persistent backend is the first snapshot, and after a clean shutdown the file+ // is current again. Only the .wal file next to it is new.+ Path walFilePath = basePath.resolve(fileName.replace(".json", ".wal"));+ WalStorage<String, V> wal = new WalStorage<>(filePath, walFilePath, typeReference,+ config.storage().wal().compactionIntervalMs());+ walBackends.add(wal);+ LOG.infov("Journaling {0} for service {1} to {2}; the store file is rewritten on the WAL compaction interval",+ filePath, serviceName, walFilePath);+ yield wal;+ }case "hybrid" -> {var hybrid = new HybridStorage<>(filePath, typeReference, flushInterval);hybridBackends.add(hybrid);diff --git a/src/main/java/io/github/hectorvent/floci/core/storage/WalStorage.java b/src/main/java/io/github/hectorvent/floci/core/storage/WalStorage.javaindex f632207854..316374a826 100644--- a/src/main/java/io/github/hectorvent/floci/core/storage/WalStorage.java+++ b/src/main/java/io/github/hectorvent/floci/core/storage/WalStorage.java@@ -1,5 +1,6 @@package io.github.hectorvent.floci.core.storage;+import com.fasterxml.jackson.core.JsonProcessingException;import com.fasterxml.jackson.core.type.TypeReference;import com.fasterxml.jackson.databind.ObjectMapper;import com.fasterxml.jackson.databind.SerializationFeature;@@ -8,9 +9,11 @@ import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;import org.jboss.logging.Logger;import java.io.*;+import java.nio.channels.FileChannel;import java.nio.file.Files;import java.nio.file.Path;import java.nio.file.StandardCopyOption;+import java.nio.file.StandardOpenOption;import java.util.ArrayList;import java.util.Collections;import java.util.List;@@ -24,7 +27,8 @@ import java.util.stream.Collectors;/*** Write-Ahead Log storage: in-memory reads with append-only binary WAL for durability.- * Periodic compaction writes a full snapshot and truncates the WAL.+ * Periodic compaction writes a full snapshot and truncates the WAL, and is skipped while+ * nothing has changed since the last snapshot.* On startup: load snapshot, then replay WAL entries after snapshot.** Binary WAL entry format:@@ -50,6 +54,9 @@ public class WalStorage<K, V> implements StorageBackend<K, V> {private final ReentrantReadWriteLock compactionLock = new ReentrantReadWriteLock();private final ScheduledExecutorService scheduler;private volatile DataOutputStream walWriter;+ // Set under the read lock by every mutation and cleared under the write lock by compaction,+ // so an idle store does not rewrite its snapshot on every compaction tick.+ private volatile boolean dirty;public WalStorage(Path snapshotPath, Path walPath, TypeReference<Map<K, V>> typeReference,long compactionIntervalMs) {@@ -80,12 +87,32 @@ public class WalStorage<K, V> implements StorageBackend<K, V> {compactionLock.readLock().lock();try {store.put(key, value);+ dirty = true;appendPut(key, value);} finally {compactionLock.readLock().unlock();}}+ /**+ * Appends the whole batch as one contiguous run of WAL records with a single flush, so a+ * batch costs one write instead of one per entry. An empty batch touches nothing.+ */+ @Override+ public void putAll(Map<K, V> entries) {+ if (entries.isEmpty()) {+ return;+ }+ compactionLock.readLock().lock();+ try {+ store.putAll(entries);+ dirty = true;+ appendPuts(entries);+ } finally {+ compactionLock.readLock().unlock();+ }+ }+@Overridepublic Optional<V> get(K key) {return Optional.ofNullable(store.get(key));@@ -96,6 +123,7 @@ public class WalStorage<K, V> implements StorageBackend<K, V> {compactionLock.readLock().lock();try {store.remove(key);+ dirty = true;appendDelete(key);} finally {compactionLock.readLock().unlock();@@ -129,13 +157,17 @@ public class WalStorage<K, V> implements StorageBackend<K, V> {store.putAll(data);LOG.infov("Loaded {0} entries from snapshot {1}", store.size(), snapshotPath);} catch (IOException e) {- LOG.errorv(e, "Failed to load snapshot from {0}", snapshotPath);+ StorageQuarantine.quarantine(snapshotPath, e, LOG);}}if (Files.exists(walPath)) {int replayed = replayWal();LOG.infov("Replayed {0} WAL entries from {1}", replayed, walPath);+ if (replayed > 0) {+ // The snapshot is behind the log; the next compaction folds the log in.+ dirty = true;+ }}openWalWriter();@@ -150,8 +182,11 @@ public class WalStorage<K, V> implements StorageBackend<K, V> {try {Files.deleteIfExists(walPath);Files.deleteIfExists(snapshotPath);+ dirty = false;} catch (IOException e) {LOG.errorv(e, "Failed to delete WAL/snapshot files");+ // A snapshot that survived must be overwritten by the next compaction.+ dirty = true;}openWalWriter();} finally {@@ -176,11 +211,15 @@ public class WalStorage<K, V> implements StorageBackend<K, V> {private void compact() {compactionLock.writeLock().lock();try {+ if (!dirty) {+ return;+ }Files.createDirectories(snapshotPath.getParent());Path tempFile = snapshotPath.resolveSibling(snapshotPath.getFileName() + ".tmp");snapshotMapper.writeValue(tempFile.toFile(), store);Files.move(tempFile, snapshotPath, StandardCopyOption.REPLACE_EXISTING,StandardCopyOption.ATOMIC_MOVE);+ dirty = false;closeWalWriter();Files.deleteIfExists(walPath);@@ -201,11 +240,7 @@ public class WalStorage<K, V> implements StorageBackend<K, V> {byte[] keyBytes = walMapper.writeValueAsBytes(key);byte[] valueBytes = walMapper.writeValueAsBytes(value);synchronized (out) {- out.writeByte(OP_PUT);- out.writeInt(keyBytes.length);- out.write(keyBytes);- out.writeInt(valueBytes.length);- out.write(valueBytes);+ writePutRecord(out, keyBytes, valueBytes);out.flush();}} catch (IOException e) {@@ -213,6 +248,38 @@ public class WalStorage<K, V> implements StorageBackend<K, V> {}}+ private void appendPuts(Map<K, V> entries) {+ DataOutputStream out = walWriter;+ if (out == null) {+ return;+ }+ try {+ // Encode the whole batch off the shared writer so the lock is held for one write.+ ByteArrayOutputStream batch = new ByteArrayOutputStream();+ DataOutputStream records = new DataOutputStream(batch);+ for (Map.Entry<K, V> entry : entries.entrySet()) {+ writePutRecord(records, walMapper.writeValueAsBytes(entry.getKey()),+ walMapper.writeValueAsBytes(entry.getValue()));+ }+ records.flush();+ synchronized (out) {+ batch.writeTo(out);+ out.flush();+ }+ } catch (IOException e) {+ LOG.errorv(e, "Failed to append {0} PUT WAL entries", entries.size());+ }+ }++ private static void writePutRecord(DataOutputStream out, byte[] keyBytes, byte[] valueBytes)+ throws IOException {+ out.writeByte(OP_PUT);+ out.writeInt(keyBytes.length);+ out.write(keyBytes);+ out.writeInt(valueBytes.length);+ out.write(valueBytes);+ }+private void appendDelete(K key) {DataOutputStream out = walWriter;if (out == null) return;@@ -229,56 +296,102 @@ public class WalStorage<K, V> implements StorageBackend<K, V> {}}+ /**+ * Replays the log on top of the snapshot and returns the number of records applied.+ * <p>+ * Replay stops at the first record that cannot be framed or decoded: a tail torn by a crash,+ * a length header that is negative or points past the end of the file, an unknown op byte, or+ * a payload CBOR cannot read. The store is never failed by such a record; the bytes from that+ * record onwards are cut off before the writer reopens, so records appended afterwards sit+ * behind a valid record and are replayed on the next load instead of being lost behind the+ * garbage. A read error that is not a framing problem leaves the file alone.+ */@SuppressWarnings("unchecked")private int replayWal() {int replayed = 0;+ long fileSize;+ try {+ fileSize = Files.size(walPath);+ } catch (IOException e) {+ LOG.errorv(e, "Failed to read the size of WAL {0}", walPath);+ return 0;+ }+ long consumed = 0;+ boolean cutTail = true;try (DataInputStream in = new DataInputStream(new BufferedInputStream(Files.newInputStream(walPath)))) {- while (true) {- int op;- try {- op = in.readByte();- } catch (EOFException e) {+ while (consumed < fileSize) {+ int op = in.readByte();+ byte[] keyBytes = readFrame(in, fileSize - consumed - 1);+ if (keyBytes == null) {break;}-- int keyLen = in.readInt();- byte[] keyBytes = in.readNBytes(keyLen);- if (keyBytes.length < keyLen) break; // truncated entry-+ long recordLength = 1 + 4 + keyBytes.length;if (op == OP_PUT) {- int valueLen = in.readInt();- byte[] valueBytes = in.readNBytes(valueLen);- if (valueBytes.length < valueLen) break; // truncated entry-+ byte[] valueBytes = readFrame(in, fileSize - consumed - recordLength);+ if (valueBytes == null) {+ break;+ }+ recordLength += 4 + valueBytes.length;K key = (K) walMapper.readValue(keyBytes, Object.class);V value = walMapper.readValue(valueBytes,walMapper.constructType(typeReference.getType()).getContentType());store.put(key, value);- replayed++;} else if (op == OP_DELETE) {K key = (K) walMapper.readValue(keyBytes, Object.class);store.remove(key);- replayed++;} else {- LOG.errorv("Unknown WAL op byte: {0}, stopping replay", op);+ LOG.warnv("Unknown WAL op byte {0} at offset {1} in {2}, stopping replay",+ op, consumed, walPath);break;}+ consumed += recordLength;+ replayed++;}+ } catch (EOFException | JsonProcessingException e) {+ LOG.warnv("WAL {0} is unreadable from offset {1}: {2}", walPath, consumed, e.getMessage());} catch (IOException e) {+ cutTail = false;LOG.errorv(e, "Failed to replay WAL from {0} (replayed {1} entries before error)",walPath, replayed);}+ if (cutTail && consumed < fileSize) {+ cutWalTail(consumed, fileSize);+ }return replayed;}+ /**+ * Reads one length-prefixed frame, or returns {@code null} when the length is negative or+ * larger than the bytes left in the file, which marks the record as torn or corrupt.+ */+ private static byte[] readFrame(DataInputStream in, long bytesLeft) throws IOException {+ int length = in.readInt();+ if (length < 0 || length > bytesLeft - 4) {+ return null;+ }+ byte[] bytes = in.readNBytes(length);+ return bytes.length == length ? bytes : null;+ }++ private void cutWalTail(long lastGoodOffset, long fileSize) {+ try (FileChannel channel = FileChannel.open(walPath, StandardOpenOption.WRITE)) {+ channel.truncate(lastGoodOffset);+ LOG.warnv("Cut {0} unreadable bytes from the end of WAL {1} at offset {2}; "+ + "the entries in them are lost, entries appended from now on are replayed on the next load",+ fileSize - lastGoodOffset, walPath, lastGoodOffset);+ } catch (IOException e) {+ LOG.errorv(e, "Failed to cut the unreadable tail of WAL {0}; entries appended after it "+ + "will not be replayed until the next compaction", walPath);+ }+ }+private void openWalWriter() {try {Files.createDirectories(walPath.getParent());walWriter = new DataOutputStream(new BufferedOutputStream(Files.newOutputStream(walPath,- java.nio.file.StandardOpenOption.CREATE,- java.nio.file.StandardOpenOption.APPEND)));+ StandardOpenOption.CREATE, StandardOpenOption.APPEND)));} catch (IOException e) {LOG.errorv(e, "Failed to open WAL writer at {0}", walPath);}diff --git a/src/main/java/io/github/hectorvent/floci/core/storage/WriteProfile.java b/src/main/java/io/github/hectorvent/floci/core/storage/WriteProfile.javanew file mode 100644index 0000000000..cddd002e04--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/core/storage/WriteProfile.java@@ -0,0 +1,20 @@+package io.github.hectorvent.floci.core.storage;++/**+ * How a store is written, so {@link StorageFactory} can pick a backend that fits the workload+ * under the configured storage mode. The profile describes the data, not a backend: the+ * {@code memory}, {@code hybrid} and {@code wal} modes treat both profiles alike.+ */+public enum WriteProfile {+ /**+ * Small sets that change rarely, such as resource definitions. Under {@code persistent} mode+ * the store file is rewritten in full on every mutation, so it is current after every call.+ */+ DEFAULT,+ /**+ * Append-heavy data such as log events. Under {@code persistent} mode the store is journaled:+ * every mutation is appended to a {@code .wal} file next to the store, and the store file is+ * rewritten only on the WAL compaction interval and at shutdown.+ */+ APPEND_HEAVY+}diff --git a/src/main/java/io/github/hectorvent/floci/services/cloudwatch/logs/CloudWatchLogsService.java b/src/main/java/io/github/hectorvent/floci/services/cloudwatch/logs/CloudWatchLogsService.javaindex a475196b08..57b18a8123 100644--- a/src/main/java/io/github/hectorvent/floci/services/cloudwatch/logs/CloudWatchLogsService.java+++ b/src/main/java/io/github/hectorvent/floci/services/cloudwatch/logs/CloudWatchLogsService.java@@ -7,6 +7,7 @@ import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;import io.github.hectorvent.floci.core.storage.InMemoryStorage;import io.github.hectorvent.floci.core.storage.StorageBackend;import io.github.hectorvent.floci.core.storage.StorageFactory;+import io.github.hectorvent.floci.core.storage.WriteProfile;import io.github.hectorvent.floci.services.cloudwatch.logs.filter.FilterPattern;import io.github.hectorvent.floci.services.cloudwatch.logs.filter.FilterPatternException;import io.github.hectorvent.floci.services.cloudwatch.logs.model.LogEvent;@@ -117,8 +118,10 @@ public class CloudWatchLogsService implements ResourceProvider {new TypeReference<>() {}),storageFactory.create("cloudwatchlogs", "cwlogs-streams.json",new TypeReference<>() {}),+ // Every PutLogEvents call lands here, so under persistent mode the store is+ // journaled instead of rewritten in full per batch (#2500).storageFactory.create("cloudwatchlogs", "cwlogs-events.json",- new TypeReference<>() {}),+ new TypeReference<>() {}, WriteProfile.APPEND_HEAVY),storageFactory.create("cloudwatchlogs", "cwlogs-subscription-filters.json",new TypeReference<>() {}),storageFactory.create("cloudwatchlogs", "cwlogs-metric-filters.json",@@ -689,9 +692,10 @@ public class CloudWatchLogsService implements ResourceProvider {/*** Keeps an account's event store under {@link #maxStoredEvents} by dropping the oldest events.- * The store is persisted as a single document rewritten in full on each flush, so its size- * is the cost of every flush; without a ceiling a chatty function turns log ingestion into a- * sustained disk writer.+ * The ceiling bounds the memory footprint and the snapshot that the store is compacted into:+ * under persistent mode a batch is appended to the journal and the whole store is rewritten+ * only on the compaction interval, but without a ceiling a chatty function would still grow+ * every snapshot without bound.* <p>* The ceiling is best-effort rather than atomic: this method is not synchronized, so two* concurrent PutLogEvents calls for the same account can each scan and evict independently anddiff --git a/src/main/resources/application.yml b/src/main/resources/application.ymlindex 02634c83fd..6d883ea9b5 100644--- a/src/main/resources/application.yml+++ b/src/main/resources/application.yml@@ -153,6 +153,8 @@ floci:# mount-user: # run mounting containers as "uid[:gid]" (access point PosixUser)# mount-group-add: # supplementary gid added to mounting containerswal:+ # Also the cadence at which journaled stores under persistent mode (CloudWatch Logs events)+ # fold their .wal file into the store's JSON file.compaction-interval-ms: 30000services:ssm:diff --git a/src/test/java/io/github/hectorvent/floci/core/storage/StorageFactoryWriteProfileTest.java b/src/test/java/io/github/hectorvent/floci/core/storage/StorageFactoryWriteProfileTest.javanew file mode 100644index 0000000000..56bd38b454--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/core/storage/StorageFactoryWriteProfileTest.java@@ -0,0 +1,129 @@+package io.github.hectorvent.floci.core.storage;++import com.fasterxml.jackson.core.type.TypeReference;+import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.ServiceConfigAccess;+import org.junit.jupiter.api.Test;+import org.junit.jupiter.api.io.TempDir;++import java.io.IOException;+import java.nio.file.Files;+import java.nio.file.Path;+import java.util.Map;+import java.util.Optional;+import java.util.stream.Stream;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.mockito.ArgumentMatchers.anyString;+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.when;++/**+ * How {@link StorageFactory} maps a store's {@link WriteProfile} onto a backend: under+ * {@code persistent} mode an append-heavy store is journaled instead of rewritten on every+ * mutation, while its snapshot stays at the store's own JSON file so the data can be read by a+ * plain persistent store and back.+ */+class StorageFactoryWriteProfileTest {++ private static final TypeReference<Map<String, String>> TYPE = new TypeReference<>() {};++ @TempDir+ Path tempDir;++ @Test+ void persistentModeJournalsAppendHeavyStoresSoAWriteCostsTheWriteNotTheStore() throws IOException {+ StorageFactory factory = newFactory("persistent");+ AccountAwareStorageBackend<String> events =+ factory.create("svc", "svc.json", TYPE, WriteProfile.APPEND_HEAVY);++ fill(events, 0, 1_000);+ long smallStore = walSize();+ events.put(key(1_000), "value");+ long costAtOneThousand = walSize() - smallStore;++ fill(events, 1_001, 10_000);+ long largeStore = walSize();+ events.put(key(10_000), "value");+ long costAtTenThousand = walSize() - largeStore;++ assertTrue(costAtOneThousand > 0, "a put must reach the journal");+ assertEquals(costAtOneThousand, costAtTenThousand, "the cost of one put must not grow with the store");+ assertFalse(Files.exists(tempDir.resolve("svc.json")), "the store file is only written by compaction");++ factory.shutdownAll();+ assertTrue(Files.readString(tempDir.resolve("svc.json")).contains(key(10_000)),+ "shutdown must fold the journal into the store file");+ }++ @Test+ void persistentModeStillRewritesDefaultProfileStores() {+ StorageFactory factory = newFactory("persistent");+ factory.create("svc", "svc.json", TYPE).put("only", "v1");++ assertTrue(Files.exists(tempDir.resolve("svc.json")), "a default-profile store is written on every mutation");+ assertFalse(Files.exists(tempDir.resolve("svc.wal")));+ factory.shutdownAll();+ }++ @Test+ void aPersistentStoreIsReadByTheJournaledBackendAndBack() {+ StorageFactory before = newFactory("persistent");+ before.create("svc", "svc.json", TYPE).put("first", "v1");+ before.shutdownAll();++ StorageFactory journaled = newFactory("persistent");+ AccountAwareStorageBackend<String> upgraded =+ journaled.create("svc", "svc.json", TYPE, WriteProfile.APPEND_HEAVY);+ assertEquals(Optional.of("v1"), upgraded.get("first"));+ upgraded.put("second", "v2");+ journaled.shutdownAll();++ StorageFactory after = newFactory("persistent");+ AccountAwareStorageBackend<String> downgraded = after.create("svc", "svc.json", TYPE);+ assertEquals(Optional.of("v1"), downgraded.get("first"));+ assertEquals(Optional.of("v2"), downgraded.get("second"));+ after.shutdownAll();+ }++ @Test+ void memoryModeIgnoresTheWriteProfile() throws IOException {+ StorageFactory factory = newFactory("memory");+ factory.create("svc", "svc.json", TYPE, WriteProfile.APPEND_HEAVY).put("only", "v1");+ factory.shutdownAll();++ try (Stream<Path> files = Files.list(tempDir)) {+ assertEquals(0, files.count(), "memory mode must not touch the filesystem");+ }+ }++ private static void fill(AccountAwareStorageBackend<String> store, int from, int toExclusive) {+ for (int i = from; i < toExclusive; i++) {+ store.put(key(i), "value");+ }+ }++ /** Fixed-width keys, so every journal record has the same size. */+ private static String key(int i) {+ return String.format("k%05d", i);+ }++ private long walSize() throws IOException {+ Path wal = tempDir.resolve("svc.wal");+ return Files.exists(wal) ? Files.size(wal) : 0L;+ }++ private StorageFactory newFactory(String mode) {+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);+ when(config.defaultAccountId()).thenReturn("000000000000");+ when(config.storage().persistentPath()).thenReturn(tempDir.toString());+ when(config.storage().wal().compactionIntervalMs()).thenReturn(60_000L);+ ServiceConfigAccess serviceConfigAccess = mock(ServiceConfigAccess.class);+ when(serviceConfigAccess.storageMode(anyString())).thenReturn(mode);+ when(serviceConfigAccess.storageFlushInterval(anyString())).thenReturn(60_000L);+ return new StorageFactory(config, serviceConfigAccess);+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/core/storage/WalStorageTest.java b/src/test/java/io/github/hectorvent/floci/core/storage/WalStorageTest.javaindex e8539a9626..520c1d65b2 100644--- a/src/test/java/io/github/hectorvent/floci/core/storage/WalStorageTest.java+++ b/src/test/java/io/github/hectorvent/floci/core/storage/WalStorageTest.java@@ -9,6 +9,7 @@ import org.junit.jupiter.api.Test;import org.junit.jupiter.api.io.TempDir;import java.io.DataInputStream;+import java.io.DataOutputStream;import java.io.IOException;import java.io.RandomAccessFile;import java.nio.file.Files;@@ -196,4 +197,198 @@ class WalStorageTest {store1.shutdown();store2.shutdown();}++ @Test+ void putAllEntriesAreReplayedByAFreshInstance() {+ Path snapshotPath = tempDir.resolve("batch-snapshot.json");+ Path walPath = tempDir.resolve("batch-data.wal");++ WalStorage<String, String> store1 = new WalStorage<>(snapshotPath, walPath,+ new TypeReference<Map<String, String>>() {}, 600000);+ store1.load();+ store1.put("key1", "old");+ store1.putAll(Map.of("key1", "new", "key2", "value2", "key3", "value3"));++ WalStorage<String, String> store2 = new WalStorage<>(snapshotPath, walPath,+ new TypeReference<Map<String, String>>() {}, 600000);+ store2.load();+ assertEquals("new", store2.get("key1").orElseThrow());+ assertEquals("value2", store2.get("key2").orElseThrow());+ assertEquals("value3", store2.get("key3").orElseThrow());++ store1.shutdown();+ store2.shutdown();+ }++ @Test+ void putAllOfAnEmptyBatchLeavesTheWalUntouched() throws IOException {+ Path walPath = tempDir.resolve("data.wal");+ storage.put("key1", "value1");+ long sizeBeforeEmptyBatch = Files.size(walPath);++ storage.putAll(Map.of());++ assertEquals(sizeBeforeEmptyBatch, Files.size(walPath));+ }++ @Test+ void flushOfAnUnchangedStoreLeavesTheSnapshotUntouched() throws IOException {+ Path snapshotPath = tempDir.resolve("idle-snapshot.json");+ Path walPath = tempDir.resolve("idle-data.wal");++ WalStorage<String, String> store = new WalStorage<>(snapshotPath, walPath,+ new TypeReference<Map<String, String>>() {}, 600000);+ store.load();+ store.put("key1", "value1");+ store.flush();++ // Nothing changed since the last snapshot, so a flush has nothing to write: a marker+ // placed in the file afterwards must survive it.+ Files.writeString(snapshotPath, "left alone");+ store.flush();+ assertEquals("left alone", Files.readString(snapshotPath));++ store.put("key2", "value2");+ store.flush();+ assertTrue(Files.readString(snapshotPath).contains("key2"));++ store.delete("key2");+ store.flush();+ assertFalse(Files.readString(snapshotPath).contains("key2"));++ store.shutdown();+ }++ @Test+ void entriesReplayedFromTheWalAreCompactedByTheNextFlush() throws IOException {+ Path snapshotPath = tempDir.resolve("replayed-snapshot.json");+ Path walPath = tempDir.resolve("replayed-data.wal");++ WalStorage<String, String> store1 = new WalStorage<>(snapshotPath, walPath,+ new TypeReference<Map<String, String>>() {}, 600000);+ store1.load();+ store1.put("key1", "value1");++ WalStorage<String, String> store2 = new WalStorage<>(snapshotPath, walPath,+ new TypeReference<Map<String, String>>() {}, 600000);+ store2.load();+ store2.flush();++ assertTrue(Files.readString(snapshotPath).contains("key1"));+ assertEquals(0L, Files.size(walPath));++ store1.shutdown();+ store2.shutdown();+ }++ @Test+ void appendsAfterATornTailAreReplayedOnTheNextLoad() throws IOException {+ Path walPath = tempDir.resolve("torn.wal");+ Path snapshotPath = tempDir.resolve("torn-snapshot.json");++ WalStorage<String, String> store1 = new WalStorage<>(snapshotPath, walPath,+ new TypeReference<Map<String, String>>() {}, 600000);+ store1.load();+ store1.put("good", "data");+ store1.put("another", "entry");+ try (RandomAccessFile raf = new RandomAccessFile(walPath.toFile(), "rw")) {+ raf.setLength(Files.size(walPath) - 3);+ }++ // A restart after the crash keeps running and appends behind the torn record.+ WalStorage<String, String> store2 = new WalStorage<>(snapshotPath, walPath,+ new TypeReference<Map<String, String>>() {}, 600000);+ store2.load();+ store2.put("later", "write");++ WalStorage<String, String> store3 = new WalStorage<>(snapshotPath, walPath,+ new TypeReference<Map<String, String>>() {}, 600000);+ store3.load();+ assertEquals("data", store3.get("good").orElseThrow());+ assertEquals("write", store3.get("later").orElseThrow());++ store1.shutdown();+ store2.shutdown();+ store3.shutdown();+ }++ @Test+ void aNegativeRecordLengthStopsReplayInsteadOfFailingLoad() throws IOException {+ Path walPath = tempDir.resolve("negative.wal");+ Path snapshotPath = tempDir.resolve("negative-snapshot.json");+ try (DataOutputStream out = new DataOutputStream(Files.newOutputStream(walPath))) {+ writePutRecord(out, "good", "data");+ out.writeByte(WalStorage.OP_PUT);+ out.writeInt(Integer.MIN_VALUE);+ }++ WalStorage<String, String> store = new WalStorage<>(snapshotPath, walPath,+ new TypeReference<Map<String, String>>() {}, 600000);+ assertDoesNotThrow(store::load);+ assertEquals("data", store.get("good").orElseThrow());+ assertEquals(1, store.keys().size());++ store.shutdown();+ }++ @Test+ void aRecordLengthBeyondTheFileEndIsTreatedAsATornTail() throws IOException {+ Path walPath = tempDir.resolve("oversized.wal");+ Path snapshotPath = tempDir.resolve("oversized-snapshot.json");+ try (DataOutputStream out = new DataOutputStream(Files.newOutputStream(walPath))) {+ writePutRecord(out, "good", "data");+ out.writeByte(WalStorage.OP_PUT);+ out.writeInt(Integer.MAX_VALUE);+ out.write(new byte[] {0x61, 0x78});+ }++ WalStorage<String, String> store1 = new WalStorage<>(snapshotPath, walPath,+ new TypeReference<Map<String, String>>() {}, 600000);+ assertDoesNotThrow(store1::load);+ assertEquals("data", store1.get("good").orElseThrow());+ assertEquals(1, store1.keys().size());+ store1.put("later", "write");++ WalStorage<String, String> store2 = new WalStorage<>(snapshotPath, walPath,+ new TypeReference<Map<String, String>>() {}, 600000);+ store2.load();+ assertEquals("data", store2.get("good").orElseThrow());+ assertEquals("write", store2.get("later").orElseThrow());++ store1.shutdown();+ store2.shutdown();+ }++ @Test+ void loadQuarantinesAnUnreadableSnapshot() throws IOException {+ Path snapshotPath = tempDir.resolve("corrupt-snapshot.json");+ Path walPath = tempDir.resolve("corrupt-data.wal");+ String unreadableContents = "{ not valid json";+ Path quarantinePath = tempDir.resolve("corrupt-snapshot.json.corrupt");+ Files.writeString(snapshotPath, unreadableContents);++ WalStorage<String, String> store = new WalStorage<>(snapshotPath, walPath,+ new TypeReference<Map<String, String>>() {}, 600000);+ try {+ store.load();++ assertTrue(store.keys().isEmpty());+ assertFalse(Files.exists(snapshotPath));+ assertEquals(unreadableContents, Files.readString(quarantinePath));+ } finally {+ store.shutdown();+ }+ }++ /** Writes one PUT record in the documented WAL layout, independent of the store's writer. */+ private static void writePutRecord(DataOutputStream out, String key, String value) throws IOException {+ ObjectMapper cbor = new ObjectMapper(new CBORFactory());+ byte[] keyBytes = cbor.writeValueAsBytes(key);+ byte[] valueBytes = cbor.writeValueAsBytes(value);+ out.writeByte(WalStorage.OP_PUT);+ out.writeInt(keyBytes.length);+ out.write(keyBytes);+ out.writeInt(valueBytes.length);+ out.write(valueBytes);+ }}diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/LogsMetricFilterCfnProvisionerTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/LogsMetricFilterCfnProvisionerTest.javaindex b307ca5668..cc3cc0d2d4 100644--- a/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/LogsMetricFilterCfnProvisionerTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/LogsMetricFilterCfnProvisionerTest.java@@ -82,6 +82,8 @@ class LogsMetricFilterCfnProvisionerTest {"cwlogs-metric-filters.json".equals(i.getArgument(1))? canonicalStore: AccountAwareStorageBackend.inMemory("000000000000"));+ when(storage.create(any(), any(), any(), any())).thenAnswer(i ->+ AccountAwareStorageBackend.inMemory("000000000000"));EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);when(config.services().cloudwatchlogs().maxStoredEvents()).thenReturn(Integer.MAX_VALUE);logs = new CloudWatchLogsService(storage, config, resolver, null, null);diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudwatch/logs/CloudWatchLogsEventJournalTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudwatch/logs/CloudWatchLogsEventJournalTest.javanew file mode 100644index 0000000000..32ca16b592--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/cloudwatch/logs/CloudWatchLogsEventJournalTest.java@@ -0,0 +1,107 @@+package io.github.hectorvent.floci.services.cloudwatch.logs;++import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.RegionResolver;+import io.github.hectorvent.floci.core.common.ServiceConfigAccess;+import io.github.hectorvent.floci.core.storage.StorageFactory;+import io.github.hectorvent.floci.services.cloudwatch.logs.model.LogEvent;+import org.junit.jupiter.api.Test;+import org.junit.jupiter.api.io.TempDir;++import java.io.IOException;+import java.nio.file.Files;+import java.nio.file.Path;+import java.util.ArrayList;+import java.util.HashSet;+import java.util.List;+import java.util.Map;+import java.util.Set;+import java.util.stream.Collectors;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.mockito.ArgumentMatchers.anyString;+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.when;++/**+ * Under {@code persistent} storage a PutLogEvents batch is appended to the events journal instead+ * of rewriting the whole events store on every call, while the definition stores keep their+ * rewritten-on-every-call file (issue #2500).+ */+class CloudWatchLogsEventJournalTest {++ private static final String ACCOUNT = "000000000000";+ private static final String REGION = "eu-west-1";+ private static final String GROUP = "/journal";+ private static final String STREAM = "s";+ private static final int EVENTS_PER_BATCH = 10;++ @TempDir+ Path directory;++ @Test+ void putLogEventsUnderPersistentModeAppendsToTheJournalInsteadOfRewritingTheStore() throws IOException {+ Path journal = directory.resolve("cwlogs-events.wal");+ Path eventStore = directory.resolve("cwlogs-events.json");+ Path groupStore = directory.resolve("cwlogs-groups.json");++ StorageFactory first = newFactory();+ CloudWatchLogsService logs = newService(first);+ logs.createLogGroup(GROUP, null, null, REGION);+ logs.createLogStream(GROUP, STREAM, REGION);+ assertTrue(Files.readString(groupStore).contains(GROUP), "definition stores are still written on every call");++ Set<String> written = new HashSet<>();+ long journalSize = 0;+ for (int batch = 0; batch < 3; batch++) {+ List<Map<String, Object>> events = batch(batch);+ logs.putLogEvents(GROUP, STREAM, events, REGION);+ events.forEach(event -> written.add((String) event.get("message")));+ assertTrue(journalSize(journal) > journalSize, "every batch must append to the journal");+ journalSize = journalSize(journal);+ assertFalse(Files.exists(eventStore), "a batch must not rewrite the events store");+ }+ first.shutdownAll();++ assertTrue(Files.exists(eventStore), "a clean shutdown folds the journal into the events store");+ StorageFactory second = newFactory();+ CloudWatchLogsService reopened = newService(second);+ List<LogEvent> read = reopened.getLogEvents(GROUP, STREAM, null, null, 100, true, null, REGION).events();+ assertEquals(written, read.stream().map(LogEvent::getMessage).collect(Collectors.toSet()));+ second.shutdownAll();+ }++ private static long journalSize(Path journal) throws IOException {+ return Files.exists(journal) ? Files.size(journal) : 0L;+ }++ private static List<Map<String, Object>> batch(int batch) {+ long now = System.currentTimeMillis();+ List<Map<String, Object>> events = new ArrayList<>();+ for (int i = 0; i < EVENTS_PER_BATCH; i++) {+ events.add(Map.of("timestamp", now + i, "message", "batch " + batch + " event " + i));+ }+ return events;+ }++ private CloudWatchLogsService newService(StorageFactory factory) {+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);+ when(config.services().cloudwatchlogs().maxEventsPerQuery()).thenReturn(10_000);+ when(config.services().cloudwatchlogs().maxStoredEvents()).thenReturn(10_000);+ return new CloudWatchLogsService(factory, config, new RegionResolver(REGION, ACCOUNT), null, null);+ }++ private StorageFactory newFactory() {+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);+ when(config.defaultAccountId()).thenReturn(ACCOUNT);+ when(config.storage().persistentPath()).thenReturn(directory.toString());+ when(config.storage().wal().compactionIntervalMs()).thenReturn(3_600_000L);+ ServiceConfigAccess access = mock(ServiceConfigAccess.class);+ when(access.storageMode(anyString())).thenReturn("persistent");+ when(access.storageFlushInterval(anyString())).thenReturn(3_600_000L);+ return new StorageFactory(config, access);+ }+}