Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

fix(cloudwatchlogs): journal log events instead of rewriting the store under persistent mode (#4187)

Niels Kootstraa0cb86cc133f
diff --git a/docs/configuration/advanced/application-yml.md b/docs/configuration/advanced/application-yml.md
index ac6cf27628..c108c91494 100644
--- a/docs/configuration/advanced/application-yml.md
+++ b/docs/configuration/advanced/application-yml.md
@@ -55,6 +55,8 @@ floci:
# mount-user: "1001:1001" # PosixUser: run mounting containers as uid[:gid]
# mount-group-add: 2000 # supplementary gid added to mounting containers
wal:
+ # Also the cadence at which journaled stores under persistent mode (CloudWatch Logs events)
+ # fold their .wal file into the store's JSON file.
compaction-interval-ms: 30000
services:
ssm:
diff --git a/docs/configuration/environment-variables.md b/docs/configuration/environment-variables.md
index c17becfdbd..742ed0073e 100644
--- a/docs/configuration/environment-variables.md
+++ b/docs/configuration/environment-variables.md
@@ -81,7 +81,7 @@ See [TLS / HTTPS](./tls.md) for SDK configuration examples and WebSocket (`wss:/
| `FLOCI_STORAGE_PERSISTENT_PATH` | `./data` | Container-side directory for persistent and hybrid storage |
| `FLOCI_STORAGE_HOST_PERSISTENT_PATH` | `./data` | Host-side path for Docker volume bind-mounts (RDS, OpenSearch, MSK, ECR data). When unset, Floci uses named Docker volumes |
| `FLOCI_STORAGE_PRUNE_VOLUMES_ON_DELETE` | `false` | Remove named Docker volumes immediately when the resource is deleted |
-| `FLOCI_STORAGE_WAL_COMPACTION_INTERVAL_MS` | `30000` | How often (ms) the WAL compaction runs. Only applies when `FLOCI_STORAGE_MODE=wal` |
+| `FLOCI_STORAGE_WAL_COMPACTION_INTERVAL_MS` | `30000` | How often (ms) the WAL compaction runs. Applies to `wal` mode and to the stores that are journaled under `persistent` mode (CloudWatch Logs events) |
### Per-service storage overrides
diff --git a/docs/configuration/storage.md b/docs/configuration/storage.md
index 240905da0f..88cb62d636 100644
--- a/docs/configuration/storage.md
+++ b/docs/configuration/storage.md
@@ -7,7 +7,7 @@ Floci supports four storage backends. You can set a global default and override
| Mode | Data survives restart | Write performance | Use case |
|---|---|---|---|
| `memory` | No | Fastest | Unit tests, CI pipelines |
-| `persistent` | Yes | Synchronous disk write on every change | Development with durable state |
+| `persistent` | Yes | Synchronous disk write on every change; append-heavy stores are journaled (see below) | Development with durable state |
| `hybrid` | Yes | In-memory reads, async flush to disk | General local development |
| `wal` | Yes | Append-only write-ahead log with compaction | High-write workloads |
@@ -22,6 +22,18 @@ Floci supports four storage backends. You can set a global default and override
!!! note "Code default vs shipped default"
The Java `@WithDefault` for `storage.mode` is `hybrid`, but the published Docker image ships with `memory` set in `application.yml`. Running the stock image gives you `memory` unless you set `FLOCI_STORAGE_MODE`.
+### Journaled stores under `persistent` mode
+
+Most stores under `persistent` mode are rewritten in full on every change, which keeps the file
+current after every call but makes the cost of one write grow with the size of the store.
+Append-heavy stores are journaled instead: a change is appended to a `.wal` file next to the
+store, and the store's JSON file is rewritten from memory on the `FLOCI_STORAGE_WAL_COMPACTION_INTERVAL_MS`
+cadence and at shutdown, and only when something changed. Today this applies to CloudWatch Logs
+events (`cwlogs-events.json` with `cwlogs-events.wal`). After a clean shutdown the JSON file holds
+every event; while Floci runs it can be up to one compaction interval behind, and the journal is
+replayed on the next start. An existing `cwlogs-events.json` from an older version is picked up as
+the first snapshot without any migration.
+
## Per-Service Override
When not set for a service, it inherits `FLOCI_STORAGE_MODE`. Only override when you need different behavior for a specific service.
@@ -39,7 +51,7 @@ When not set for a service, it inherits `FLOCI_STORAGE_MODE`. Only override when
| `FLOCI_STORAGE_SERVICES_LAMBDA_MODE` | global default | Lambda storage mode |
| `FLOCI_STORAGE_SERVICES_LAMBDA_FLUSH_INTERVAL_MS` | `5000` | Lambda flush interval (ms) |
| `FLOCI_STORAGE_SERVICES_CLOUDWATCHLOGS_MODE` | global default | CloudWatch Logs storage mode |
-| `FLOCI_STORAGE_SERVICES_CLOUDWATCHLOGS_FLUSH_INTERVAL_MS` | `5000` | CloudWatch Logs flush interval (ms) |
+| `FLOCI_STORAGE_SERVICES_CLOUDWATCHLOGS_FLUSH_INTERVAL_MS` | `15000` | CloudWatch Logs flush interval (ms) |
| `FLOCI_STORAGE_SERVICES_CLOUDWATCHMETRICS_MODE` | global default | CloudWatch Metrics storage mode |
| `FLOCI_STORAGE_SERVICES_CLOUDWATCHMETRICS_FLUSH_INTERVAL_MS` | `5000` | CloudWatch Metrics flush interval (ms) |
| `FLOCI_STORAGE_SERVICES_SECRETSMANAGER_MODE` | global default | Secrets Manager storage mode |
diff --git a/docs/services/cloudwatch.md b/docs/services/cloudwatch.md
index e9e1b61e56..10ed9dd3f1 100644
--- a/docs/services/cloudwatch.md
+++ b/docs/services/cloudwatch.md
@@ -243,6 +243,18 @@ given rather than by omitting it, which is what its SDK paginators expect.
| `FLOCI_SERVICES_CLOUDWATCHLOGS_MAX_EVENTS_PER_QUERY` | `10000` | Maximum events returned per `FilterLogEvents` / `GetLogEvents` call, and the upper bound for a Logs Insights `limit` |
| `FLOCI_SERVICES_CLOUDWATCHLOGS_QUERY_COMPLETION_DELAY_MS` | `0` | Artificial Logs Insights query delay. With `0` a query completes immediately. A positive value emulates the real asynchronous lifecycle (`Running` → `Complete` after the delay), which also makes `StopQuery` on a still-running query return `success=true` |
+### Storage
+
+Log groups, streams, filters and resource policies follow the configured storage mode as every
+other service does. The event store is the one CloudWatch Logs store under steady append load, so
+under `persistent` mode it is journaled instead of rewritten on every `PutLogEvents` call: a
+batch is appended to `cwlogs-events.wal`, and `cwlogs-events.json` is rewritten from memory on
+the `FLOCI_STORAGE_WAL_COMPACTION_INTERVAL_MS` cadence and at shutdown. The store keeps at most
+20,000 events per account (`FLOCI_SERVICES_CLOUDWATCHLOGS_MAX_STORED_EVENTS`), which bounds both
+the memory footprint and the size of each snapshot. `hybrid` remains the alternative when a
+bounded delay for every CloudWatch Logs store is acceptable. See
+[Storage Modes](../configuration/storage.md#journaled-stores-under-persistent-mode).
+
### Examples
```bash
diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
index d9fea803d0..b4a6547ede 100644
--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
@@ -1663,9 +1663,10 @@ public interface EmulatorConfig {
int maxEventsPerQuery();
/**
- * Upper bound on log events kept across all groups. The store is one JSON document rewritten
- * in full on every flush, so an unbounded store turns a chatty or retrying Lambda into a
- * sustained multi-hundred-MB/s disk writer. Oldest events are evicted first once exceeded.
+ * Upper bound on log events kept across all groups. The store is compacted into one JSON
+ * document on every flush (under persistent mode it is journaled in between), so an
+ * unbounded store turns a chatty or retrying Lambda into a sustained multi-hundred-MB/s
+ * disk writer. Oldest events are evicted first once exceeded.
*/
@WithDefault("20000")
int maxStoredEvents();
diff --git a/src/main/java/io/github/hectorvent/floci/core/storage/StorageFactory.java b/src/main/java/io/github/hectorvent/floci/core/storage/StorageFactory.java
index a8750030b3..5bbf043764 100644
--- a/src/main/java/io/github/hectorvent/floci/core/storage/StorageFactory.java
+++ b/src/main/java/io/github/hectorvent/floci/core/storage/StorageFactory.java
@@ -63,6 +63,24 @@ public class StorageFactory {
*/
public synchronized <V> AccountAwareStorageBackend<V> create(String serviceName, String fileName,
TypeReference<Map<String, V>> typeReference) {
+ return create(serviceName, fileName, typeReference, WriteProfile.DEFAULT);
+ }
+
+ /**
+ * Create an account-aware storage backend for the given service with a write profile.
+ * The profile only matters under {@code persistent} mode: an {@link WriteProfile#APPEND_HEAVY}
+ * store is journaled to {@code <file>.wal} and its JSON file becomes the snapshot that
+ * compaction rewrites, instead of the file being rewritten on every mutation. The first
+ * {@code create()} for a path decides the profile; repeat calls reuse that backend.
+ *
+ * @param serviceName the service name (ssm, sqs, s3, …)
+ * @param fileName the JSON file name for persistent storage
+ * @param typeReference Jackson type reference for deserialization
+ * @param profile how the store is written
+ */
+ public synchronized <V> AccountAwareStorageBackend<V> create(String serviceName, String fileName,
+ TypeReference<Map<String, V>> typeReference,
+ WriteProfile profile) {
String mode = resolveMode(serviceName);
long flushInterval = resolveFlushInterval(serviceName);
Path basePath = Path.of(config.storage().persistentPath());
@@ -83,7 +101,21 @@ public class StorageFactory {
StorageBackend<String, V> inner = switch (mode) {
case "memory" -> new InMemoryStorage<>();
- case "persistent" -> new PersistentStorage<>(filePath, typeReference);
+ case "persistent" -> {
+ if (profile != WriteProfile.APPEND_HEAVY) {
+ yield new PersistentStorage<>(filePath, typeReference);
+ }
+ // The store file doubles as the WAL snapshot: a file written by the plain
+ // persistent backend is the first snapshot, and after a clean shutdown the file
+ // is current again. Only the .wal file next to it is new.
+ Path walFilePath = basePath.resolve(fileName.replace(".json", ".wal"));
+ WalStorage<String, V> wal = new WalStorage<>(filePath, walFilePath, typeReference,
+ config.storage().wal().compactionIntervalMs());
+ walBackends.add(wal);
+ LOG.infov("Journaling {0} for service {1} to {2}; the store file is rewritten on the WAL compaction interval",
+ filePath, serviceName, walFilePath);
+ yield wal;
+ }
case "hybrid" -> {
var hybrid = new HybridStorage<>(filePath, typeReference, flushInterval);
hybridBackends.add(hybrid);
diff --git a/src/main/java/io/github/hectorvent/floci/core/storage/WalStorage.java b/src/main/java/io/github/hectorvent/floci/core/storage/WalStorage.java
index f632207854..316374a826 100644
--- a/src/main/java/io/github/hectorvent/floci/core/storage/WalStorage.java
+++ b/src/main/java/io/github/hectorvent/floci/core/storage/WalStorage.java
@@ -1,5 +1,6 @@
package io.github.hectorvent.floci.core.storage;
+import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
@@ -8,9 +9,11 @@ import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
import org.jboss.logging.Logger;
import java.io.*;
+import java.nio.channels.FileChannel;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
+import java.nio.file.StandardOpenOption;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
@@ -24,7 +27,8 @@ import java.util.stream.Collectors;
/**
* Write-Ahead Log storage: in-memory reads with append-only binary WAL for durability.
- * Periodic compaction writes a full snapshot and truncates the WAL.
+ * Periodic compaction writes a full snapshot and truncates the WAL, and is skipped while
+ * nothing has changed since the last snapshot.
* On startup: load snapshot, then replay WAL entries after snapshot.
*
* Binary WAL entry format:
@@ -50,6 +54,9 @@ public class WalStorage<K, V> implements StorageBackend<K, V> {
private final ReentrantReadWriteLock compactionLock = new ReentrantReadWriteLock();
private final ScheduledExecutorService scheduler;
private volatile DataOutputStream walWriter;
+ // Set under the read lock by every mutation and cleared under the write lock by compaction,
+ // so an idle store does not rewrite its snapshot on every compaction tick.
+ private volatile boolean dirty;
public WalStorage(Path snapshotPath, Path walPath, TypeReference<Map<K, V>> typeReference,
long compactionIntervalMs) {
@@ -80,12 +87,32 @@ public class WalStorage<K, V> implements StorageBackend<K, V> {
compactionLock.readLock().lock();
try {
store.put(key, value);
+ dirty = true;
appendPut(key, value);
} finally {
compactionLock.readLock().unlock();
}
}
+ /**
+ * Appends the whole batch as one contiguous run of WAL records with a single flush, so a
+ * batch costs one write instead of one per entry. An empty batch touches nothing.
+ */
+ @Override
+ public void putAll(Map<K, V> entries) {
+ if (entries.isEmpty()) {
+ return;
+ }
+ compactionLock.readLock().lock();
+ try {
+ store.putAll(entries);
+ dirty = true;
+ appendPuts(entries);
+ } finally {
+ compactionLock.readLock().unlock();
+ }
+ }
+
@Override
public Optional<V> get(K key) {
return Optional.ofNullable(store.get(key));
@@ -96,6 +123,7 @@ public class WalStorage<K, V> implements StorageBackend<K, V> {
compactionLock.readLock().lock();
try {
store.remove(key);
+ dirty = true;
appendDelete(key);
} finally {
compactionLock.readLock().unlock();
@@ -129,13 +157,17 @@ public class WalStorage<K, V> implements StorageBackend<K, V> {
store.putAll(data);
LOG.infov("Loaded {0} entries from snapshot {1}", store.size(), snapshotPath);
} catch (IOException e) {
- LOG.errorv(e, "Failed to load snapshot from {0}", snapshotPath);
+ StorageQuarantine.quarantine(snapshotPath, e, LOG);
}
}
if (Files.exists(walPath)) {
int replayed = replayWal();
LOG.infov("Replayed {0} WAL entries from {1}", replayed, walPath);
+ if (replayed > 0) {
+ // The snapshot is behind the log; the next compaction folds the log in.
+ dirty = true;
+ }
}
openWalWriter();
@@ -150,8 +182,11 @@ public class WalStorage<K, V> implements StorageBackend<K, V> {
try {
Files.deleteIfExists(walPath);
Files.deleteIfExists(snapshotPath);
+ dirty = false;
} catch (IOException e) {
LOG.errorv(e, "Failed to delete WAL/snapshot files");
+ // A snapshot that survived must be overwritten by the next compaction.
+ dirty = true;
}
openWalWriter();
} finally {
@@ -176,11 +211,15 @@ public class WalStorage<K, V> implements StorageBackend<K, V> {
private void compact() {
compactionLock.writeLock().lock();
try {
+ if (!dirty) {
+ return;
+ }
Files.createDirectories(snapshotPath.getParent());
Path tempFile = snapshotPath.resolveSibling(snapshotPath.getFileName() + ".tmp");
snapshotMapper.writeValue(tempFile.toFile(), store);
Files.move(tempFile, snapshotPath, StandardCopyOption.REPLACE_EXISTING,
StandardCopyOption.ATOMIC_MOVE);
+ dirty = false;
closeWalWriter();
Files.deleteIfExists(walPath);
@@ -201,11 +240,7 @@ public class WalStorage<K, V> implements StorageBackend<K, V> {
byte[] keyBytes = walMapper.writeValueAsBytes(key);
byte[] valueBytes = walMapper.writeValueAsBytes(value);
synchronized (out) {
- out.writeByte(OP_PUT);
- out.writeInt(keyBytes.length);
- out.write(keyBytes);
- out.writeInt(valueBytes.length);
- out.write(valueBytes);
+ writePutRecord(out, keyBytes, valueBytes);
out.flush();
}
} catch (IOException e) {
@@ -213,6 +248,38 @@ public class WalStorage<K, V> implements StorageBackend<K, V> {
}
}
+ private void appendPuts(Map<K, V> entries) {
+ DataOutputStream out = walWriter;
+ if (out == null) {
+ return;
+ }
+ try {
+ // Encode the whole batch off the shared writer so the lock is held for one write.
+ ByteArrayOutputStream batch = new ByteArrayOutputStream();
+ DataOutputStream records = new DataOutputStream(batch);
+ for (Map.Entry<K, V> entry : entries.entrySet()) {
+ writePutRecord(records, walMapper.writeValueAsBytes(entry.getKey()),
+ walMapper.writeValueAsBytes(entry.getValue()));
+ }
+ records.flush();
+ synchronized (out) {
+ batch.writeTo(out);
+ out.flush();
+ }
+ } catch (IOException e) {
+ LOG.errorv(e, "Failed to append {0} PUT WAL entries", entries.size());
+ }
+ }
+
+ private static void writePutRecord(DataOutputStream out, byte[] keyBytes, byte[] valueBytes)
+ throws IOException {
+ out.writeByte(OP_PUT);
+ out.writeInt(keyBytes.length);
+ out.write(keyBytes);
+ out.writeInt(valueBytes.length);
+ out.write(valueBytes);
+ }
+
private void appendDelete(K key) {
DataOutputStream out = walWriter;
if (out == null) return;
@@ -229,56 +296,102 @@ public class WalStorage<K, V> implements StorageBackend<K, V> {
}
}
+ /**
+ * Replays the log on top of the snapshot and returns the number of records applied.
+ * <p>
+ * Replay stops at the first record that cannot be framed or decoded: a tail torn by a crash,
+ * a length header that is negative or points past the end of the file, an unknown op byte, or
+ * a payload CBOR cannot read. The store is never failed by such a record; the bytes from that
+ * record onwards are cut off before the writer reopens, so records appended afterwards sit
+ * behind a valid record and are replayed on the next load instead of being lost behind the
+ * garbage. A read error that is not a framing problem leaves the file alone.
+ */
@SuppressWarnings("unchecked")
private int replayWal() {
int replayed = 0;
+ long fileSize;
+ try {
+ fileSize = Files.size(walPath);
+ } catch (IOException e) {
+ LOG.errorv(e, "Failed to read the size of WAL {0}", walPath);
+ return 0;
+ }
+ long consumed = 0;
+ boolean cutTail = true;
try (DataInputStream in = new DataInputStream(
new BufferedInputStream(Files.newInputStream(walPath)))) {
- while (true) {
- int op;
- try {
- op = in.readByte();
- } catch (EOFException e) {
+ while (consumed < fileSize) {
+ int op = in.readByte();
+ byte[] keyBytes = readFrame(in, fileSize - consumed - 1);
+ if (keyBytes == null) {
break;
}
-
- int keyLen = in.readInt();
- byte[] keyBytes = in.readNBytes(keyLen);
- if (keyBytes.length < keyLen) break; // truncated entry
-
+ long recordLength = 1 + 4 + keyBytes.length;
if (op == OP_PUT) {
- int valueLen = in.readInt();
- byte[] valueBytes = in.readNBytes(valueLen);
- if (valueBytes.length < valueLen) break; // truncated entry
-
+ byte[] valueBytes = readFrame(in, fileSize - consumed - recordLength);
+ if (valueBytes == null) {
+ break;
+ }
+ recordLength += 4 + valueBytes.length;
K key = (K) walMapper.readValue(keyBytes, Object.class);
V value = walMapper.readValue(valueBytes,
walMapper.constructType(typeReference.getType()).getContentType());
store.put(key, value);
- replayed++;
} else if (op == OP_DELETE) {
K key = (K) walMapper.readValue(keyBytes, Object.class);
store.remove(key);
- replayed++;
} else {
- LOG.errorv("Unknown WAL op byte: {0}, stopping replay", op);
+ LOG.warnv("Unknown WAL op byte {0} at offset {1} in {2}, stopping replay",
+ op, consumed, walPath);
break;
}
+ consumed += recordLength;
+ replayed++;
}
+ } catch (EOFException | JsonProcessingException e) {
+ LOG.warnv("WAL {0} is unreadable from offset {1}: {2}", walPath, consumed, e.getMessage());
} catch (IOException e) {
+ cutTail = false;
LOG.errorv(e, "Failed to replay WAL from {0} (replayed {1} entries before error)",
walPath, replayed);
}
+ if (cutTail && consumed < fileSize) {
+ cutWalTail(consumed, fileSize);
+ }
return replayed;
}
+ /**
+ * Reads one length-prefixed frame, or returns {@code null} when the length is negative or
+ * larger than the bytes left in the file, which marks the record as torn or corrupt.
+ */
+ private static byte[] readFrame(DataInputStream in, long bytesLeft) throws IOException {
+ int length = in.readInt();
+ if (length < 0 || length > bytesLeft - 4) {
+ return null;
+ }
+ byte[] bytes = in.readNBytes(length);
+ return bytes.length == length ? bytes : null;
+ }
+
+ private void cutWalTail(long lastGoodOffset, long fileSize) {
+ try (FileChannel channel = FileChannel.open(walPath, StandardOpenOption.WRITE)) {
+ channel.truncate(lastGoodOffset);
+ LOG.warnv("Cut {0} unreadable bytes from the end of WAL {1} at offset {2}; "
+ + "the entries in them are lost, entries appended from now on are replayed on the next load",
+ fileSize - lastGoodOffset, walPath, lastGoodOffset);
+ } catch (IOException e) {
+ LOG.errorv(e, "Failed to cut the unreadable tail of WAL {0}; entries appended after it "
+ + "will not be replayed until the next compaction", walPath);
+ }
+ }
+
private void openWalWriter() {
try {
Files.createDirectories(walPath.getParent());
walWriter = new DataOutputStream(
new BufferedOutputStream(Files.newOutputStream(walPath,
- java.nio.file.StandardOpenOption.CREATE,
- java.nio.file.StandardOpenOption.APPEND)));
+ StandardOpenOption.CREATE, StandardOpenOption.APPEND)));
} catch (IOException e) {
LOG.errorv(e, "Failed to open WAL writer at {0}", walPath);
}
diff --git a/src/main/java/io/github/hectorvent/floci/core/storage/WriteProfile.java b/src/main/java/io/github/hectorvent/floci/core/storage/WriteProfile.java
new file mode 100644
index 0000000000..cddd002e04
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/core/storage/WriteProfile.java
@@ -0,0 +1,20 @@
+package io.github.hectorvent.floci.core.storage;
+
+/**
+ * How a store is written, so {@link StorageFactory} can pick a backend that fits the workload
+ * under the configured storage mode. The profile describes the data, not a backend: the
+ * {@code memory}, {@code hybrid} and {@code wal} modes treat both profiles alike.
+ */
+public enum WriteProfile {
+ /**
+ * Small sets that change rarely, such as resource definitions. Under {@code persistent} mode
+ * the store file is rewritten in full on every mutation, so it is current after every call.
+ */
+ DEFAULT,
+ /**
+ * Append-heavy data such as log events. Under {@code persistent} mode the store is journaled:
+ * every mutation is appended to a {@code .wal} file next to the store, and the store file is
+ * rewritten only on the WAL compaction interval and at shutdown.
+ */
+ APPEND_HEAVY
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/cloudwatch/logs/CloudWatchLogsService.java b/src/main/java/io/github/hectorvent/floci/services/cloudwatch/logs/CloudWatchLogsService.java
index a475196b08..57b18a8123 100644
--- a/src/main/java/io/github/hectorvent/floci/services/cloudwatch/logs/CloudWatchLogsService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/cloudwatch/logs/CloudWatchLogsService.java
@@ -7,6 +7,7 @@ import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
import io.github.hectorvent.floci.core.storage.InMemoryStorage;
import io.github.hectorvent.floci.core.storage.StorageBackend;
import io.github.hectorvent.floci.core.storage.StorageFactory;
+import io.github.hectorvent.floci.core.storage.WriteProfile;
import io.github.hectorvent.floci.services.cloudwatch.logs.filter.FilterPattern;
import io.github.hectorvent.floci.services.cloudwatch.logs.filter.FilterPatternException;
import io.github.hectorvent.floci.services.cloudwatch.logs.model.LogEvent;
@@ -117,8 +118,10 @@ public class CloudWatchLogsService implements ResourceProvider {
new TypeReference<>() {}),
storageFactory.create("cloudwatchlogs", "cwlogs-streams.json",
new TypeReference<>() {}),
+ // Every PutLogEvents call lands here, so under persistent mode the store is
+ // journaled instead of rewritten in full per batch (#2500).
storageFactory.create("cloudwatchlogs", "cwlogs-events.json",
- new TypeReference<>() {}),
+ new TypeReference<>() {}, WriteProfile.APPEND_HEAVY),
storageFactory.create("cloudwatchlogs", "cwlogs-subscription-filters.json",
new TypeReference<>() {}),
storageFactory.create("cloudwatchlogs", "cwlogs-metric-filters.json",
@@ -689,9 +692,10 @@ public class CloudWatchLogsService implements ResourceProvider {
/**
* Keeps an account's event store under {@link #maxStoredEvents} by dropping the oldest events.
- * The store is persisted as a single document rewritten in full on each flush, so its size
- * is the cost of every flush; without a ceiling a chatty function turns log ingestion into a
- * sustained disk writer.
+ * The ceiling bounds the memory footprint and the snapshot that the store is compacted into:
+ * under persistent mode a batch is appended to the journal and the whole store is rewritten
+ * only on the compaction interval, but without a ceiling a chatty function would still grow
+ * every snapshot without bound.
* <p>
* The ceiling is best-effort rather than atomic: this method is not synchronized, so two
* concurrent PutLogEvents calls for the same account can each scan and evict independently and
diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml
index 02634c83fd..6d883ea9b5 100644
--- a/src/main/resources/application.yml
+++ b/src/main/resources/application.yml
@@ -153,6 +153,8 @@ floci:
# mount-user: # run mounting containers as "uid[:gid]" (access point PosixUser)
# mount-group-add: # supplementary gid added to mounting containers
wal:
+ # Also the cadence at which journaled stores under persistent mode (CloudWatch Logs events)
+ # fold their .wal file into the store's JSON file.
compaction-interval-ms: 30000
services:
ssm:
diff --git a/src/test/java/io/github/hectorvent/floci/core/storage/StorageFactoryWriteProfileTest.java b/src/test/java/io/github/hectorvent/floci/core/storage/StorageFactoryWriteProfileTest.java
new file mode 100644
index 0000000000..56bd38b454
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/core/storage/StorageFactoryWriteProfileTest.java
@@ -0,0 +1,129 @@
+package io.github.hectorvent.floci.core.storage;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.core.common.ServiceConfigAccess;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * How {@link StorageFactory} maps a store's {@link WriteProfile} onto a backend: under
+ * {@code persistent} mode an append-heavy store is journaled instead of rewritten on every
+ * mutation, while its snapshot stays at the store's own JSON file so the data can be read by a
+ * plain persistent store and back.
+ */
+class StorageFactoryWriteProfileTest {
+
+ private static final TypeReference<Map<String, String>> TYPE = new TypeReference<>() {};
+
+ @TempDir
+ Path tempDir;
+
+ @Test
+ void persistentModeJournalsAppendHeavyStoresSoAWriteCostsTheWriteNotTheStore() throws IOException {
+ StorageFactory factory = newFactory("persistent");
+ AccountAwareStorageBackend<String> events =
+ factory.create("svc", "svc.json", TYPE, WriteProfile.APPEND_HEAVY);
+
+ fill(events, 0, 1_000);
+ long smallStore = walSize();
+ events.put(key(1_000), "value");
+ long costAtOneThousand = walSize() - smallStore;
+
+ fill(events, 1_001, 10_000);
+ long largeStore = walSize();
+ events.put(key(10_000), "value");
+ long costAtTenThousand = walSize() - largeStore;
+
+ assertTrue(costAtOneThousand > 0, "a put must reach the journal");
+ assertEquals(costAtOneThousand, costAtTenThousand, "the cost of one put must not grow with the store");
+ assertFalse(Files.exists(tempDir.resolve("svc.json")), "the store file is only written by compaction");
+
+ factory.shutdownAll();
+ assertTrue(Files.readString(tempDir.resolve("svc.json")).contains(key(10_000)),
+ "shutdown must fold the journal into the store file");
+ }
+
+ @Test
+ void persistentModeStillRewritesDefaultProfileStores() {
+ StorageFactory factory = newFactory("persistent");
+ factory.create("svc", "svc.json", TYPE).put("only", "v1");
+
+ assertTrue(Files.exists(tempDir.resolve("svc.json")), "a default-profile store is written on every mutation");
+ assertFalse(Files.exists(tempDir.resolve("svc.wal")));
+ factory.shutdownAll();
+ }
+
+ @Test
+ void aPersistentStoreIsReadByTheJournaledBackendAndBack() {
+ StorageFactory before = newFactory("persistent");
+ before.create("svc", "svc.json", TYPE).put("first", "v1");
+ before.shutdownAll();
+
+ StorageFactory journaled = newFactory("persistent");
+ AccountAwareStorageBackend<String> upgraded =
+ journaled.create("svc", "svc.json", TYPE, WriteProfile.APPEND_HEAVY);
+ assertEquals(Optional.of("v1"), upgraded.get("first"));
+ upgraded.put("second", "v2");
+ journaled.shutdownAll();
+
+ StorageFactory after = newFactory("persistent");
+ AccountAwareStorageBackend<String> downgraded = after.create("svc", "svc.json", TYPE);
+ assertEquals(Optional.of("v1"), downgraded.get("first"));
+ assertEquals(Optional.of("v2"), downgraded.get("second"));
+ after.shutdownAll();
+ }
+
+ @Test
+ void memoryModeIgnoresTheWriteProfile() throws IOException {
+ StorageFactory factory = newFactory("memory");
+ factory.create("svc", "svc.json", TYPE, WriteProfile.APPEND_HEAVY).put("only", "v1");
+ factory.shutdownAll();
+
+ try (Stream<Path> files = Files.list(tempDir)) {
+ assertEquals(0, files.count(), "memory mode must not touch the filesystem");
+ }
+ }
+
+ private static void fill(AccountAwareStorageBackend<String> store, int from, int toExclusive) {
+ for (int i = from; i < toExclusive; i++) {
+ store.put(key(i), "value");
+ }
+ }
+
+ /** Fixed-width keys, so every journal record has the same size. */
+ private static String key(int i) {
+ return String.format("k%05d", i);
+ }
+
+ private long walSize() throws IOException {
+ Path wal = tempDir.resolve("svc.wal");
+ return Files.exists(wal) ? Files.size(wal) : 0L;
+ }
+
+ private StorageFactory newFactory(String mode) {
+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
+ when(config.defaultAccountId()).thenReturn("000000000000");
+ when(config.storage().persistentPath()).thenReturn(tempDir.toString());
+ when(config.storage().wal().compactionIntervalMs()).thenReturn(60_000L);
+ ServiceConfigAccess serviceConfigAccess = mock(ServiceConfigAccess.class);
+ when(serviceConfigAccess.storageMode(anyString())).thenReturn(mode);
+ when(serviceConfigAccess.storageFlushInterval(anyString())).thenReturn(60_000L);
+ return new StorageFactory(config, serviceConfigAccess);
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/core/storage/WalStorageTest.java b/src/test/java/io/github/hectorvent/floci/core/storage/WalStorageTest.java
index e8539a9626..520c1d65b2 100644
--- a/src/test/java/io/github/hectorvent/floci/core/storage/WalStorageTest.java
+++ b/src/test/java/io/github/hectorvent/floci/core/storage/WalStorageTest.java
@@ -9,6 +9,7 @@ import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.io.DataInputStream;
+import java.io.DataOutputStream;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.file.Files;
@@ -196,4 +197,198 @@ class WalStorageTest {
store1.shutdown();
store2.shutdown();
}
+
+ @Test
+ void putAllEntriesAreReplayedByAFreshInstance() {
+ Path snapshotPath = tempDir.resolve("batch-snapshot.json");
+ Path walPath = tempDir.resolve("batch-data.wal");
+
+ WalStorage<String, String> store1 = new WalStorage<>(snapshotPath, walPath,
+ new TypeReference<Map<String, String>>() {}, 600000);
+ store1.load();
+ store1.put("key1", "old");
+ store1.putAll(Map.of("key1", "new", "key2", "value2", "key3", "value3"));
+
+ WalStorage<String, String> store2 = new WalStorage<>(snapshotPath, walPath,
+ new TypeReference<Map<String, String>>() {}, 600000);
+ store2.load();
+ assertEquals("new", store2.get("key1").orElseThrow());
+ assertEquals("value2", store2.get("key2").orElseThrow());
+ assertEquals("value3", store2.get("key3").orElseThrow());
+
+ store1.shutdown();
+ store2.shutdown();
+ }
+
+ @Test
+ void putAllOfAnEmptyBatchLeavesTheWalUntouched() throws IOException {
+ Path walPath = tempDir.resolve("data.wal");
+ storage.put("key1", "value1");
+ long sizeBeforeEmptyBatch = Files.size(walPath);
+
+ storage.putAll(Map.of());
+
+ assertEquals(sizeBeforeEmptyBatch, Files.size(walPath));
+ }
+
+ @Test
+ void flushOfAnUnchangedStoreLeavesTheSnapshotUntouched() throws IOException {
+ Path snapshotPath = tempDir.resolve("idle-snapshot.json");
+ Path walPath = tempDir.resolve("idle-data.wal");
+
+ WalStorage<String, String> store = new WalStorage<>(snapshotPath, walPath,
+ new TypeReference<Map<String, String>>() {}, 600000);
+ store.load();
+ store.put("key1", "value1");
+ store.flush();
+
+ // Nothing changed since the last snapshot, so a flush has nothing to write: a marker
+ // placed in the file afterwards must survive it.
+ Files.writeString(snapshotPath, "left alone");
+ store.flush();
+ assertEquals("left alone", Files.readString(snapshotPath));
+
+ store.put("key2", "value2");
+ store.flush();
+ assertTrue(Files.readString(snapshotPath).contains("key2"));
+
+ store.delete("key2");
+ store.flush();
+ assertFalse(Files.readString(snapshotPath).contains("key2"));
+
+ store.shutdown();
+ }
+
+ @Test
+ void entriesReplayedFromTheWalAreCompactedByTheNextFlush() throws IOException {
+ Path snapshotPath = tempDir.resolve("replayed-snapshot.json");
+ Path walPath = tempDir.resolve("replayed-data.wal");
+
+ WalStorage<String, String> store1 = new WalStorage<>(snapshotPath, walPath,
+ new TypeReference<Map<String, String>>() {}, 600000);
+ store1.load();
+ store1.put("key1", "value1");
+
+ WalStorage<String, String> store2 = new WalStorage<>(snapshotPath, walPath,
+ new TypeReference<Map<String, String>>() {}, 600000);
+ store2.load();
+ store2.flush();
+
+ assertTrue(Files.readString(snapshotPath).contains("key1"));
+ assertEquals(0L, Files.size(walPath));
+
+ store1.shutdown();
+ store2.shutdown();
+ }
+
+ @Test
+ void appendsAfterATornTailAreReplayedOnTheNextLoad() throws IOException {
+ Path walPath = tempDir.resolve("torn.wal");
+ Path snapshotPath = tempDir.resolve("torn-snapshot.json");
+
+ WalStorage<String, String> store1 = new WalStorage<>(snapshotPath, walPath,
+ new TypeReference<Map<String, String>>() {}, 600000);
+ store1.load();
+ store1.put("good", "data");
+ store1.put("another", "entry");
+ try (RandomAccessFile raf = new RandomAccessFile(walPath.toFile(), "rw")) {
+ raf.setLength(Files.size(walPath) - 3);
+ }
+
+ // A restart after the crash keeps running and appends behind the torn record.
+ WalStorage<String, String> store2 = new WalStorage<>(snapshotPath, walPath,
+ new TypeReference<Map<String, String>>() {}, 600000);
+ store2.load();
+ store2.put("later", "write");
+
+ WalStorage<String, String> store3 = new WalStorage<>(snapshotPath, walPath,
+ new TypeReference<Map<String, String>>() {}, 600000);
+ store3.load();
+ assertEquals("data", store3.get("good").orElseThrow());
+ assertEquals("write", store3.get("later").orElseThrow());
+
+ store1.shutdown();
+ store2.shutdown();
+ store3.shutdown();
+ }
+
+ @Test
+ void aNegativeRecordLengthStopsReplayInsteadOfFailingLoad() throws IOException {
+ Path walPath = tempDir.resolve("negative.wal");
+ Path snapshotPath = tempDir.resolve("negative-snapshot.json");
+ try (DataOutputStream out = new DataOutputStream(Files.newOutputStream(walPath))) {
+ writePutRecord(out, "good", "data");
+ out.writeByte(WalStorage.OP_PUT);
+ out.writeInt(Integer.MIN_VALUE);
+ }
+
+ WalStorage<String, String> store = new WalStorage<>(snapshotPath, walPath,
+ new TypeReference<Map<String, String>>() {}, 600000);
+ assertDoesNotThrow(store::load);
+ assertEquals("data", store.get("good").orElseThrow());
+ assertEquals(1, store.keys().size());
+
+ store.shutdown();
+ }
+
+ @Test
+ void aRecordLengthBeyondTheFileEndIsTreatedAsATornTail() throws IOException {
+ Path walPath = tempDir.resolve("oversized.wal");
+ Path snapshotPath = tempDir.resolve("oversized-snapshot.json");
+ try (DataOutputStream out = new DataOutputStream(Files.newOutputStream(walPath))) {
+ writePutRecord(out, "good", "data");
+ out.writeByte(WalStorage.OP_PUT);
+ out.writeInt(Integer.MAX_VALUE);
+ out.write(new byte[] {0x61, 0x78});
+ }
+
+ WalStorage<String, String> store1 = new WalStorage<>(snapshotPath, walPath,
+ new TypeReference<Map<String, String>>() {}, 600000);
+ assertDoesNotThrow(store1::load);
+ assertEquals("data", store1.get("good").orElseThrow());
+ assertEquals(1, store1.keys().size());
+ store1.put("later", "write");
+
+ WalStorage<String, String> store2 = new WalStorage<>(snapshotPath, walPath,
+ new TypeReference<Map<String, String>>() {}, 600000);
+ store2.load();
+ assertEquals("data", store2.get("good").orElseThrow());
+ assertEquals("write", store2.get("later").orElseThrow());
+
+ store1.shutdown();
+ store2.shutdown();
+ }
+
+ @Test
+ void loadQuarantinesAnUnreadableSnapshot() throws IOException {
+ Path snapshotPath = tempDir.resolve("corrupt-snapshot.json");
+ Path walPath = tempDir.resolve("corrupt-data.wal");
+ String unreadableContents = "{ not valid json";
+ Path quarantinePath = tempDir.resolve("corrupt-snapshot.json.corrupt");
+ Files.writeString(snapshotPath, unreadableContents);
+
+ WalStorage<String, String> store = new WalStorage<>(snapshotPath, walPath,
+ new TypeReference<Map<String, String>>() {}, 600000);
+ try {
+ store.load();
+
+ assertTrue(store.keys().isEmpty());
+ assertFalse(Files.exists(snapshotPath));
+ assertEquals(unreadableContents, Files.readString(quarantinePath));
+ } finally {
+ store.shutdown();
+ }
+ }
+
+ /** Writes one PUT record in the documented WAL layout, independent of the store's writer. */
+ private static void writePutRecord(DataOutputStream out, String key, String value) throws IOException {
+ ObjectMapper cbor = new ObjectMapper(new CBORFactory());
+ byte[] keyBytes = cbor.writeValueAsBytes(key);
+ byte[] valueBytes = cbor.writeValueAsBytes(value);
+ out.writeByte(WalStorage.OP_PUT);
+ out.writeInt(keyBytes.length);
+ out.write(keyBytes);
+ out.writeInt(valueBytes.length);
+ out.write(valueBytes);
+ }
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/LogsMetricFilterCfnProvisionerTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/LogsMetricFilterCfnProvisionerTest.java
index b307ca5668..cc3cc0d2d4 100644
--- a/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/LogsMetricFilterCfnProvisionerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/LogsMetricFilterCfnProvisionerTest.java
@@ -82,6 +82,8 @@ class LogsMetricFilterCfnProvisionerTest {
"cwlogs-metric-filters.json".equals(i.getArgument(1))
? canonicalStore
: AccountAwareStorageBackend.inMemory("000000000000"));
+ when(storage.create(any(), any(), any(), any())).thenAnswer(i ->
+ AccountAwareStorageBackend.inMemory("000000000000"));
EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
when(config.services().cloudwatchlogs().maxStoredEvents()).thenReturn(Integer.MAX_VALUE);
logs = new CloudWatchLogsService(storage, config, resolver, null, null);
diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudwatch/logs/CloudWatchLogsEventJournalTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudwatch/logs/CloudWatchLogsEventJournalTest.java
new file mode 100644
index 0000000000..32ca16b592
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/cloudwatch/logs/CloudWatchLogsEventJournalTest.java
@@ -0,0 +1,107 @@
+package io.github.hectorvent.floci.services.cloudwatch.logs;
+
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.core.common.RegionResolver;
+import io.github.hectorvent.floci.core.common.ServiceConfigAccess;
+import io.github.hectorvent.floci.core.storage.StorageFactory;
+import io.github.hectorvent.floci.services.cloudwatch.logs.model.LogEvent;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Under {@code persistent} storage a PutLogEvents batch is appended to the events journal instead
+ * of rewriting the whole events store on every call, while the definition stores keep their
+ * rewritten-on-every-call file (issue #2500).
+ */
+class CloudWatchLogsEventJournalTest {
+
+ private static final String ACCOUNT = "000000000000";
+ private static final String REGION = "eu-west-1";
+ private static final String GROUP = "/journal";
+ private static final String STREAM = "s";
+ private static final int EVENTS_PER_BATCH = 10;
+
+ @TempDir
+ Path directory;
+
+ @Test
+ void putLogEventsUnderPersistentModeAppendsToTheJournalInsteadOfRewritingTheStore() throws IOException {
+ Path journal = directory.resolve("cwlogs-events.wal");
+ Path eventStore = directory.resolve("cwlogs-events.json");
+ Path groupStore = directory.resolve("cwlogs-groups.json");
+
+ StorageFactory first = newFactory();
+ CloudWatchLogsService logs = newService(first);
+ logs.createLogGroup(GROUP, null, null, REGION);
+ logs.createLogStream(GROUP, STREAM, REGION);
+ assertTrue(Files.readString(groupStore).contains(GROUP), "definition stores are still written on every call");
+
+ Set<String> written = new HashSet<>();
+ long journalSize = 0;
+ for (int batch = 0; batch < 3; batch++) {
+ List<Map<String, Object>> events = batch(batch);
+ logs.putLogEvents(GROUP, STREAM, events, REGION);
+ events.forEach(event -> written.add((String) event.get("message")));
+ assertTrue(journalSize(journal) > journalSize, "every batch must append to the journal");
+ journalSize = journalSize(journal);
+ assertFalse(Files.exists(eventStore), "a batch must not rewrite the events store");
+ }
+ first.shutdownAll();
+
+ assertTrue(Files.exists(eventStore), "a clean shutdown folds the journal into the events store");
+ StorageFactory second = newFactory();
+ CloudWatchLogsService reopened = newService(second);
+ List<LogEvent> read = reopened.getLogEvents(GROUP, STREAM, null, null, 100, true, null, REGION).events();
+ assertEquals(written, read.stream().map(LogEvent::getMessage).collect(Collectors.toSet()));
+ second.shutdownAll();
+ }
+
+ private static long journalSize(Path journal) throws IOException {
+ return Files.exists(journal) ? Files.size(journal) : 0L;
+ }
+
+ private static List<Map<String, Object>> batch(int batch) {
+ long now = System.currentTimeMillis();
+ List<Map<String, Object>> events = new ArrayList<>();
+ for (int i = 0; i < EVENTS_PER_BATCH; i++) {
+ events.add(Map.of("timestamp", now + i, "message", "batch " + batch + " event " + i));
+ }
+ return events;
+ }
+
+ private CloudWatchLogsService newService(StorageFactory factory) {
+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
+ when(config.services().cloudwatchlogs().maxEventsPerQuery()).thenReturn(10_000);
+ when(config.services().cloudwatchlogs().maxStoredEvents()).thenReturn(10_000);
+ return new CloudWatchLogsService(factory, config, new RegionResolver(REGION, ACCOUNT), null, null);
+ }
+
+ private StorageFactory newFactory() {
+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
+ when(config.defaultAccountId()).thenReturn(ACCOUNT);
+ when(config.storage().persistentPath()).thenReturn(directory.toString());
+ when(config.storage().wal().compactionIntervalMs()).thenReturn(3_600_000L);
+ ServiceConfigAccess access = mock(ServiceConfigAccess.class);
+ when(access.storageMode(anyString())).thenReturn("persistent");
+ when(access.storageFlushInterval(anyString())).thenReturn(3_600_000L);
+ return new StorageFactory(config, access);
+ }
+}