Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(dynamodb): opt-in per-table item files, split into shards (#6303)

Ben Wart763534547470
diff --git a/docs/configuration/storage.md b/docs/configuration/storage.md
index ca3170b1dc..df1fa66f58 100644
--- a/docs/configuration/storage.md
+++ b/docs/configuration/storage.md
@@ -48,6 +48,8 @@ When not set for a service, it inherits `FLOCI_STORAGE_MODE`. Only override when
| `FLOCI_STORAGE_SERVICES_S3_MODE` | global default | S3 storage mode |
| `FLOCI_STORAGE_SERVICES_DYNAMODB_MODE` | global default | DynamoDB storage mode |
| `FLOCI_STORAGE_SERVICES_DYNAMODB_FLUSH_INTERVAL_MS` | `5000` | DynamoDB flush interval (ms) |
+| `FLOCI_STORAGE_SERVICES_DYNAMODB_ITEM_LAYOUT` | `single-file` | `single-file` or `per-table` item files, see [DynamoDB](../services/dynamodb.md#item-layout-on-disk) |
+| `FLOCI_STORAGE_SERVICES_DYNAMODB_ITEM_SHARDS` | `1` | Files per table with the `per-table` item layout |
| `FLOCI_STORAGE_SERVICES_SNS_MODE` | global default | SNS storage mode |
| `FLOCI_STORAGE_SERVICES_SNS_FLUSH_INTERVAL_MS` | `5000` | SNS flush interval (ms) |
| `FLOCI_STORAGE_SERVICES_LAMBDA_MODE` | global default | Lambda storage mode |
diff --git a/docs/services/dynamodb.md b/docs/services/dynamodb.md
index 2124576bf3..ef2a88516c 100644
--- a/docs/services/dynamodb.md
+++ b/docs/services/dynamodb.md
@@ -230,6 +230,8 @@ region is rejected.
| `FLOCI_SERVICES_DYNAMODB_ENABLED` | `true` | Enable or disable the service |
| `FLOCI_STORAGE_SERVICES_DYNAMODB_MODE` | *(global default)* | Storage mode override for DynamoDB (`memory`, `persistent`, `hybrid`, `wal`) |
| `FLOCI_STORAGE_SERVICES_DYNAMODB_FLUSH_INTERVAL_MS` | `5000` | Flush interval for `hybrid`/`wal` storage modes (milliseconds) |
+| `FLOCI_STORAGE_SERVICES_DYNAMODB_ITEM_LAYOUT` | `single-file` | How items are laid out on disk: `single-file` or `per-table`. See [Item layout on disk](#item-layout-on-disk) |
+| `FLOCI_STORAGE_SERVICES_DYNAMODB_ITEM_SHARDS` | `1` | With the `per-table` layout, how many files each table is split into |
| `FLOCI_SERVICES_DYNAMODB_VECTOR_INDEX_ALLOCATION_SECONDS` | `4` | Seconds a vector index added by `UpdateTable` spends in resource allocation |
| `FLOCI_SERVICES_DYNAMODB_VECTOR_INDEX_BACKFILL_SECONDS` | `10` | Seconds that index then spends backfilling before it goes `ACTIVE` |
| `FLOCI_SERVICES_DYNAMODB_LATEST_RESTORABLE_LAG_SECONDS` | `300` | Seconds `LatestRestorableDateTime` of point-in-time recovery trails the current time. Set `0` to restore the latest writes with `UseLatestRestorableTime` |
@@ -246,6 +248,20 @@ For write-heavy workloads under persistent setups, configuring `FLOCI_STORAGE_SE
- `wal`: Uses an append-only write-ahead log with background compaction.
- `hybrid`: Keeps data in memory with periodic asynchronous disk flushes controlled by `FLOCI_STORAGE_SERVICES_DYNAMODB_FLUSH_INTERVAL_MS`.
+### Item layout on disk
+
+With the default `single-file` layout, every table's items live in one `dynamodb-items.json`, and any item write saves all of them: `persistent` rewrites the file on the write, `hybrid` on the next flush, and `wal` appends the whole table to its log and rewrites the whole snapshot on compaction. A small table that changes often, such as a heartbeat table, then keeps rewriting every large table beside it.
+
+`FLOCI_STORAGE_SERVICES_DYNAMODB_ITEM_LAYOUT=per-table` stores items under `dynamodb-items/` instead, as `FLOCI_STORAGE_SERVICES_DYNAMODB_ITEM_SHARDS` files per table (one by default). An item belongs to the file chosen by a hash of its key, and a write saves only the items it touched:
+
+| Mode | What an item write costs on disk |
+|---|---|
+| `persistent` | Rewrites the one file that holds the item |
+| `hybrid` | Marks that file dirty; dirty files are rewritten on the flush interval |
+| `wal` | Not supported yet: the items stay in the `single-file` layout, and a warning is logged at startup |
+
+On first start with `per-table`, an existing `dynamodb-items.json` or a `wal`-mode `dynamodb-items-snapshot.json` and `dynamodb-items.wal` is migrated into the directory, and those files are renamed with a `.migrated` suffix. When both kinds exist, the one written last is migrated. The migration is written to a staging directory and moved in place in one step, so a crash leaves either no directory or a complete one. If it cannot be written, Floci fails to start and the old files stay untouched. Once the directory holds data, old files are ignored. Switching back to `single-file` does not read the directory. A write that cannot be saved where its mode promises fails with an error, instead of only being logged: in `persistent` mode the shard file must be written before the write returns. The change stays in memory and is written by the next flush. Changing the shard count rewrites the files on the next start. The other DynamoDB stores (tables, backups, point-in-time recovery history, exports, imports) keep their single files.
+
## Examples
```bash
diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
index 5e73ae33ea..0a70fd0a06 100644
--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
@@ -435,6 +435,21 @@ public interface EmulatorConfig {
@WithDefault("5000")
long flushIntervalMs();
+
+ /**
+ * How table items are laid out on disk: {@code single-file} keeps every table in one
+ * {@code dynamodb-items.json}, rewritten on any change; {@code per-table} writes one file
+ * per table under {@code dynamodb-items/}, so a change rewrites only its own table.
+ */
+ @WithDefault("single-file")
+ String itemLayout();
+
+ /**
+ * With the {@code per-table} layout, how many files each table is split into. An item's
+ * write rewrites only its own file. Changing it rewrites the files on the next start.
+ */
+ @WithDefault("1")
+ int itemShards();
}
interface SnsStorageConfig {
diff --git a/src/main/java/io/github/hectorvent/floci/core/storage/AccountAwareStorageBackend.java b/src/main/java/io/github/hectorvent/floci/core/storage/AccountAwareStorageBackend.java
index 70331379ba..230f06c012 100644
--- a/src/main/java/io/github/hectorvent/floci/core/storage/AccountAwareStorageBackend.java
+++ b/src/main/java/io/github/hectorvent/floci/core/storage/AccountAwareStorageBackend.java
@@ -426,6 +426,38 @@ public class AccountAwareStorageBackend<V> implements StorageBackend<String, V>
delegate.delete(accountId + "/" + key);
}
+ // --- Entry writes, for map-valued stores whose backend persists one entry at a time ---
+
+ /** Whether the backend persists one map entry at a time; the entry methods require it. */
+ public boolean writesEntries() {
+ return delegate instanceof MapEntryWriter<?>;
+ }
+
+ public <E> void putEntry(String key, String entryKey, E entryValue) {
+ putEntryForAccount(prefix(), key, entryKey, entryValue);
+ }
+
+ public void removeEntry(String key, String entryKey) {
+ removeEntryForAccount(prefix(), key, entryKey);
+ }
+
+ public <E> void putEntryForAccount(String accountId, String key, String entryKey, E entryValue) {
+ this.<E>entryWriter().putEntry(accountId + "/" + key, entryKey, entryValue);
+ trackOwner(accountId);
+ }
+
+ public void removeEntryForAccount(String accountId, String key, String entryKey) {
+ entryWriter().removeEntry(accountId + "/" + key, entryKey);
+ }
+
+ @SuppressWarnings("unchecked")
+ private <E> MapEntryWriter<E> entryWriter() {
+ if (delegate instanceof MapEntryWriter<?> writer) {
+ return (MapEntryWriter<E>) writer;
+ }
+ throw new UnsupportedOperationException("This store does not persist single map entries");
+ }
+
/** Deletes the entry stored under {@code rawKey} exactly, as {@link #scanAllAccountsWithRawKeys} reports it. */
public void deleteRaw(String rawKey) {
delegate.delete(rawKey);
diff --git a/src/main/java/io/github/hectorvent/floci/core/storage/MapEntryWriter.java b/src/main/java/io/github/hectorvent/floci/core/storage/MapEntryWriter.java
new file mode 100644
index 0000000000..cc18739335
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/core/storage/MapEntryWriter.java
@@ -0,0 +1,17 @@
+package io.github.hectorvent.floci.core.storage;
+
+/**
+ * A store whose values are maps and that can persist one entry of a value without rewriting the
+ * rest of it. A caller that changes a single entry of a large value (one DynamoDB item of a table)
+ * uses these instead of {@link StorageBackend#put}, which hands the backend the whole value.
+ *
+ * @param <E> the type of one map entry's value
+ */
+public interface MapEntryWriter<E> {
+
+ /** Sets {@code entryKey} in the map stored under {@code key}, creating the map when absent. */
+ void putEntry(String key, String entryKey, E entryValue);
+
+ /** Removes {@code entryKey} from the map stored under {@code key}; a missing map stays missing. */
+ void removeEntry(String key, String entryKey);
+}
diff --git a/src/main/java/io/github/hectorvent/floci/core/storage/PartitionedFileStorage.java b/src/main/java/io/github/hectorvent/floci/core/storage/PartitionedFileStorage.java
new file mode 100644
index 0000000000..921cd8d533
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/core/storage/PartitionedFileStorage.java
@@ -0,0 +1,689 @@
+package io.github.hectorvent.floci.core.storage;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+import com.fasterxml.jackson.databind.ObjectWriter;
+import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
+import org.jboss.logging.Logger;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.DirectoryStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardCopyOption;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HexFormat;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * File storage for a store whose values are maps, written one file per shard of each key. A
+ * change to one map entry rewrites the one shard file that holds it, so a store holding one large
+ * value and one small, busy value (a large DynamoDB table next to a heartbeat table) no longer
+ * rewrites the large one for every change to the small one, as {@link PersistentStorage} and
+ * {@link HybridStorage} do with their single file. With more than one shard, a change to a large
+ * value rewrites only a fraction of it.
+ *
+ * <p>An entry's shard is {@code floorMod(entryKey.hashCode(), shards)}; {@link String#hashCode}
+ * is fixed by the language, so it is stable across runs. Each file records its own key, so a file
+ * name only has to be unique and safe: it is a readable prefix of the key, a hash of it and, with
+ * more than one shard, the shard's number and the shard count. Files written under another shard
+ * count are merged on load, rewritten under the current count and then removed.
+ *
+ * <p>A {@link Mode#SYNC} store writes a shard's file before the mutating call returns, like
+ * {@link PersistentStorage}. A {@link Mode#PERIODIC} store marks the shard dirty and writes every
+ * dirty shard on the flush interval, like {@link HybridStorage}.
+ *
+ * <p>While the directory holds no data, a legacy single-file store (the {@code Map<String,
+ * Map<String, E>>} file the other backends write, or a {@code wal} snapshot and log) is migrated:
+ * its keys are written to a staging directory that is moved in place in one step, and the legacy
+ * files are then renamed with a {@code .migrated} suffix so they are never read again.
+ *
+ * @param <E> the type of one map entry's value
+ */
+public class PartitionedFileStorage<E> implements StorageBackend<String, Map<String, E>>, MapEntryWriter<E> {
+
+ private static final Logger LOG = Logger.getLogger(PartitionedFileStorage.class);
+
+ static final String FILE_SUFFIX = ".json";
+ static final String MIGRATING_SUFFIX = ".migrating";
+ private static final String KEY_FIELD = "key";
+ private static final String ENTRIES_FIELD = "entries";
+ private static final int READABLE_PREFIX_LENGTH = 80;
+
+ /** When a change reaches disk. */
+ public enum Mode {
+ /** Its shard file is rewritten before the mutating call returns. */
+ SYNC,
+ /** Dirty shard files are rewritten on the flush interval. */
+ PERIODIC
+ }
+
+ /** One shard of one key: the unit that is written to a file. */
+ private record ShardId(String key, int shard) {}
+
+ /** The value stored under one key, split into its shards. */
+ private static final class Shards<E> {
+ private final List<ConcurrentHashMap<String, E>> parts;
+
+ Shards(int count) {
+ List<ConcurrentHashMap<String, E>> created = new ArrayList<>(count);
+ for (int i = 0; i < count; i++) {
+ created.add(new ConcurrentHashMap<>());
+ }
+ this.parts = List.copyOf(created);
+ }
+
+ ConcurrentHashMap<String, E> part(int shard) {
+ return parts.get(shard);
+ }
+
+ Map<String, E> merged() {
+ Map<String, E> merged = new HashMap<>();
+ parts.forEach(merged::putAll);
+ return Collections.unmodifiableMap(merged);
+ }
+ }
+
+ private final ConcurrentHashMap<String, Shards<E>> store = new ConcurrentHashMap<>();
+ private final Set<ShardId> dirty = ConcurrentHashMap.newKeySet();
+ private final ConcurrentHashMap<ShardId, Object> fileLocks = new ConcurrentHashMap<>();
+ private final Path directory;
+ private final LegacyStore<E> legacy;
+ private final int shardCount;
+ private final Mode mode;
+ private final ObjectMapper mapper;
+ private final ObjectReader entriesReader;
+ private final ObjectWriter entriesWriter;
+ private final ScheduledExecutorService scheduler;
+ // Shard writes hold the read lock, and load and clear hold the write lock, so a write that
+ // read a shard before a clear cannot put its file back afterwards, and the flush schedule,
+ // which starts with the constructor, cannot write while load is filling the store.
+ private final ReentrantReadWriteLock fileLock = new ReentrantReadWriteLock();
+ // Runs inside a shard write after the shard was read and before its file is replaced: the
+ // moment a clear() that slipped past the lock would be undone. Tests use it to stage that.
+ private volatile Runnable beforeShardWrite = () -> { };
+ // Runs on load after superseded files were rewritten under their current names and before
+ // the old files are removed. Tests use it to make only the removal fail.
+ private volatile Runnable beforeSupersededDelete = () -> { };
+
+ /**
+ * @param directory where the shard files live
+ * @param legacyFile a plain single-file store to migrate on first load, or {@code null}
+ * @param entryType the type of one map entry's value
+ * @param shardCount how many files each key is split into, at least one
+ * @param mode when a change reaches disk
+ * @param flushIntervalMs the flush interval of a {@link Mode#PERIODIC} store
+ */
+ public PartitionedFileStorage(Path directory, Path legacyFile, Class<E> entryType, int shardCount,
+ Mode mode, long flushIntervalMs) {
+ this(directory, legacyFile == null ? null : LegacyStore.jsonFile(legacyFile, entryType), entryType,
+ shardCount, mode, flushIntervalMs);
+ }
+
+ /**
+ * @param legacy a single-file store to migrate into the directory while it holds no data,
+ * or {@code null}
+ */
+ public PartitionedFileStorage(Path directory, LegacyStore<E> legacy, Class<E> entryType, int shardCount,
+ Mode mode, long flushIntervalMs) {
+ if (shardCount < 1) {
+ throw new IllegalArgumentException("shardCount must be at least 1, was " + shardCount);
+ }
+ this.directory = directory;
+ this.legacy = legacy;
+ this.shardCount = shardCount;
+ this.mode = mode;
+ this.mapper = new ObjectMapper();
+ this.mapper.registerModule(new JavaTimeModule());
+ JavaType entriesType = mapper.getTypeFactory().constructMapType(Map.class, String.class, entryType);
+ this.entriesReader = mapper.readerFor(entriesType);
+ this.entriesWriter = mapper.writerFor(entriesType);
+ this.scheduler = mode == Mode.SYNC
+ ? null
+ : StorageFiles.scheduleEvery("partitioned-storage-flush", flushIntervalMs, this::flushQuietly);
+ }
+
+ // --- StorageBackend ---
+
+ @Override
+ public void put(String key, Map<String, E> value) {
+ applyPut(key, value);
+ persistChange(allShardsOf(key));
+ }
+
+ @Override
+ public void putAll(Map<String, Map<String, E>> entries) {
+ entries.forEach(this::put);
+ }
+
+ /** A read-only copy of the stored map. */
+ @Override
+ public Optional<Map<String, E>> get(String key) {
+ Shards<E> value = store.get(key);
+ return value == null ? Optional.empty() : Optional.of(value.merged());
+ }
+
+ @Override
+ public void delete(String key) {
+ applyDelete(key);
+ persistChange(allShardsOf(key));
+ }
+
+ @Override
+ public List<Map<String, E>> scan(Predicate<String> keyFilter) {
+ return store.entrySet().stream()
+ .filter(e -> keyFilter.test(e.getKey()))
+ .map(e -> e.getValue().merged())
+ .collect(Collectors.toCollection(ArrayList::new));
+ }
+
+ @Override
+ public Set<String> keys() {
+ return Collections.unmodifiableSet(store.keySet());
+ }
+
+ /**
+ * Writes every dirty shard.
+ *
+ * @throws UncheckedIOException when a shard could not be written; it stays dirty and is
+ * written again by the next flush
+ */
+ @Override
+ public void flush() {
+ int failed = writeDirtyShards();
+ if (failed > 0) {
+ throw new UncheckedIOException(new IOException(
+ "Could not write " + failed + " shards of " + directory + "; see the errors logged above"));
+ }
+ }
+
+ @Override
+ public synchronized void load() {
+ // The flush schedule starts with the constructor: hold it off while the store is filled.
+ fileLock.writeLock().lock();
+ try {
+ store.clear();
+ dirty.clear();
+ migrateLegacyFileIfNeeded();
+ if (Files.isDirectory(directory)) {
+ loadDirectory();
+ }
+ } finally {
+ fileLock.writeLock().unlock();
+ }
+ }
+
+ @Override
+ public synchronized void clear() {
+ fileLock.writeLock().lock();
+ try {
+ store.clear();
+ dirty.clear();
+ if (Files.isDirectory(directory)) {
+ try (DirectoryStream<Path> files = Files.newDirectoryStream(directory, "*" + FILE_SUFFIX)) {
+ for (Path file : files) {
+ Files.deleteIfExists(file);
+ }
+ } catch (IOException e) {
+ LOG.errorv(e, "Failed to clear {0}", directory);
+ }
+ }
+ } finally {
+ fileLock.writeLock().unlock();
+ }
+ }
+
+ // --- MapEntryWriter ---
+
+ @Override
+ public void putEntry(String key, String entryKey, E entryValue) {
+ persistChange(List.of(applyPutEntry(key, entryKey, entryValue)));
+ }
+
+ @Override
+ public void removeEntry(String key, String entryKey) {
+ ShardId id = applyRemoveEntry(key, entryKey);
+ if (id != null) {
+ persistChange(List.of(id));
+ }
+ }
+
+ // ---
+
+ /** Stops the flush schedule and writes what is still dirty. */
+ public void shutdown() {
+ if (scheduler != null) {
+ StorageFiles.stop(scheduler);
+ }
+ flush();
+ }
+
+ /**
+ * Writes the changed shards of a sync store before the call returns; a periodic store leaves
+ * them to the next flush.
+ *
+ * @throws UncheckedIOException when a shard could not be written; the change stays in memory
+ * and dirty, so a later flush writes it
+ */
+ private void persistChange(List<ShardId> changed) {
+ if (mode != Mode.SYNC) {
+ return;
+ }
+ int failed = 0;
+ for (ShardId id : changed) {
+ if (!writeShard(id)) {
+ failed++;
+ }
+ }
+ if (failed > 0) {
+ throw new UncheckedIOException(new IOException(
+ "A change to " + changed.getFirst().key() + " in " + directory
+ + " is in memory but could not be written; see the errors logged above"));
+ }
+ }
+
+ // The apply methods change memory and mark what they change dirty.
+
+ private ShardId applyPutEntry(String key, String entryKey, E entryValue) {
+ ShardId id = new ShardId(key, shardOf(entryKey));
+ store.computeIfAbsent(key, ignored -> new Shards<>(shardCount)).part(id.shard()).put(entryKey, entryValue);
+ dirty.add(id);
+ return id;
+ }
+
+ /** The shard the entry was removed from, or {@code null} when there was no such entry. */
+ private ShardId applyRemoveEntry(String key, String entryKey) {
+ ShardId id = new ShardId(key, shardOf(entryKey));
+ Shards<E> value = store.get(key);
+ if (value == null || value.part(id.shard()).remove(entryKey) == null) {
+ return null;
+ }
+ dirty.add(id);
+ return id;
+ }
+
+ private void applyPut(String key, Map<String, E> value) {
+ store.put(key, split(value));
+ markAll(key);
+ }
+
+ private void applyDelete(String key) {
+ store.remove(key);
+ markAll(key);
+ }
+
+ /** Writes every dirty shard and returns how many could not be written. */
+ private int writeDirtyShards() {
+ int failed = 0;
+ for (ShardId id : List.copyOf(dirty)) {
+ if (!writeShard(id)) {
+ failed++;
+ }
+ }
+ return failed;
+ }
+
+ void setBeforeShardWrite(Runnable hook) {
+ this.beforeShardWrite = hook;
+ }
+
+ void setBeforeSupersededDelete(Runnable hook) {
+ this.beforeSupersededDelete = hook;
+ }
+
+ /** The file that holds {@code entryKey} of {@code key}. */
+ Path fileFor(String key, String entryKey) {
+ return fileFor(new ShardId(key, shardOf(entryKey)));
+ }
+
+ /** The file of a single-shard store that holds {@code key}. */
+ Path fileFor(String key) {
+ return fileFor(new ShardId(key, 0));
+ }
+
+ int shardOf(String entryKey) {
+ return Math.floorMod(entryKey.hashCode(), shardCount);
+ }
+
+ private Path fileFor(ShardId id) {
+ return directory.resolve(fileNameFor(id.key(), id.shard(), shardCount));
+ }
+
+ static String fileNameFor(String key, int shard, int shardCount) {
+ StringBuilder name = new StringBuilder();
+ for (int i = 0; i < key.length() && name.length() < READABLE_PREFIX_LENGTH; i++) {
+ char c = key.charAt(i);
+ boolean safe = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9')
+ || c == '.' || c == '-' || c == '_';
+ name.append(safe ? c : '_');
+ }
+ name.append('-').append(shortHash(key));
+ if (shardCount > 1) {
+ name.append('.').append(shard).append("-of-").append(shardCount);
+ }
+ return name.append(FILE_SUFFIX).toString();
+ }
+
+ private static String shortHash(String key) {
+ try {
+ byte[] digest = MessageDigest.getInstance("SHA-256").digest(key.getBytes(StandardCharsets.UTF_8));
+ return HexFormat.of().formatHex(digest, 0, 8);
+ } catch (NoSuchAlgorithmException e) {
+ throw new IllegalStateException("SHA-256 is unavailable", e);
+ }
+ }
+
+ private Shards<E> split(Map<String, E> value) {
+ Shards<E> shards = new Shards<>(shardCount);
+ value.forEach((entryKey, entryValue) -> shards.part(shardOf(entryKey)).put(entryKey, entryValue));
+ return shards;
+ }
+
+ private void markAll(String key) {
+ for (int shard = 0; shard < shardCount; shard++) {
+ dirty.add(new ShardId(key, shard));
+ }
+ }
+
+ private List<ShardId> allShardsOf(String key) {
+ List<ShardId> ids = new ArrayList<>(shardCount);
+ for (int shard = 0; shard < shardCount; shard++) {
+ ids.add(new ShardId(key, shard));
+ }
+ return ids;
+ }
+
+ /**
+ * The flush timer's task. It must not throw: an exception escaping a scheduled task cancels
+ * every later run, so one bad flush would stop the store from ever writing again. A failure
+ * is logged here and its shards stay dirty, so the next run retries them; an explicit
+ * {@link #flush()} (shutdown included) reports the same failure to its caller.
+ */
+ private void flushQuietly() {
+ try {
+ flush();
+ } catch (RuntimeException e) {
+ LOG.errorv(e, "Scheduled flush of {0} failed; its unwritten shards are retried on the next one",
+ directory);
+ }
+ }
+
+ /**
+ * Writes one shard's file, or deletes it when the shard is empty or its key is gone. The dirty
+ * mark is cleared before the shard is read, so a change racing the write marks it again and
+ * is written next.
+ *
+ * @return {@code false} when the file could not be written; the shard is then dirty again
+ */
+ private boolean writeShard(ShardId id) {
+ // clear() holds the write side, so a write that read a shard before the clear cannot put
+ // its file back after the clear removed it. Load already holds the write side, which lets
+ // it take the read side here.
+ fileLock.readLock().lock();
+ try {
+ return writeShardLocked(id);
+ } finally {
+ fileLock.readLock().unlock();
+ }
+ }
+
+ private boolean writeShardLocked(ShardId id) {
+ Object lock = fileLocks.computeIfAbsent(id, ignored -> new Object());
+ synchronized (lock) {
+ if (!dirty.remove(id)) {
+ return true;
+ }
+ Path file = fileFor(id);
+ try {
+ Shards<E> value = store.get(id.key());
+ ConcurrentHashMap<String, E> part = value == null ? null : value.part(id.shard());
+ if (part == null || part.isEmpty()) {
+ Files.deleteIfExists(file);
+ return true;
+ }
+ beforeShardWrite.run();
+ StorageFiles.writeAtomically(file, tempFile -> writeShardContent(tempFile, id.key(), part));
+ return true;
+ } catch (IOException e) {
+ dirty.add(id);
+ LOG.errorv(e, "Failed to persist shard {0} of {1} to {2}", id.shard(), id.key(), file);
+ return false;
+ }
+ }
+ }
+
+ private void writeShardFile(Path file, String key, Map<String, E> part) throws IOException {
+ StorageFiles.writeAtomically(file, tempFile -> writeShardContent(tempFile, key, part));
+ }
+
+ private void writeShardContent(Path file, String key, Map<String, E> part) throws IOException {
+ try (OutputStream out = Files.newOutputStream(file);
+ JsonGenerator generator = mapper.getFactory().createGenerator(out)) {
+ generator.writeStartObject();
+ generator.writeStringField(KEY_FIELD, key);
+ generator.writeFieldName(ENTRIES_FIELD);
+ entriesWriter.writeValue(generator, part);
+ generator.writeEndObject();
+ }
+ }
+
+ private void loadDirectory() {
+ List<Path> superseded = new ArrayList<>();
+ try (DirectoryStream<Path> files = Files.newDirectoryStream(directory, "*" + FILE_SUFFIX)) {
+ for (Path file : files) {
+ String key = readFile(file);
+ if (key != null && !isCurrentFileOf(file, key)) {
+ superseded.add(file);
+ markAll(key);
+ }
+ }
+ } catch (IOException e) {
+ LOG.errorv(e, "Failed to list {0}", directory);
+ }
+ LOG.infov("Loaded {0} keys from {1}", store.size(), directory);
+ if (superseded.isEmpty()) {
+ return;
+ }
+ // Written under another name or shard count: write each key under its current files
+ // first, and only then drop the old ones, so a crash in between loses nothing. An old
+ // file must not outlive this load: later writes change only the current files, so the
+ // next load would merge it back in and revive what they deleted or overwrote.
+ writeDirtyShards();
+ if (!dirty.isEmpty()) {
+ throw new IllegalStateException("Could not rewrite " + dirty.size() + " shards of " + directory
+ + " from superseded files; they are kept, fix the cause and restart");
+ }
+ beforeSupersededDelete.run();
+ for (Path file : superseded) {
+ try {
+ Files.deleteIfExists(file);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to remove superseded file " + file
+ + "; its entries are in the current files, remove it and restart", e);
+ }
+ }
+ LOG.infov("Rewrote {0} superseded files of {1} as {2} shards per key", superseded.size(), directory,
+ shardCount);
+ }
+
+ private boolean isCurrentFileOf(Path file, String key) {
+ for (int shard = 0; shard < shardCount; shard++) {
+ if (file.equals(fileFor(new ShardId(key, shard)))) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /** Merges one file into the store and returns its key, or {@code null} when it is unreadable. */
+ private String readFile(Path file) {
+ String key = null;
+ Map<String, E> entries = null;
+ try (JsonParser parser = mapper.getFactory().createParser(file.toFile())) {
+ if (parser.nextToken() != JsonToken.START_OBJECT) {
+ throw new IOException("not a JSON object");
+ }
+ while (parser.nextToken() == JsonToken.FIELD_NAME) {
+ String field = parser.currentName();
+ parser.nextToken();
+ switch (field) {
+ case KEY_FIELD -> key = parser.getValueAsString();
+ case ENTRIES_FIELD -> entries = entriesReader.readValue(parser);
+ default -> parser.skipChildren();
+ }
+ }
+ if (key == null || entries == null) {
+ throw new IOException("missing \"" + KEY_FIELD + "\" or \"" + ENTRIES_FIELD + "\"");
+ }
+ } catch (IOException e) {
+ StorageQuarantine.quarantine(file, e, LOG);
+ return null;
+ }
+ Shards<E> shards = store.computeIfAbsent(key, ignored -> new Shards<>(shardCount));
+ entries.forEach((entryKey, entryValue) -> shards.part(shardOf(entryKey)).put(entryKey, entryValue));
+ return key;
+ }
+
+ private void migrateLegacyFileIfNeeded() {
+ Path staging = directory.resolveSibling(directory.getFileName() + MIGRATING_SUFFIX);
+ // A staging directory left over is a migration that crashed before it was moved in place.
+ deleteRecursively(staging);
+ if (legacy == null) {
+ return;
+ }
+ List<Path> sources = legacy.existingFiles();
+ if (sources.isEmpty()) {
+ return;
+ }
+ if (holdsData(directory)) {
+ // Moved in place by a migration that crashed before renaming its sources, or written
+ // since: either way the directory is newer than the legacy files.
+ LOG.warnv("Ignoring {0}: {1} already holds data", sources, directory);
+ return;
+ }
+ Map<String, Map<String, E>> entries;
+ try {
+ entries = legacy.read();
+ } catch (IOException e) {
+ for (Path source : sources) {
+ StorageQuarantine.quarantine(source, e, LOG);
+ }
+ return;
+ }
+ // Every file is written to a staging directory that is moved in place in one step, so a
+ // crash or a failed write never leaves a directory holding only part of the legacy store.
+ try {
+ Files.createDirectories(staging);
+ for (Map.Entry<String, Map<String, E>> entry : entries.entrySet()) {
+ Shards<E> shards = split(entry.getValue());
+ for (int shard = 0; shard < shardCount; shard++) {
+ ConcurrentHashMap<String, E> part = shards.part(shard);
+ if (!part.isEmpty()) {
+ writeShardFile(staging.resolve(fileNameFor(entry.getKey(), shard, shardCount)),
+ entry.getKey(), part);
+ }
+ }
+ }
+ if (Files.isDirectory(directory)) {
+ // Empty (holdsData said so): make way for the move.
+ Files.delete(directory);
+ }
+ Files.move(staging, directory, StandardCopyOption.ATOMIC_MOVE);
+ } catch (IOException e) {
+ deleteRecursively(staging);
+ // Starting empty would let new writes fill the directory, after which the legacy
+ // files are ignored for good: refuse to start instead.
+ throw new UncheckedIOException("Failed to migrate " + sources + " into " + directory
+ + "; the legacy files are untouched", e);
+ }
+ for (Path source : sources) {
+ Path migrated = source.resolveSibling(source.getFileName() + ".migrated");
+ try {
+ Files.move(source, migrated, StandardCopyOption.REPLACE_EXISTING);
+ } catch (IOException e) {
+ LOG.errorv(e, "Migrated {0} into {1} but could not rename it", source, directory);
+ }
+ }
+ LOG.infov("Migrated {0} keys from {1} into {2}; the old files now end in .migrated",
+ entries.size(), sources, directory);
+ }
+
+ /** Whether {@code dir} holds anything at all. */
+ private static boolean holdsData(Path dir) {
+ if (!Files.isDirectory(dir)) {
+ return false;
+ }
+ try (DirectoryStream<Path> files = Files.newDirectoryStream(dir)) {
+ return files.iterator().hasNext();
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to list " + dir, e);
+ }
+ }
+
+ private static void deleteRecursively(Path dir) {
+ if (!Files.exists(dir)) {
+ return;
+ }
+ try (Stream<Path> paths = Files.walk(dir)) {
+ for (Path path : paths.sorted(Comparator.reverseOrder()).toList()) {
+ Files.delete(path);
+ }
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to remove " + dir, e);
+ }
+ }
+
+ /**
+ * A single-file store to migrate into an empty directory on load: the file a {@code
+ * persistent} or {@code hybrid} store writes, or the snapshot and log of a {@code wal} one.
+ */
+ public interface LegacyStore<E> {
+
+ /** The files the store occupies that exist now; empty when there is nothing to migrate. */
+ List<Path> existingFiles();
+
+ /** Every key and its map, read without changing what the files hold. */
+ Map<String, Map<String, E>> read() throws IOException;
+
+ /** The plain JSON file of a {@code persistent} or {@code hybrid} store. */
+ static <E> LegacyStore<E> jsonFile(Path file, Class<E> entryType) {
+ ObjectMapper mapper = new ObjectMapper();
+ mapper.registerModule(new JavaTimeModule());
+ JavaType type = mapper.getTypeFactory().constructMapType(Map.class,
+ mapper.getTypeFactory().constructType(String.class),
+ mapper.getTypeFactory().constructMapType(Map.class, String.class, entryType));
+ return new LegacyStore<>() {
+ @Override
+ public List<Path> existingFiles() {
+ return Files.exists(file) ? List.of(file) : List.of();
+ }
+
+ @Override
+ public Map<String, Map<String, E>> read() throws IOException {
+ return mapper.readValue(file.toFile(), type);
+ }
+ };
+ }
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/core/storage/StorageFactory.java b/src/main/java/io/github/hectorvent/floci/core/storage/StorageFactory.java
index 4f958b4f55..f78d17b48c 100644
--- a/src/main/java/io/github/hectorvent/floci/core/storage/StorageFactory.java
+++ b/src/main/java/io/github/hectorvent/floci/core/storage/StorageFactory.java
@@ -9,6 +9,9 @@ import jakarta.enterprise.inject.Instance;
import jakarta.inject.Inject;
import org.jboss.logging.Logger;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Collections;
@@ -17,6 +20,7 @@ import java.util.IdentityHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.stream.Stream;
/**
* Factory that creates {@link AccountAwareStorageBackend} instances based on configuration.
@@ -37,6 +41,7 @@ public class StorageFactory {
private final Map<Path, StorageBackend<?, ?>> backendsByPath = new HashMap<>();
private final List<HybridStorage<?, ?>> hybridBackends = new ArrayList<>();
private final List<WalStorage<?, ?>> walBackends = new ArrayList<>();
+ private final List<PartitionedFileStorage<?>> partitionedBackends = new ArrayList<>();
// Backends whose initial load has completed. create() loads each backend exactly once and
// loadAll() only picks up backends that were never loaded, so neither path replays a store
// that is already live or opens a second WAL writer on top of the first.
@@ -143,6 +148,122 @@ public class StorageFactory {
return backend;
}
+ /**
+ * Create an account-aware store whose values are maps, persisted under a directory named
+ * after {@code fileName} (without {@code .json}) as {@code shardCount} files per key, so a
+ * change to one entry rewrites only the file of its shard. Writers that change one map entry at a time use
+ * {@link AccountAwareStorageBackend#putEntry} and {@link AccountAwareStorageBackend#removeEntry}
+ * when {@link AccountAwareStorageBackend#writesEntries()} is true.
+ *
+ * <p>{@code persistent} writes a shard's file on every change and {@code hybrid} writes dirty
+ * shards on the flush interval. {@code memory} keeps the store in memory, and {@code wal}
+ * keeps its single-file {@link WalStorage} for now. An existing single-file store (a
+ * {@code wal}-mode snapshot and log included) is migrated into the directory on first load.
+ */
+ public synchronized <E> AccountAwareStorageBackend<Map<String, E>> createPartitioned(
+ String serviceName, String fileName, TypeReference<Map<String, Map<String, E>>> typeReference,
+ Class<E> entryType, int shardCount) {
+ String mode = resolveMode(serviceName);
+ if ("memory".equals(mode)) {
+ return create(serviceName, fileName, typeReference);
+ }
+ if ("wal".equals(mode)) {
+ LOG.warnv("The per-table layout does not support wal mode yet; storing {0} for service {1} "
+ + "in the single-file layout", fileName, serviceName);
+ return create(serviceName, fileName, typeReference);
+ }
+ Path basePath = Path.of(config.storage().persistentPath());
+ Path directory = basePath.resolve(fileName.replace(".json", ""));
+ StorageBackend<?, ?> existing = backendsByPath.get(directory);
+ if (existing != null) {
+ @SuppressWarnings("unchecked")
+ AccountAwareStorageBackend<Map<String, E>> typed = (AccountAwareStorageBackend<Map<String, E>>) existing;
+ return typed;
+ }
+ PartitionedFileStorage.Mode writeMode;
+ long interval;
+ switch (mode) {
+ case "persistent" -> {
+ writeMode = PartitionedFileStorage.Mode.SYNC;
+ interval = 0;
+ }
+ case "hybrid" -> {
+ writeMode = PartitionedFileStorage.Mode.PERIODIC;
+ interval = resolveFlushInterval(serviceName);
+ }
+ default -> throw new IllegalArgumentException("Unknown storage mode: " + mode);
+ }
+ PartitionedFileStorage<E> inner = new PartitionedFileStorage<>(directory,
+ legacyStore(basePath, fileName, typeReference, entryType), entryType, shardCount, writeMode,
+ interval);
+ partitionedBackends.add(inner);
+ LOG.infov("Storing {0} for service {1} in {2}, {3} file(s) per key", fileName, serviceName, directory,
+ shardCount);
+ AccountAwareStorageBackend<Map<String, E>> backend = new AccountAwareStorageBackend<>(
+ inner, requestContextInstance, config.defaultAccountId());
+ loadOnce(backend);
+ allBackends.add(backend);
+ backendsByPath.put(directory, backend);
+ return backend;
+ }
+
+ /**
+ * The single-file store to migrate into a partitioned store, whatever mode it last ran in:
+ * the plain file a {@code persistent} or {@code hybrid} store writes, or the snapshot and log
+ * of a {@link WalStorage}. When both exist, the one written last is the live one. Nothing is
+ * folded or deleted here; the partitioned store renames the files once its migration lands.
+ */
+ private static <E> PartitionedFileStorage.LegacyStore<E> legacyStore(
+ Path basePath, String fileName, TypeReference<Map<String, Map<String, E>>> typeReference,
+ Class<E> entryType) {
+ Path plainFile = basePath.resolve(fileName);
+ Path snapshot = basePath.resolve(fileName.replace(".json", "-snapshot.json"));
+ Path log = basePath.resolve(fileName.replace(".json", ".wal"));
+ PartitionedFileStorage.LegacyStore<E> plain = PartitionedFileStorage.LegacyStore.jsonFile(plainFile, entryType);
+ PartitionedFileStorage.LegacyStore<E> wal = new PartitionedFileStorage.LegacyStore<>() {
+ @Override
+ public List<Path> existingFiles() {
+ return Stream.of(snapshot, log).filter(Files::exists).toList();
+ }
+
+ @Override
+ public Map<String, Map<String, E>> read() {
+ return WalStorage.readContents(snapshot, log, typeReference);
+ }
+ };
+ return new PartitionedFileStorage.LegacyStore<>() {
+ private PartitionedFileStorage.LegacyStore<E> live() {
+ List<Path> walFiles = wal.existingFiles();
+ if (walFiles.isEmpty()) {
+ return plain;
+ }
+ if (!Files.exists(plainFile)) {
+ return wal;
+ }
+ return lastModified(plainFile) > walFiles.stream().mapToLong(StorageFactory::lastModified).max()
+ .orElse(Long.MIN_VALUE) ? plain : wal;
+ }
+
+ @Override
+ public List<Path> existingFiles() {
+ return live().existingFiles();
+ }
+
+ @Override
+ public Map<String, Map<String, E>> read() throws IOException {
+ return live().read();
+ }
+ };
+ }
+
+ private static long lastModified(Path file) {
+ try {
+ return Files.getLastModifiedTime(file).toMillis();
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to read the modification time of " + file, e);
+ }
+ }
+
/** Load every managed backend that has not been loaded yet. Safe to call repeatedly. */
public synchronized void loadAll() {
for (StorageBackend<?, ?> backend : allBackends) {
@@ -158,11 +279,16 @@ public class StorageFactory {
loadedBackends.add(backend);
}
- /** Flush all storage backends to disk. */
+ /**
+ * Flush all storage backends to disk. Every backend is flushed even when one fails; the first
+ * failure is then thrown, with the others attached as suppressed.
+ */
public synchronized void flushAll() {
+ List<RuntimeException> failures = new ArrayList<>();
for (StorageBackend<?, ?> backend : allBackends) {
- backend.flush();
+ runCollecting(backend::flush, failures);
}
+ throwFirst(failures);
}
/** Clear all storage backends. */
@@ -179,13 +305,37 @@ public class StorageFactory {
* every WAL backend with an open writer that nothing closes.
*/
public synchronized void shutdownAll() {
- flushAll();
+ List<RuntimeException> failures = new ArrayList<>();
+ runCollecting(this::flushAll, failures);
for (HybridStorage<?, ?> hybrid : hybridBackends) {
- hybrid.shutdown();
+ runCollecting(hybrid::shutdown, failures);
}
for (WalStorage<?, ?> wal : walBackends) {
- wal.shutdown();
+ runCollecting(wal::shutdown, failures);
+ }
+ for (PartitionedFileStorage<?> partitioned : partitionedBackends) {
+ runCollecting(partitioned::shutdown, failures);
+ }
+ throwFirst(failures);
+ }
+
+ private static void runCollecting(Runnable step, List<RuntimeException> failures) {
+ try {
+ step.run();
+ } catch (RuntimeException e) {
+ failures.add(e);
+ }
+ }
+
+ private static void throwFirst(List<RuntimeException> failures) {
+ if (failures.isEmpty()) {
+ return;
+ }
+ RuntimeException first = failures.getFirst();
+ for (RuntimeException other : failures.subList(1, failures.size())) {
+ first.addSuppressed(other);
}
+ throw first;
}
private String resolveMode(String serviceName) {
diff --git a/src/main/java/io/github/hectorvent/floci/core/storage/WalStorage.java b/src/main/java/io/github/hectorvent/floci/core/storage/WalStorage.java
index 4c1f4d0996..3c35521369 100644
--- a/src/main/java/io/github/hectorvent/floci/core/storage/WalStorage.java
+++ b/src/main/java/io/github/hectorvent/floci/core/storage/WalStorage.java
@@ -15,6 +15,7 @@ import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.util.ArrayList;
import java.util.Collections;
+import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
@@ -193,6 +194,30 @@ public class WalStorage<K, V> implements StorageBackend<K, V> {
closeWalWriter();
}
+ /**
+ * Reads the store a snapshot and its log hold, replaying the log over the snapshot, and
+ * leaves both files as they are: nothing is compacted or deleted.
+ */
+ static <K, V> Map<K, V> readContents(Path snapshotPath, Path walPath, TypeReference<Map<K, V>> typeReference) {
+ WalStorage<K, V> reader = new WalStorage<>(snapshotPath, walPath, typeReference, Long.MAX_VALUE);
+ boolean hadLog = Files.exists(walPath);
+ try {
+ reader.load();
+ return new HashMap<>(reader.store);
+ } finally {
+ StorageFiles.stop(reader.scheduler);
+ reader.closeWalWriter();
+ if (!hadLog) {
+ // load() opens the log for appends, creating it; this read must leave no file behind.
+ try {
+ Files.deleteIfExists(walPath);
+ } catch (IOException e) {
+ LOG.warnv(e, "Failed to remove the empty log {0} a read created", walPath);
+ }
+ }
+ }
+ }
+
private void compact() {
compactionLock.writeLock().lock();
try {
diff --git a/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbService.java b/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbService.java
index 62e7a3adeb..d7f01a754f 100644
--- a/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbService.java
@@ -53,6 +53,7 @@ import java.time.Duration;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
+import java.util.Collection;
import java.util.Collections;
import java.util.Comparator;
import java.util.HashMap;
@@ -169,8 +170,7 @@ public class DynamoDbService {
EmulatorConfig config) {
this(storageFactory.create("dynamodb", "dynamodb-tables.json",
new TypeReference<Map<String, TableDefinition>>() {}),
- storageFactory.create("dynamodb", "dynamodb-items.json",
- new TypeReference<Map<String, Map<String, JsonNode>>>() {}),
+ itemStore(storageFactory, config),
storageFactory.create("dynamodb", "dynamodb-exports.json",
new TypeReference<Map<String, ExportDescription>>() {}),
storageFactory.create("dynamodb", "dynamodb-imports.json",
@@ -185,6 +185,20 @@ public class DynamoDbService {
config.services().dynamodb().latestRestorableLagSeconds(), Clock.systemUTC());
}
+ /** The item store the configured {@code item-layout} selects; package-private for tests. */
+ static StorageBackend<String, Map<String, JsonNode>> itemStore(StorageFactory storageFactory,
+ EmulatorConfig config) {
+ TypeReference<Map<String, Map<String, JsonNode>>> type = new TypeReference<>() {};
+ String layout = Objects.requireNonNullElse(config.storage().services().dynamodb().itemLayout(), "single-file");
+ return switch (layout) {
+ case "single-file" -> storageFactory.create("dynamodb", "dynamodb-items.json", type);
+ case "per-table" -> storageFactory.createPartitioned("dynamodb", "dynamodb-items.json", type,
+ JsonNode.class, config.storage().services().dynamodb().itemShards());
+ default -> throw new IllegalArgumentException(
+ "Unknown floci.storage.services.dynamodb.item-layout: " + layout);
+ };
+ }
+
/** Package-private constructor for testing. */
DynamoDbService(StorageBackend<String, TableDefinition> tableStore) {
this(tableStore, null, null, null, new RegionResolver("us-east-1", "000000000000"), null, null, null, null); // partition-literal: test-shaped constructor default
@@ -316,8 +330,17 @@ public class DynamoDbService {
// scanAllAccountsRaw() returns every account's items already in the "accountId/
// region::tableName" key format itemsByTable expects.
if (itemStore instanceof AccountAwareStorageBackend<Map<String, JsonNode>> aware) {
- aware.scanAllAccountsRaw().forEach((rawKey, items) ->
- itemsByTable.put(rawKey, rekeyPersistedItems(persistedTables.get(rawKey), items)));
+ aware.scanAllAccountsRaw().forEach((rawKey, items) -> {
+ ConcurrentSkipListMap<String, JsonNode> rekeyed = rekeyPersistedItems(persistedTables.get(rawKey), items);
+ itemsByTable.put(rawKey, rekeyed);
+ int slash = rawKey.indexOf('/');
+ if (aware.writesEntries() && slash > 0 && !rekeyed.keySet().equals(items.keySet())) {
+ // Entry writes address items by their current key, so a store still holding
+ // older keys would keep an item that is later deleted under its new one.
+ aware.putForAccount(rawKey.substring(0, slash), rawKey.substring(slash + 1),
+ new HashMap<>(rekeyed));
+ }
+ });
return;
}
for (String key : itemStore.keys()) {
@@ -361,6 +384,50 @@ public class DynamoDbService {
}
}
+ /**
+ * Persists the current state of the given items of one table. A store that writes single map
+ * entries gets only those items; any other store gets the whole table, as before.
+ */
+ private void persistItems(String storageKey, Collection<String> itemKeys) {
+ AccountAwareStorageBackend<Map<String, JsonNode>> entryStore = entryWritingItemStore();
+ if (entryStore == null) {
+ persistItems(storageKey);
+ return;
+ }
+ ConcurrentSkipListMap<String, JsonNode> items = currentItems(storageKey, false);
+ if (items == null) {
+ itemStore.delete(storageKey);
+ return;
+ }
+ for (String itemKey : itemKeys) {
+ JsonNode item = items.get(itemKey);
+ if (item == null) {
+ entryStore.removeEntry(storageKey, itemKey);
+ } else {
+ entryStore.putEntry(storageKey, itemKey, item);
+ }
+ }
+ }
+
+ private void persistItem(String storageKey, String itemKey) {
+ persistItems(storageKey, List.of(itemKey));
+ }
+
+ /** Persists the given items of every table a multi-item write touched. */
+ private void persistItems(Collection<String> storageKeys, Map<String, Set<String>> itemKeysByTable) {
+ for (String storageKey : storageKeys) {
+ persistItems(storageKey, itemKeysByTable.getOrDefault(storageKey, Set.of()));
+ }
+ }
+
+ /** The item store when it persists one item at a time, otherwise {@code null}. */
+ private AccountAwareStorageBackend<Map<String, JsonNode>> entryWritingItemStore() {
+ if (itemStore instanceof AccountAwareStorageBackend<Map<String, JsonNode>> aware && aware.writesEntries()) {
+ return aware;
+ }
+ return null;
+ }
+
public TableDefinition createTable(String tableName,
List<KeySchemaElement> keySchema,
List<AttributeDefinition> attributeDefinitions,
@@ -1043,7 +1110,7 @@ public class DynamoDbService {
}
tableItems.put(itemKey, normalizedItem);
if (shouldPersist) {
- persistItems(storageKey);
+ persistItem(storageKey, itemKey);
}
LOG.debugv("Put item in {0}: key={1}", canonicalTableName, itemKey);
LOG.tracev("Put item in {0}: key={1} item={2}", canonicalTableName, itemKey, item);
@@ -1162,7 +1229,7 @@ public class DynamoDbService {
}
JsonNode removed = items.remove(itemKey);
if (shouldPersist) {
- persistItems(storageKey);
+ persistItem(storageKey, itemKey);
}
LOG.debugv("Deleted item from {0}: key={1}", canonicalTableName, itemKey);
LOG.tracev("Deleted item from {0}: key={1} removed={2}", canonicalTableName, itemKey, removed);
@@ -1398,7 +1465,7 @@ public class DynamoDbService {
}
items.put(itemKey, item);
if (shouldPersist) {
- persistItems(storageKey);
+ persistItem(storageKey, itemKey);
}
LOG.tracev("Updated item in {0}: key={1} updateExpression={2} item={3}",
canonicalTableName, itemKey, updateExpression, item);
@@ -1740,6 +1807,7 @@ public class DynamoDbService {
public BatchWriteResult batchWriteItem(Map<String, List<JsonNode>> requestItems, String region) {
// Pre-validate all write requests before applying any mutation to memory or streams
+ Map<String, Set<String>> requestedItemKeys = new HashMap<>();
for (Map.Entry<String, List<JsonNode>> entry : requestItems.entrySet()) {
String tableName = canonicalTableName(region, entry.getKey());
String storageKey = regionKey(region, tableName);
@@ -1769,6 +1837,7 @@ public class DynamoDbService {
throw new AwsException("ValidationException",
"Provided list of item keys contains duplicates", 400);
}
+ requestedItemKeys.computeIfAbsent(storageKey, ignored -> new HashSet<>()).add(itemKey);
}
}
@@ -1790,9 +1859,7 @@ public class DynamoDbService {
}
}
} finally {
- for (String storageKey : affectedStorageKeys) {
- persistItems(storageKey);
- }
+ persistItems(affectedStorageKeys, requestedItemKeys);
}
return new BatchWriteResult(Map.of());
}
@@ -1998,9 +2065,12 @@ public class DynamoDbService {
live.put(participant.itemKey(), value);
}
}
- for (String storageKey : affectedStorageKeys) {
- persistItems(storageKey);
+ Map<String, Set<String>> participantKeys = new HashMap<>();
+ for (TransactParticipant participant : toAcquire.keySet()) {
+ participantKeys.computeIfAbsent(participant.storageKey(), ignored -> new HashSet<>())
+ .add(participant.itemKey());
}
+ persistItems(affectedStorageKeys, participantKeys);
for (Runnable streamEvent : pendingStreamEvents) {
streamEvent.run();
}
@@ -3002,6 +3072,10 @@ public class DynamoDbService {
ConcurrentSkipListMap<String, JsonNode> items = scan.items();
int deletedForTable = 0;
+ // A store that writes single entries drops each item under its item lock, so a write
+ // that replaces the item right after cannot be overtaken by this removal.
+ AccountAwareStorageBackend<Map<String, JsonNode>> entryStore =
+ scan.accountId() == null ? null : entryWritingItemStore();
for (String itemKey : scan.itemKeys()) {
// A table deleted, or deleted and recreated, since the scan or during this sweep: its items are not this scan's.
if (itemsByTable.get(scan.rawKey()) != items) {
@@ -3013,7 +3087,18 @@ public class DynamoDbService {
return null;
}
pointInTimeRecovery.recordItem(scan.accountId(), scan.storageKey(), scan.table(), itemKey, current);
- return items.remove(itemKey);
+ JsonNode gone = items.remove(itemKey);
+ if (entryStore != null) {
+ // Under the table entry's lock, so a DeleteTable and a recreate cannot slip
+ // in and have this removal drop the new table's item of the same key.
+ itemsByTable.computeIfPresent(scan.rawKey(), (key, live) -> {
+ if (live == items) {
+ entryStore.removeEntryForAccount(scan.accountId(), scan.storageKey(), itemKey);
+ }
+ return live;
+ });
+ }
+ return gone;
});
if (removed == null) {
continue;
@@ -3029,7 +3114,7 @@ public class DynamoDbService {
true);
}
}
- if (deletedForTable > 0) {
+ if (deletedForTable > 0 && entryStore == null) {
// A DeleteTable during the loop detached this map; writing it back would revive its items.
// Persisting under the entry's lock holds off DeleteTable, which removes the entry before
// it deletes the stored items.
@@ -3039,8 +3124,8 @@ public class DynamoDbService {
}
return live;
});
- totalDeleted += deletedForTable;
}
+ totalDeleted += deletedForTable;
}
if (totalDeleted > 0) {
LOG.infov("TTL sweeper removed {0} expired items", totalDeleted);
diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml
index 9052ef1e0e..dae05ffb4d 100644
--- a/src/main/resources/application.yml
+++ b/src/main/resources/application.yml
@@ -209,6 +209,14 @@ floci:
flush-interval-ms: 5000
dynamodb:
flush-interval-ms: 5000
+ # single-file: every table in dynamodb-items.json, rewritten on any item change.
+ # per-table: files per table under dynamodb-items/, rewritten only where an item changed
+ # (persistent: on the write, hybrid: on the flush interval; wal keeps single-file for
+ # now). An existing single-file store is migrated on first start.
+ item-layout: single-file # FLOCI_STORAGE_SERVICES_DYNAMODB_ITEM_LAYOUT
+ # per-table only: files per table. A write rewrites only its item's file; changing the
+ # count rewrites the files on the next start.
+ item-shards: 1 # FLOCI_STORAGE_SERVICES_DYNAMODB_ITEM_SHARDS
sns:
flush-interval-ms: 5000
lambda:
diff --git a/src/test/java/io/github/hectorvent/floci/core/storage/PartitionedFileStorageTest.java b/src/test/java/io/github/hectorvent/floci/core/storage/PartitionedFileStorageTest.java
new file mode 100644
index 0000000000..2e3d0bf1c5
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/core/storage/PartitionedFileStorageTest.java
@@ -0,0 +1,462 @@
+package io.github.hectorvent.floci.core.storage;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.attribute.FileTime;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.*;
+import static org.junit.jupiter.api.Assumptions.assumeFalse;
+
+class PartitionedFileStorageTest {
+
+ private static final FileTime LONG_AGO = FileTime.from(Instant.parse("2020-01-01T00:00:00Z"));
+
+ @TempDir
+ Path tempDir;
+
+ private final List<PartitionedFileStorage<String>> opened = new ArrayList<>();
+
+ @AfterEach
+ void tearDown() {
+ opened.forEach(PartitionedFileStorage::shutdown);
+ }
+
+ private PartitionedFileStorage<String> open(PartitionedFileStorage.Mode mode) {
+ return open(1, mode);
+ }
+
+ private PartitionedFileStorage<String> open(int shards, PartitionedFileStorage.Mode mode) {
+ PartitionedFileStorage<String> storage = new PartitionedFileStorage<>(tempDir.resolve("items"),
+ tempDir.resolve("items.json"), String.class, shards, mode, 60_000);
+ storage.load();
+ opened.add(storage);
+ return storage;
+ }
+
+ private Set<String> fileNames() throws Exception {
+ try (Stream<Path> files = Files.list(tempDir.resolve("items"))) {
+ return files.map(f -> f.getFileName().toString()).collect(Collectors.toSet());
+ }
+ }
+
+ private static Map<String, String> entries(int count) {
+ Map<String, String> entries = new HashMap<>();
+ for (int i = 0; i < count; i++) {
+ entries.put("entry-" + i, "value-" + i);
+ }
+ return entries;
+ }
+
+ @Test
+ void anEntryWriteRewritesOnlyItsOwnShardFile() throws Exception {
+ PartitionedFileStorage<String> storage = open(8, PartitionedFileStorage.Mode.SYNC);
+ storage.put("big", entries(200));
+ assertEquals(8, fileNames().size());
+ for (String name : fileNames()) {
+ Files.setLastModifiedTime(tempDir.resolve("items").resolve(name), LONG_AGO);
+ }
+ Path touched = storage.fileFor("big", "entry-7");
+
+ storage.putEntry("big", "entry-7", "changed");
+ storage.removeEntry("big", "entry-7");
+ storage.putEntry("big", "entry-7", "changed-again");
+
+ for (String name : fileNames()) {
+ Path file = tempDir.resolve("items").resolve(name);
+ assertEquals(file.equals(touched), !LONG_AGO.equals(Files.getLastModifiedTime(file)), name);
+ }
+ Map<String, String> expected = entries(200);
+ expected.put("entry-7", "changed-again");
+ assertEquals(expected, open(8, PartitionedFileStorage.Mode.SYNC).get("big").orElseThrow());
+ }
+
+ @Test
+ void anEmptyShardHasNoFile() throws Exception {
+ PartitionedFileStorage<String> storage = open(16, PartitionedFileStorage.Mode.SYNC);
+ storage.putEntry("small", "only", "1");
+ assertEquals(Set.of(storage.fileFor("small", "only").getFileName().toString()), fileNames());
+
+ storage.removeEntry("small", "only");
+
+ assertTrue(fileNames().isEmpty());
+ }
+
+ @Test
+ void deletingAKeyRemovesEveryShardFile() throws Exception {
+ PartitionedFileStorage<String> storage = open(4, PartitionedFileStorage.Mode.PERIODIC);
+ storage.put("gone", entries(50));
+ storage.put("kept", entries(1));
+ storage.flush();
+ Set<String> keptFiles = Set.of(storage.fileFor("kept", "entry-0").getFileName().toString());
+
+ storage.delete("gone");
+ storage.flush();
+
+ assertEquals(keptFiles, fileNames());
+ }
+
+ @Test
+ void changingTheShardCountKeepsEveryEntryAndDropsTheOldFiles() throws Exception {
+ open(1, PartitionedFileStorage.Mode.SYNC).put("big", entries(100));
+
+ PartitionedFileStorage<String> four = open(4, PartitionedFileStorage.Mode.SYNC);
+ assertEquals(entries(100), four.get("big").orElseThrow());
+ assertEquals(4, fileNames().size());
+ assertTrue(fileNames().stream().allMatch(name -> name.contains("-of-4")), fileNames().toString());
+
+ PartitionedFileStorage<String> two = open(2, PartitionedFileStorage.Mode.PERIODIC);
+ assertEquals(entries(100), two.get("big").orElseThrow());
+ assertEquals(2, fileNames().size());
+ assertTrue(fileNames().stream().allMatch(name -> name.contains("-of-2")), fileNames().toString());
+ assertEquals(entries(100), open(2, PartitionedFileStorage.Mode.SYNC).get("big").orElseThrow());
+ }
+
+ @Test
+ void aShardCountBelowOneIsRejected() {
+ assertThrows(IllegalArgumentException.class, () -> new PartitionedFileStorage<>(tempDir.resolve("items"),
+ (Path) null, String.class, 0, PartitionedFileStorage.Mode.SYNC, 60_000));
+ }
+
+ @Test
+ void syncEntryWriteRewritesOnlyItsOwnKeysFile() throws Exception {
+ PartitionedFileStorage<String> storage = open(PartitionedFileStorage.Mode.SYNC);
+ storage.put("big", Map.of("a", "1", "b", "2"));
+ storage.putEntry("busy", "node-1", "beat-1");
+ Path bigFile = storage.fileFor("big");
+ Files.setLastModifiedTime(bigFile, LONG_AGO);
+
+ storage.putEntry("busy", "node-1", "beat-2");
+ storage.removeEntry("busy", "node-1");
+ storage.putEntry("busy", "node-2", "beat-1");
+
+ assertEquals(LONG_AGO, Files.getLastModifiedTime(bigFile));
+ PartitionedFileStorage<String> reloaded = open(PartitionedFileStorage.Mode.SYNC);
+ assertEquals(Map.of("node-2", "beat-1"), reloaded.get("busy").orElseThrow());
+ assertEquals(Map.of("a", "1", "b", "2"), reloaded.get("big").orElseThrow());
+ }
+
+ @Test
+ void periodicStoreWritesOnlyDirtyKeysOnFlush() throws Exception {
+ PartitionedFileStorage<String> storage = open(PartitionedFileStorage.Mode.PERIODIC);
+ storage.put("big", Map.of("a", "1"));
+ storage.putEntry("busy", "node-1", "beat-1");
+ assertFalse(Files.exists(storage.fileFor("big")), "a periodic store writes nothing before a flush");
+
+ storage.flush();
+ Path bigFile = storage.fileFor("big");
+ Files.setLastModifiedTime(bigFile, LONG_AGO);
+ storage.putEntry("busy", "node-1", "beat-2");
+ storage.flush();
+
+ assertEquals(LONG_AGO, Files.getLastModifiedTime(bigFile));
+ assertEquals(Map.of("node-1", "beat-2"), open(PartitionedFileStorage.Mode.PERIODIC).get("busy").orElseThrow());
+ }
+
+ @Test
+ void aFlushThatFailsIsRetriedOnTheNextFlush() throws Exception {
+ PartitionedFileStorage<String> storage = open(PartitionedFileStorage.Mode.PERIODIC);
+ storage.putEntry("other", "a", "1");
+ storage.flush();
+ storage.putEntry("busy", "node-1", "beat-1");
+ Path directory = tempDir.resolve("items");
+ StorageTestFiles.assumePosixPermissions(directory);
+ Set<PosixFilePermission> original = Files.getPosixFilePermissions(directory);
+ Files.setPosixFilePermissions(directory, PosixFilePermissions.fromString("r-xr-xr-x"));
+ try {
+ // A root user writes through the read-only mode, so there is nothing to test there.
+ assumeFalse(Files.isWritable(directory), "the directory stays writable for this user");
+ assertThrows(UncheckedIOException.class, storage::flush, "a failed flush is reported");
+ assertFalse(Files.exists(storage.fileFor("busy")));
+ } finally {
+ Files.setPosixFilePermissions(directory, original);
+ }
+
+ storage.flush();
+
+ assertEquals(Map.of("node-1", "beat-1"), open(PartitionedFileStorage.Mode.PERIODIC).get("busy").orElseThrow());
+ }
+
+ @Test
+ void aSyncWriteThatCannotReachDiskFailsAndIsWrittenByTheNextFlush() throws Exception {
+ PartitionedFileStorage<String> storage = open(PartitionedFileStorage.Mode.SYNC);
+ Path file = storage.fileFor("table");
+ // A non-empty directory where the write's temporary file goes makes that write fail.
+ Path blocker = file.resolveSibling(file.getFileName() + ".tmp");
+ Files.createDirectories(blocker);
+ Files.writeString(blocker.resolve("child"), "x");
+
+ assertThrows(UncheckedIOException.class, () -> storage.putEntry("table", "a", "1"));
+
+ assertEquals(Map.of("a", "1"), storage.get("table").orElseThrow(), "the change stays in memory");
+ Files.delete(blocker.resolve("child"));
+ Files.delete(blocker);
+ storage.flush();
+ assertEquals(Map.of("a", "1"), open(PartitionedFileStorage.Mode.SYNC).get("table").orElseThrow());
+ }
+
+ @Test
+ void deletingAKeyRemovesItsFile() {
+ PartitionedFileStorage<String> storage = open(PartitionedFileStorage.Mode.SYNC);
+ storage.put("gone", Map.of("a", "1"));
+ Path file = storage.fileFor("gone");
+ assertTrue(Files.exists(file));
+
+ storage.delete("gone");
+
+ assertFalse(Files.exists(file));
+ assertTrue(open(PartitionedFileStorage.Mode.SYNC).get("gone").isEmpty());
+ }
+
+ @Test
+ void removingAnEntryOfAMissingKeyDoesNotCreateIt() {
+ PartitionedFileStorage<String> storage = open(PartitionedFileStorage.Mode.SYNC);
+ storage.removeEntry("missing", "a");
+ assertTrue(storage.get("missing").isEmpty());
+ assertFalse(Files.exists(storage.fileFor("missing")));
+ }
+
+ @Test
+ void keysOfAnyLengthAndCharacterMapToSafeDistinctFiles() {
+ PartitionedFileStorage<String> storage = open(PartitionedFileStorage.Mode.SYNC);
+ String longKey = "000000000000/us-east-1::" + "t".repeat(255);
+ String otherLongKey = "000000000000/us-east-1::" + "t".repeat(254) + "u";
+ storage.put(longKey, Map.of("a", "1"));
+ storage.put(otherLongKey, Map.of("a", "2"));
+ storage.put("../escape", Map.of("a", "3"));
+
+ assertNotEquals(storage.fileFor(longKey), storage.fileFor(otherLongKey));
+ assertTrue(storage.fileFor(longKey).getFileName().toString().length() < 120);
+ assertEquals(tempDir.resolve("items"), storage.fileFor("../escape").getParent());
+ PartitionedFileStorage<String> reloaded = open(PartitionedFileStorage.Mode.SYNC);
+ assertEquals("1", reloaded.get(longKey).orElseThrow().get("a"));
+ assertEquals("2", reloaded.get(otherLongKey).orElseThrow().get("a"));
+ assertEquals("3", reloaded.get("../escape").orElseThrow().get("a"));
+ }
+
+ @Test
+ void firstLoadMigratesTheLegacySingleFile() throws Exception {
+ Path legacy = tempDir.resolve("items.json");
+ new ObjectMapper().writeValue(legacy.toFile(), Map.of(
+ "000000000000/us-east-1::a", Map.of("k1", "v1"),
+ "000000000000/us-east-1::b", Map.of("k2", "v2")));
+
+ PartitionedFileStorage<String> storage = open(PartitionedFileStorage.Mode.PERIODIC);
+
+ assertEquals(Map.of("k1", "v1"), storage.get("000000000000/us-east-1::a").orElseThrow());
+ assertFalse(Files.exists(legacy));
+ assertTrue(Files.exists(tempDir.resolve("items.json.migrated")));
+ assertTrue(Files.exists(storage.fileFor("000000000000/us-east-1::b")),
+ "migration writes the files at once, not on the next flush");
+ assertEquals(Map.of("k2", "v2"),
+ open(PartitionedFileStorage.Mode.SYNC).get("000000000000/us-east-1::b").orElseThrow());
+ }
+
+ @Test
+ void aMigrationThatCrashedInStagingIsRedoneInFull() throws Exception {
+ Path legacy = tempDir.resolve("items.json");
+ new ObjectMapper().writeValue(legacy.toFile(), Map.of("a", Map.of("k", "1"), "b", Map.of("k", "2")));
+ // What a crash mid-migration leaves: one key staged, nothing moved in place.
+ Path staging = Files.createDirectories(tempDir.resolve("items" + PartitionedFileStorage.MIGRATING_SUFFIX));
+ Files.writeString(staging.resolve("a.json"), "{\"key\":\"a\",\"entries\":{\"k\":\"1\"}}");
+
+ PartitionedFileStorage<String> storage = open(PartitionedFileStorage.Mode.SYNC);
+
+ assertEquals(Map.of("k", "1"), storage.get("a").orElseThrow());
+ assertEquals(Map.of("k", "2"), storage.get("b").orElseThrow());
+ assertFalse(Files.exists(staging));
+ assertFalse(Files.exists(legacy));
+ }
+
+ @Test
+ void anEmptyDirectoryLeftBehindStillMigratesTheLegacyFile() throws Exception {
+ Path legacy = tempDir.resolve("items.json");
+ new ObjectMapper().writeValue(legacy.toFile(), Map.of("a", Map.of("k", "1")));
+ Files.createDirectories(tempDir.resolve("items"));
+
+ assertEquals(Map.of("k", "1"), open(PartitionedFileStorage.Mode.SYNC).get("a").orElseThrow());
+ assertTrue(Files.exists(tempDir.resolve("items.json.migrated")));
+ }
+
+ @Test
+ void aMigrationThatCannotWriteFailsAndKeepsTheLegacyFile() throws Exception {
+ Path legacy = tempDir.resolve("items.json");
+ new ObjectMapper().writeValue(legacy.toFile(), Map.of("a", Map.of("k", "1")));
+ StorageTestFiles.assumePosixPermissions(tempDir);
+ Set<PosixFilePermission> original = Files.getPosixFilePermissions(tempDir);
+ Files.setPosixFilePermissions(tempDir, PosixFilePermissions.fromString("r-xr-xr-x"));
+ try {
+ // A root user writes through the read-only mode, so there is nothing to test there.
+ assumeFalse(Files.isWritable(tempDir), "the directory stays writable for this user");
+ PartitionedFileStorage<String> storage = new PartitionedFileStorage<>(tempDir.resolve("items"),
+ legacy, String.class, 4, PartitionedFileStorage.Mode.SYNC, 60_000);
+
+ assertThrows(UncheckedIOException.class, storage::load);
+ storage.shutdown();
+ } finally {
+ Files.setPosixFilePermissions(tempDir, original);
+ }
+ assertTrue(Files.exists(legacy));
+ assertFalse(Files.exists(tempDir.resolve("items")));
+ assertEquals(Map.of("k", "1"), open(4, PartitionedFileStorage.Mode.SYNC).get("a").orElseThrow());
+ }
+
+ @Test
+ void aLegacyFileIsIgnoredOnceTheDirectoryHoldsData() throws Exception {
+ PartitionedFileStorage<String> storage = open(PartitionedFileStorage.Mode.SYNC);
+ storage.put("a", Map.of("k", "new"));
+ Path legacy = tempDir.resolve("items.json");
+ new ObjectMapper().writeValue(legacy.toFile(), Map.of("a", Map.of("k", "old"), "b", Map.of("k", "old")));
+
+ PartitionedFileStorage<String> reloaded = open(PartitionedFileStorage.Mode.SYNC);
+
+ assertEquals(Map.of("k", "new"), reloaded.get("a").orElseThrow());
+ assertTrue(reloaded.get("b").isEmpty());
+ assertTrue(Files.exists(legacy), "a legacy file that was not migrated is left alone");
+ }
+
+ @Test
+ void aSupersededFileThatCannotBeRewrittenStopsTheLoad() throws Exception {
+ PartitionedFileStorage<String> storage = open(PartitionedFileStorage.Mode.SYNC);
+ storage.put("key", Map.of("a", "1", "b", "2"));
+ Path current = storage.fileFor("key");
+ Path old = current.resolveSibling("old-name.json");
+ Files.move(current, old);
+ // A non-empty directory where the rewrite's temporary file goes makes the rewrite fail.
+ Path blocker = current.resolveSibling(current.getFileName() + ".tmp");
+ Files.createDirectories(blocker);
+ Files.writeString(blocker.resolve("child"), "x");
+
+ PartitionedFileStorage<String> blocked = new PartitionedFileStorage<>(tempDir.resolve("items"),
+ tempDir.resolve("items.json"), String.class, 1, PartitionedFileStorage.Mode.SYNC, 60_000);
+ opened.add(blocked);
+ // Running on would let later writes change only the current file, and the next load
+ // would merge the old file back in and revive what they removed.
+ assertThrows(IllegalStateException.class, blocked::load);
+
+ assertTrue(Files.exists(old));
+ Files.delete(blocker.resolve("child"));
+ Files.delete(blocker);
+ assertEquals(Map.of("a", "1", "b", "2"), open(PartitionedFileStorage.Mode.SYNC).get("key").orElseThrow());
+ assertFalse(Files.exists(old));
+ }
+
+ @Test
+ void aSupersededFileThatCannotBeRemovedStopsTheLoad() throws Exception {
+ PartitionedFileStorage<String> storage = open(PartitionedFileStorage.Mode.SYNC);
+ storage.put("key", Map.of("a", "1", "b", "2"));
+ Path current = storage.fileFor("key");
+ Path old = current.resolveSibling("old-name.json");
+ Files.move(current, old);
+ Path directory = tempDir.resolve("items");
+ StorageTestFiles.assumePosixPermissions(directory);
+ Set<PosixFilePermission> original = Files.getPosixFilePermissions(directory);
+
+ PartitionedFileStorage<String> blocked = new PartitionedFileStorage<>(directory,
+ tempDir.resolve("items.json"), String.class, 1, PartitionedFileStorage.Mode.SYNC, 60_000);
+ opened.add(blocked);
+ blocked.setBeforeSupersededDelete(() -> {
+ // The keys are rewritten under their current names; now make removing the old file fail.
+ try {
+ Files.setPosixFilePermissions(directory, PosixFilePermissions.fromString("r-xr-xr-x"));
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+ });
+ try {
+ Throwable failure = assertThrows(RuntimeException.class, blocked::load);
+ // A root user deletes through the read-only mode, so there is nothing to test there.
+ assumeFalse(Files.isWritable(directory), "the directory stays writable for this user");
+ // Running on would let later writes change only the current file, and the next load
+ // would merge the old file back in and revive what they removed.
+ assertInstanceOf(UncheckedIOException.class, failure);
+ assertTrue(Files.exists(current), "the rewrite landed before the removal failed");
+ assertTrue(Files.exists(old));
+ } finally {
+ Files.setPosixFilePermissions(directory, original);
+ }
+
+ assertEquals(Map.of("a", "1", "b", "2"), open(PartitionedFileStorage.Mode.SYNC).get("key").orElseThrow());
+ assertFalse(Files.exists(old));
+ }
+
+ @Test
+ void aWriteThatReadItsShardBeforeAClearDoesNotBringItBack() throws Exception {
+ PartitionedFileStorage<String> storage = open(PartitionedFileStorage.Mode.PERIODIC);
+ storage.putEntry("table", "a", "1");
+ Thread[] clearer = new Thread[1];
+ storage.setBeforeShardWrite(() -> {
+ // The shard is read and its file is about to be written: a clear now must wait for
+ // the write, or the write puts the cleared file back.
+ storage.setBeforeShardWrite(() -> { });
+ clearer[0] = Thread.ofPlatform().start(storage::clear);
+ StorageTestThreads.awaitParkedOrDone(clearer[0]);
+ });
+
+ storage.flush();
+ clearer[0].join(TimeUnit.SECONDS.toMillis(10));
+
+ assertFalse(clearer[0].isAlive());
+ assertTrue(open(PartitionedFileStorage.Mode.SYNC).get("table").isEmpty());
+ }
+
+ @Test
+ void anUnreadableFileIsQuarantinedAndTheRestStillLoad() throws Exception {
+ PartitionedFileStorage<String> storage = open(PartitionedFileStorage.Mode.SYNC);
+ storage.put("good", Map.of("a", "1"));
+ Files.writeString(tempDir.resolve("items").resolve("broken-0000000000000000.json"), "{not json");
+
+ PartitionedFileStorage<String> reloaded = open(PartitionedFileStorage.Mode.SYNC);
+
+ assertEquals(Map.of("a", "1"), reloaded.get("good").orElseThrow());
+ try (Stream<Path> files = Files.list(tempDir.resolve("items"))) {
+ assertTrue(files.noneMatch(f -> f.getFileName().toString().equals("broken-0000000000000000.json")));
+ }
+ }
+
+ @Test
+ void aFileUnderAnOldNameIsRewrittenUnderItsCurrentName() throws Exception {
+ PartitionedFileStorage<String> storage = open(PartitionedFileStorage.Mode.SYNC);
+ storage.put("key", Map.of("a", "1"));
+ Path current = storage.fileFor("key");
+ Path old = current.resolveSibling("old-name.json");
+ Files.move(current, old);
+
+ PartitionedFileStorage<String> reloaded = open(PartitionedFileStorage.Mode.SYNC);
+
+ assertEquals(Map.of("a", "1"), reloaded.get("key").orElseThrow());
+ assertTrue(Files.exists(current));
+ assertFalse(Files.exists(old));
+ }
+
+ @Test
+ void clearRemovesEveryFile() throws Exception {
+ PartitionedFileStorage<String> storage = open(PartitionedFileStorage.Mode.SYNC);
+ storage.put("a", Map.of("x", "1"));
+ storage.put("b", Map.of("x", "2"));
+
+ storage.clear();
+
+ try (Stream<Path> files = Files.list(tempDir.resolve("items"))) {
+ assertEquals(0, files.count());
+ }
+ assertTrue(open(PartitionedFileStorage.Mode.SYNC).keys().isEmpty());
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/core/storage/StorageFactoryPartitionedTest.java b/src/test/java/io/github/hectorvent/floci/core/storage/StorageFactoryPartitionedTest.java
new file mode 100644
index 0000000000..d397d171ee
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/core/storage/StorageFactoryPartitionedTest.java
@@ -0,0 +1,244 @@
+package io.github.hectorvent.floci.core.storage;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.core.common.ServiceConfigAccess;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.attribute.FileTime;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.time.Instant;
+import java.util.Map;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.*;
+import static org.junit.jupiter.api.Assumptions.assumeFalse;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/** How {@link StorageFactory#createPartitioned} maps each storage mode onto a backend. */
+class StorageFactoryPartitionedTest {
+
+ private static final TypeReference<Map<String, Map<String, String>>> TYPE = new TypeReference<>() {};
+
+ @TempDir
+ Path tempDir;
+
+ @Test
+ void persistentModeWritesOneFilePerKeyAsItChanges() {
+ StorageFactory factory = newFactory("persistent");
+ AccountAwareStorageBackend<Map<String, String>> store =
+ factory.createPartitioned("svc", "svc-items.json", TYPE, String.class, 1);
+
+ assertTrue(store.writesEntries());
+ store.putEntry("table-a", "k", "v");
+ store.putEntry("table-b", "k", "v");
+
+ assertEquals(2, fileCount(tempDir.resolve("svc-items")));
+ assertFalse(Files.exists(tempDir.resolve("svc-items.json")));
+ factory.shutdownAll();
+ }
+
+ @Test
+ void hybridModeWritesDirtyKeysOnFlushAndShutdown() {
+ StorageFactory factory = newFactory("hybrid");
+ AccountAwareStorageBackend<Map<String, String>> store =
+ factory.createPartitioned("svc", "svc-items.json", TYPE, String.class, 1);
+
+ store.putEntry("table-a", "k", "v");
+ assertEquals(0, fileCount(tempDir.resolve("svc-items")));
+
+ factory.shutdownAll();
+ assertEquals(1, fileCount(tempDir.resolve("svc-items")));
+ AccountAwareStorageBackend<Map<String, String>> reopened = newFactory("hybrid")
+ .createPartitioned("svc", "svc-items.json", TYPE, String.class, 1);
+ assertEquals(Map.of("k", "v"), reopened.get("table-a").orElseThrow());
+ }
+
+ @Test
+ void anExistingSingleFileIsMigratedOnFirstUse() throws Exception {
+ new ObjectMapper().writeValue(tempDir.resolve("svc-items.json").toFile(),
+ Map.of("000000000000/table-a", Map.of("k", "v")));
+
+ AccountAwareStorageBackend<Map<String, String>> store = newFactory("persistent")
+ .createPartitioned("svc", "svc-items.json", TYPE, String.class, 1);
+
+ assertEquals(Map.of("k", "v"), store.get("table-a").orElseThrow());
+ assertTrue(Files.exists(tempDir.resolve("svc-items.json.migrated")));
+ }
+
+ @Test
+ void walModeKeepsTheSingleFileStoreForNow() {
+ AccountAwareStorageBackend<Map<String, String>> store = newFactory("wal")
+ .createPartitioned("svc", "svc-items.json", TYPE, String.class, 4);
+
+ assertFalse(store.writesEntries());
+ store.put("table-a", Map.of("k", "v"));
+ assertFalse(Files.exists(tempDir.resolve("svc-items")));
+ }
+
+ @Test
+ void hybridModeMigratesALegacyWalStoreIncludingItsUncompactedLog() {
+ WalStorage<String, Map<String, String>> legacy = new WalStorage<>(tempDir.resolve("svc-items-snapshot.json"),
+ tempDir.resolve("svc-items.wal"), TYPE, Long.MAX_VALUE);
+ legacy.load();
+ legacy.put("000000000000/table-a", Map.of("k", "v"));
+ legacy.flush();
+ legacy.put("000000000000/table-b", Map.of("k", "only-in-the-log"));
+
+ AccountAwareStorageBackend<Map<String, String>> store = newFactory("hybrid")
+ .createPartitioned("svc", "svc-items.json", TYPE, String.class, 1);
+
+ assertEquals(Map.of("k", "v"), store.get("table-a").orElseThrow());
+ assertEquals(Map.of("k", "only-in-the-log"), store.get("table-b").orElseThrow());
+ assertTrue(Files.exists(tempDir.resolve("svc-items.wal.migrated")));
+ }
+
+ @Test
+ void anEmptyDirectoryLeftBehindStillMigratesALegacyWalStore() throws Exception {
+ WalStorage<String, Map<String, String>> legacy = new WalStorage<>(tempDir.resolve("svc-items-snapshot.json"),
+ tempDir.resolve("svc-items.wal"), TYPE, Long.MAX_VALUE);
+ legacy.load();
+ legacy.put("000000000000/table-a", Map.of("k", "v"));
+ legacy.shutdown();
+ // What a crash right after the directory was created leaves behind.
+ Files.createDirectories(tempDir.resolve("svc-items"));
+
+ AccountAwareStorageBackend<Map<String, String>> store = newFactory("hybrid")
+ .createPartitioned("svc", "svc-items.json", TYPE, String.class, 1);
+
+ assertEquals(Map.of("k", "v"), store.get("table-a").orElseThrow());
+ }
+
+ @Test
+ void theLegacyStoreWrittenLastIsTheOneMigrated() throws Exception {
+ WalStorage<String, Map<String, String>> legacy = new WalStorage<>(tempDir.resolve("svc-items-snapshot.json"),
+ tempDir.resolve("svc-items.wal"), TYPE, Long.MAX_VALUE);
+ legacy.load();
+ legacy.put("000000000000/table-a", Map.of("k", "from-wal"));
+ legacy.shutdown();
+ Path plain = tempDir.resolve("svc-items.json");
+ new ObjectMapper().writeValue(plain.toFile(), Map.of("000000000000/table-a", Map.of("k", "from-plain")));
+ FileTime older = FileTime.from(Instant.parse("2020-01-01T00:00:00Z"));
+ Files.setLastModifiedTime(tempDir.resolve("svc-items-snapshot.json"), older);
+ Files.setLastModifiedTime(tempDir.resolve("svc-items.wal"), older);
+
+ AccountAwareStorageBackend<Map<String, String>> store = newFactory("hybrid")
+ .createPartitioned("svc", "svc-items.json", TYPE, String.class, 1);
+
+ assertEquals(Map.of("k", "from-plain"), store.get("table-a").orElseThrow());
+ assertTrue(Files.exists(tempDir.resolve("svc-items-snapshot.json")), "the older store is left alone");
+ }
+
+ @Test
+ void aMigrationThatCannotWriteLeavesTheLegacyLogInPlace() throws Exception {
+ StorageTestFiles.assumePosixPermissions(tempDir);
+ WalStorage<String, Map<String, String>> legacy = new WalStorage<>(tempDir.resolve("svc-items-snapshot.json"),
+ tempDir.resolve("svc-items.wal"), TYPE, Long.MAX_VALUE);
+ legacy.load();
+ legacy.put("000000000000/table-a", Map.of("k", "only-in-the-log"));
+ Path log = tempDir.resolve("svc-items.wal");
+ byte[] logged = Files.readAllBytes(log);
+ // A non-empty file where the staging directory goes makes the migration's first write fail.
+ Path staging = tempDir.resolve("svc-items" + PartitionedFileStorage.MIGRATING_SUFFIX);
+ Files.createDirectories(staging.resolve("blocker"));
+ Files.setPosixFilePermissions(staging, PosixFilePermissions.fromString("r-xr-xr-x"));
+ try {
+ assumeFalse(Files.isWritable(staging), "the directory stays writable for this user");
+ assertThrows(UncheckedIOException.class, () -> newFactory("hybrid")
+ .createPartitioned("svc", "svc-items.json", TYPE, String.class, 1));
+ } finally {
+ Files.setPosixFilePermissions(staging, PosixFilePermissions.fromString("rwxr-xr-x"));
+ }
+
+ assertArrayEquals(logged, Files.readAllBytes(log));
+ assertEquals(Map.of("k", "only-in-the-log"), newFactory("hybrid")
+ .createPartitioned("svc", "svc-items.json", TYPE, String.class, 1).get("table-a").orElseThrow());
+ legacy.shutdown();
+ }
+
+ @Test
+ void aBackendThatFailsToFlushDoesNotStopTheOthersShuttingDown() throws Exception {
+ StorageFactory factory = newFactory("hybrid");
+ AccountAwareStorageBackend<Map<String, String>> failing =
+ factory.createPartitioned("svc", "svc-a.json", TYPE, String.class, 1);
+ AccountAwareStorageBackend<Map<String, String>> healthy =
+ factory.createPartitioned("svc", "svc-b.json", TYPE, String.class, 1);
+ failing.putEntry("table", "k", "v");
+ healthy.putEntry("table", "k", "v");
+ Path blocker = tempDir.resolve("svc-a")
+ .resolve(PartitionedFileStorage.fileNameFor("000000000000/table", 0, 1) + ".tmp");
+ Files.createDirectories(blocker.resolve("child"));
+
+ assertThrows(UncheckedIOException.class, factory::shutdownAll);
+
+ assertEquals(1, fileCount(tempDir.resolve("svc-b")));
+ }
+
+ @Test
+ void memoryModeKeepsTheSingleValueStore() {
+ AccountAwareStorageBackend<Map<String, String>> store = newFactory("memory")
+ .createPartitioned("svc", "svc-items.json", TYPE, String.class, 1);
+
+ assertFalse(store.writesEntries());
+ store.put("table-a", Map.of("k", "v"));
+ assertFalse(Files.exists(tempDir.resolve("svc-items")));
+ }
+
+ @Test
+ void entriesOfTwoAccountsUnderOneKeyStaySeparate() {
+ StorageFactory factory = newFactory("persistent");
+ AccountAwareStorageBackend<Map<String, String>> store =
+ factory.createPartitioned("svc", "svc-items.json", TYPE, String.class, 4);
+
+ store.putEntryForAccount("111111111111", "table", "k", "one");
+ store.putEntryForAccount("222222222222", "table", "k", "two");
+ store.putEntryForAccount("222222222222", "table", "gone", "x");
+ store.removeEntryForAccount("111111111111", "table", "gone");
+ store.removeEntryForAccount("222222222222", "table", "gone");
+ factory.shutdownAll();
+
+ AccountAwareStorageBackend<Map<String, String>> reopened = newFactory("persistent")
+ .createPartitioned("svc", "svc-items.json", TYPE, String.class, 4);
+ assertEquals(Map.of(
+ "111111111111/table", Map.of("k", "one"),
+ "222222222222/table", Map.of("k", "two")), reopened.scanAllAccountsRaw());
+ }
+
+ @Test
+ void repeatCallsShareOneBackend() {
+ StorageFactory factory = newFactory("persistent");
+ assertSame(factory.createPartitioned("svc", "svc-items.json", TYPE, String.class, 1),
+ factory.createPartitioned("svc", "svc-items.json", TYPE, String.class, 1));
+ }
+
+ private static long fileCount(Path directory) {
+ if (!Files.isDirectory(directory)) {
+ return 0;
+ }
+ try (Stream<Path> files = Files.list(directory)) {
+ return files.filter(f -> f.getFileName().toString().endsWith(".json")).count();
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+ }
+
+ private StorageFactory newFactory(String mode) {
+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
+ when(config.defaultAccountId()).thenReturn("000000000000");
+ when(config.storage().persistentPath()).thenReturn(tempDir.toString());
+ when(config.storage().wal().compactionIntervalMs()).thenReturn(60_000L);
+ ServiceConfigAccess serviceConfigAccess = mock(ServiceConfigAccess.class);
+ when(serviceConfigAccess.storageMode(anyString())).thenReturn(mode);
+ when(serviceConfigAccess.storageFlushInterval(anyString())).thenReturn(60_000L);
+ return new StorageFactory(config, serviceConfigAccess);
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/core/storage/StorageTestFiles.java b/src/test/java/io/github/hectorvent/floci/core/storage/StorageTestFiles.java
new file mode 100644
index 0000000000..21ddfd16f4
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/core/storage/StorageTestFiles.java
@@ -0,0 +1,25 @@
+package io.github.hectorvent.floci.core.storage;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.attribute.PosixFileAttributeView;
+
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/** File system helpers for the storage tests. */
+final class StorageTestFiles {
+
+ private StorageTestFiles() {
+ }
+
+ /**
+ * Skips the calling test unless {@code path}'s file store supports POSIX permissions. Tests
+ * that make a directory read-only to force a write to fail call it first, because the POSIX
+ * permission calls throw on a file system without them, such as Windows.
+ */
+ static void assumePosixPermissions(Path path) throws IOException {
+ assumeTrue(Files.getFileStore(path).supportsFileAttributeView(PosixFileAttributeView.class),
+ "POSIX file permissions are not supported here");
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/core/storage/StorageTestThreads.java b/src/test/java/io/github/hectorvent/floci/core/storage/StorageTestThreads.java
new file mode 100644
index 0000000000..a80c10c2ab
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/core/storage/StorageTestThreads.java
@@ -0,0 +1,29 @@
+package io.github.hectorvent.floci.core.storage;
+
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.LockSupport;
+
+import static org.junit.jupiter.api.Assertions.fail;
+
+/** Thread helpers for the storage race tests. */
+final class StorageTestThreads {
+
+ private StorageTestThreads() {
+ }
+
+ /**
+ * Waits until the thread has finished or is parked, on a lock for instance. Fails by name
+ * after ten seconds, so a thread that neither finishes nor waits cannot hang the test.
+ */
+ static void awaitParkedOrDone(Thread thread) {
+ long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10);
+ while (System.nanoTime() < deadline) {
+ Thread.State state = thread.getState();
+ if (state == Thread.State.TERMINATED || state == Thread.State.WAITING) {
+ return;
+ }
+ LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(1));
+ }
+ fail("the thread neither finished nor waited: " + thread.getState());
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbPartitionedItemStoreTest.java b/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbPartitionedItemStoreTest.java
new file mode 100644
index 0000000000..14fd3b45eb
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbPartitionedItemStoreTest.java
@@ -0,0 +1,371 @@
+package io.github.hectorvent.floci.services.dynamodb;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.core.common.RegionResolver;
+import io.github.hectorvent.floci.core.common.ServiceConfigAccess;
+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
+import io.github.hectorvent.floci.core.storage.PartitionedFileStorage;
+import io.github.hectorvent.floci.core.storage.StorageBackend;
+import io.github.hectorvent.floci.core.storage.StorageFactory;
+import io.github.hectorvent.floci.services.dynamodb.model.AttributeDefinition;
+import io.github.hectorvent.floci.services.dynamodb.model.KeySchemaElement;
+import io.github.hectorvent.floci.services.dynamodb.model.TableDefinition;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.attribute.FileTime;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.*;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/** DynamoDB items on a store that persists one table file at a time and one item at a time. */
+class DynamoDbPartitionedItemStoreTest {
+
+ private static final String REGION = "us-east-1";
+ private static final String ACCOUNT = "111111111111";
+ private static final FileTime LONG_AGO = FileTime.from(Instant.parse("2020-01-01T00:00:00Z"));
+
+ private final ObjectMapper mapper = new ObjectMapper();
+ private final List<PartitionedFileStorage<JsonNode>> opened = new ArrayList<>();
+ private final RegionResolver resolver = new RegionResolver(REGION, ACCOUNT);
+
+ @TempDir
+ Path tempDir;
+
+ private StorageBackend<String, TableDefinition> tableStore;
+ private DynamoDbService service;
+
+ @BeforeEach
+ void setUp() {
+ tableStore = AccountAwareStorageBackend.inMemory(ACCOUNT);
+ service = new DynamoDbService(tableStore, openItemStore(), resolver);
+ createTable(service, "Logs");
+ createTable(service, "Nodes");
+ }
+
+ @AfterEach
+ void tearDown() {
+ opened.forEach(PartitionedFileStorage::shutdown);
+ }
+
+ @Test
+ void writesToOneTableLeaveAnotherTablesFileUntouched() throws Exception {
+ for (int i = 0; i < 50; i++) {
+ service.putItem("Logs", item("entry-" + i, "payload"), REGION);
+ }
+ Path logsFile = tableFile("Logs");
+ Files.setLastModifiedTime(logsFile, LONG_AGO);
+
+ service.putItem("Nodes", item("node-1", "beat-1"), REGION);
+ service.updateItem("Nodes", key("node-1"), null, "SET v = :v", null,
+ mapper.createObjectNode().set(":v", attribute("beat-2")), "NONE", REGION);
+ service.putItem("Nodes", item("node-2", "beat-1"), REGION);
+ service.deleteItem("Nodes", key("node-2"), REGION);
+
+ assertEquals(LONG_AGO, Files.getLastModifiedTime(logsFile));
+ DynamoDbService restarted = restart();
+ assertEquals("beat-2", value(restarted.getItem("Nodes", key("node-1"), REGION)));
+ assertNull(restarted.getItem("Nodes", key("node-2"), REGION));
+ assertEquals("payload", value(restarted.getItem("Logs", key("entry-49"), REGION)));
+ }
+
+ @Test
+ void batchAndTransactWritesSurviveARestart() {
+ service.putItem("Nodes", item("gone", "x"), REGION);
+ service.batchWriteItem(Map.of("Nodes", List.of(
+ putRequest(item("batch-1", "b1")),
+ deleteRequest(key("gone")))), REGION);
+ service.transactWriteItems(List.of(
+ transactPut("Logs", item("tx-1", "t1")),
+ transactPut("Nodes", item("tx-2", "t2"))), REGION);
+
+ DynamoDbService restarted = restart();
+
+ assertEquals("b1", value(restarted.getItem("Nodes", key("batch-1"), REGION)));
+ assertNull(restarted.getItem("Nodes", key("gone"), REGION));
+ assertEquals("t1", value(restarted.getItem("Logs", key("tx-1"), REGION)));
+ assertEquals("t2", value(restarted.getItem("Nodes", key("tx-2"), REGION)));
+ }
+
+ @Test
+ void anItemStoredUnderAnOlderKeyStaysDeletedAfterARestart() {
+ AccountAwareStorageBackend<Map<String, JsonNode>> itemStore = openItemStore();
+ itemStore.put(REGION + "::Nodes", Map.of("legacy-address-of-node-9", item("node-9", "old")));
+ DynamoDbService rekeyed = new DynamoDbService(tableStore, itemStore, resolver);
+
+ rekeyed.deleteItem("Nodes", key("node-9"), REGION);
+
+ assertNull(restart().getItem("Nodes", key("node-9"), REGION));
+ }
+
+ @Test
+ void deletingATableRemovesItsFile() throws Exception {
+ service.putItem("Nodes", item("node-1", "beat-1"), REGION);
+ assertTrue(Files.exists(tableFile("Nodes")));
+
+ service.deleteTable("Nodes", REGION);
+
+ try (Stream<Path> files = Files.list(tempDir.resolve("items"))) {
+ assertTrue(files.noneMatch(f -> f.getFileName().toString().contains("Nodes")));
+ }
+ }
+
+ @Test
+ void expiredItemsTheSweeperRemovesStayRemovedAfterARestart() {
+ service.updateTimeToLive("Nodes", "expires", true, REGION);
+ ObjectNode expired = item("node-1", "beat-1");
+ expired.set("expires", mapper.createObjectNode().put("N", "1"));
+ service.putItem("Nodes", expired, REGION);
+ service.putItem("Nodes", item("node-2", "beat-1"), REGION);
+
+ service.deleteScannedItems(service.scanExpiredItems());
+
+ DynamoDbService restarted = restart();
+ assertNull(restarted.getItem("Nodes", key("node-1"), REGION));
+ assertEquals("beat-1", value(restarted.getItem("Nodes", key("node-2"), REGION)));
+ }
+
+ @Test
+ void aWriteRightAfterTheSweeperRemovesAnItemSurvivesARestart() {
+ DynamoDbStreamService streams = mock(DynamoDbStreamService.class);
+ DynamoDbService swept = new DynamoDbService(tableStore, openItemStore(), resolver, streams, null);
+ swept.updateTimeToLive("Nodes", "expires", true, REGION);
+ ObjectNode expired = item("node-1", "beat-1");
+ expired.set("expires", mapper.createObjectNode().put("N", "1"));
+ swept.putItem("Nodes", expired, REGION);
+ // The REMOVE event fires after the sweeper drops the item and releases its lock: the
+ // moment a new write of the same key can land before the removal reaches disk.
+ doAnswer(invocation -> {
+ swept.putItem("Nodes", item("node-1", "beat-2"), REGION);
+ return null;
+ }).when(streams).captureEvent(eq("REMOVE"), any(), any(), any(), any(), eq(true));
+
+ swept.deleteScannedItems(swept.scanExpiredItems());
+
+ assertEquals("beat-2", value(swept.getItem("Nodes", key("node-1"), REGION)));
+ assertEquals("beat-2", value(restart().getItem("Nodes", key("node-1"), REGION)));
+ }
+
+ @Test
+ void aWriteToAShardedTableRewritesOneOfItsFiles() throws Exception {
+ Path directory = tempDir.resolve("sharded");
+ DynamoDbService sharded = new DynamoDbService(tableStore, openItemStore(directory, 8), resolver);
+ for (int i = 0; i < 200; i++) {
+ sharded.putItem("Logs", item("entry-" + i, "payload"), REGION);
+ }
+ List<Path> files;
+ try (Stream<Path> listed = Files.list(directory)) {
+ files = listed.toList();
+ }
+ assertEquals(8, files.size());
+ for (Path file : files) {
+ Files.setLastModifiedTime(file, LONG_AGO);
+ }
+
+ sharded.putItem("Logs", item("entry-200", "payload"), REGION);
+
+ assertEquals(1, files.stream().filter(f -> !isUntouched(f)).count());
+ DynamoDbService restarted = new DynamoDbService(tableStore, openItemStore(directory, 8), resolver);
+ assertEquals(201, restarted.scan("Logs", null, null, null, null, null, null, REGION).items().size());
+ }
+
+ @Test
+ void failedConditionalWritesLeaveTheTableFileUntouched() throws Exception {
+ service.putItem("Nodes", item("node-1", "beat-1"), REGION);
+ Path nodesFile = tableFile("Nodes");
+ Files.setLastModifiedTime(nodesFile, LONG_AGO);
+
+ assertThrows(AwsException.class, () -> service.putItem("Nodes", item("node-1", "beat-2"),
+ "attribute_not_exists(id)", null, null, REGION, "NONE"));
+ assertThrows(AwsException.class, () -> service.deleteItem("Nodes", key("node-1"),
+ "attribute_not_exists(id)", null, null, REGION, "NONE"));
+ assertThrows(AwsException.class, () -> service.updateItem("Nodes", key("node-1"), null,
+ "SET v = :v", null, mapper.createObjectNode().set(":v", attribute("beat-3")), "NONE",
+ "attribute_not_exists(id)", REGION, "NONE"));
+
+ assertEquals(LONG_AGO, Files.getLastModifiedTime(nodesFile));
+ assertEquals("beat-1", value(restart().getItem("Nodes", key("node-1"), REGION)));
+ }
+
+ @Test
+ void aCancelledTransactionWritesNothing() throws Exception {
+ service.putItem("Nodes", item("node-1", "beat-1"), REGION);
+ Path nodesFile = tableFile("Nodes");
+ Files.setLastModifiedTime(nodesFile, LONG_AGO);
+
+ assertThrows(TransactionCanceledException.class, () -> service.transactWriteItems(List.of(
+ transactPut("Nodes", item("node-2", "new")),
+ transactConditionCheck("Nodes", key("missing"), "attribute_exists(id)")), REGION));
+
+ assertEquals(LONG_AGO, Files.getLastModifiedTime(nodesFile));
+ assertNull(restart().getItem("Nodes", key("node-2"), REGION));
+ }
+
+ @Test
+ void aBatchToAShardedTableRewritesOnlyTheShardsItTouches() throws Exception {
+ Path directory = tempDir.resolve("sharded");
+ DynamoDbService sharded = new DynamoDbService(tableStore, openItemStore(directory, 8), resolver);
+ for (int i = 0; i < 200; i++) {
+ sharded.putItem("Logs", item("entry-" + i, "payload"), REGION);
+ }
+ List<Path> files;
+ try (Stream<Path> listed = Files.list(directory)) {
+ files = listed.toList();
+ }
+ for (Path file : files) {
+ Files.setLastModifiedTime(file, LONG_AGO);
+ }
+
+ sharded.batchWriteItem(Map.of("Logs", List.of(
+ putRequest(item("entry-200", "payload")),
+ deleteRequest(key("entry-0")))), REGION);
+
+ long rewritten = files.stream().filter(f -> !isUntouched(f)).count();
+ assertTrue(rewritten >= 1 && rewritten <= 2, "rewrote " + rewritten + " of 8 shard files");
+ DynamoDbService restarted = new DynamoDbService(tableStore, openItemStore(directory, 8), resolver);
+ assertNull(restarted.getItem("Logs", key("entry-0"), REGION));
+ assertEquals("payload", value(restarted.getItem("Logs", key("entry-200"), REGION)));
+ assertEquals(200, restarted.scan("Logs", null, null, null, null, null, null, REGION).items().size());
+ }
+
+ @Test
+ void aRestoredTableSurvivesARestart() {
+ createTable(service, "Restored");
+
+ service.finishRestore("Restored", List.of(item("a", "1"), item("b", "2")), REGION);
+
+ DynamoDbService restarted = restart();
+ assertEquals("1", value(restarted.getItem("Restored", key("a"), REGION)));
+ assertEquals("2", value(restarted.getItem("Restored", key("b"), REGION)));
+ }
+
+ @Test
+ void anUnknownItemLayoutIsRejected() {
+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
+ when(config.storage().services().dynamodb().itemLayout()).thenReturn("per-item");
+
+ IllegalArgumentException error = assertThrows(IllegalArgumentException.class,
+ () -> DynamoDbService.itemStore(mock(StorageFactory.class), config));
+ assertTrue(error.getMessage().contains("per-item"), error.getMessage());
+ }
+
+ @Test
+ void aShardCountBelowOneIsRejected() {
+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
+ when(config.defaultAccountId()).thenReturn(ACCOUNT);
+ when(config.storage().persistentPath()).thenReturn(tempDir.toString());
+ when(config.storage().services().dynamodb().itemLayout()).thenReturn("per-table");
+ when(config.storage().services().dynamodb().itemShards()).thenReturn(0);
+ ServiceConfigAccess serviceConfigAccess = mock(ServiceConfigAccess.class);
+ when(serviceConfigAccess.storageMode(anyString())).thenReturn("persistent");
+ StorageFactory factory = new StorageFactory(config, serviceConfigAccess);
+
+ assertThrows(IllegalArgumentException.class, () -> DynamoDbService.itemStore(factory, config));
+ }
+
+ private static boolean isUntouched(Path file) {
+ try {
+ return LONG_AGO.equals(Files.getLastModifiedTime(file));
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+ }
+
+ private AccountAwareStorageBackend<Map<String, JsonNode>> openItemStore() {
+ return openItemStore(tempDir.resolve("items"), 1);
+ }
+
+ private AccountAwareStorageBackend<Map<String, JsonNode>> openItemStore(Path directory, int shards) {
+ PartitionedFileStorage<JsonNode> inner = new PartitionedFileStorage<>(directory, (Path) null,
+ JsonNode.class, shards, PartitionedFileStorage.Mode.SYNC, 60_000);
+ inner.load();
+ opened.add(inner);
+ return new AccountAwareStorageBackend<>(inner, null, ACCOUNT);
+ }
+
+ private DynamoDbService restart() {
+ return new DynamoDbService(tableStore, openItemStore(), resolver);
+ }
+
+ private Path tableFile(String table) throws Exception {
+ try (Stream<Path> files = Files.list(tempDir.resolve("items"))) {
+ return files.filter(f -> f.getFileName().toString().contains(table)).findFirst().orElseThrow();
+ }
+ }
+
+ private void createTable(DynamoDbService target, String name) {
+ target.createTable(name, List.of(new KeySchemaElement("id", "HASH")),
+ List.of(new AttributeDefinition("id", "S")), 5L, 5L, REGION);
+ }
+
+ private ObjectNode key(String id) {
+ ObjectNode key = mapper.createObjectNode();
+ key.set("id", attribute(id));
+ return key;
+ }
+
+ private ObjectNode item(String id, String value) {
+ ObjectNode item = key(id);
+ item.set("v", attribute(value));
+ return item;
+ }
+
+ private ObjectNode attribute(String value) {
+ return mapper.createObjectNode().put("S", value);
+ }
+
+ private ObjectNode putRequest(ObjectNode item) {
+ ObjectNode request = mapper.createObjectNode();
+ request.putObject("PutRequest").set("Item", item);
+ return request;
+ }
+
+ private ObjectNode deleteRequest(ObjectNode key) {
+ ObjectNode request = mapper.createObjectNode();
+ request.putObject("DeleteRequest").set("Key", key);
+ return request;
+ }
+
+ private ObjectNode transactPut(String table, ObjectNode item) {
+ ObjectNode request = mapper.createObjectNode();
+ ObjectNode put = request.putObject("Put");
+ put.put("TableName", table);
+ put.set("Item", item);
+ return request;
+ }
+
+ private ObjectNode transactConditionCheck(String table, ObjectNode key, String condition) {
+ ObjectNode request = mapper.createObjectNode();
+ ObjectNode check = request.putObject("ConditionCheck");
+ check.put("TableName", table);
+ check.set("Key", key);
+ check.put("ConditionExpression", condition);
+ return request;
+ }
+
+ private static String value(JsonNode item) {
+ assertNotNull(item);
+ return item.path("v").path("S").asText();
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbPartitionedStorageIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbPartitionedStorageIntegrationTest.java
new file mode 100644
index 0000000000..725f59bf0e
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbPartitionedStorageIntegrationTest.java
@@ -0,0 +1,112 @@
+package io.github.hectorvent.floci.services.dynamodb;
+
+import io.github.hectorvent.floci.testing.DynamoDbPartitionedStorageProfile;
+import io.github.hectorvent.floci.testing.RestAssuredJsonUtils;
+import io.quarkus.test.junit.QuarkusTest;
+import io.quarkus.test.junit.TestProfile;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.UUID;
+import java.util.stream.Stream;
+
+import static io.github.hectorvent.floci.testing.DynamoDbPartitionedStorageProfile.STORAGE_DIR;
+import static io.restassured.RestAssured.given;
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+
+/**
+ * The {@code per-table} item layout in {@code persistent} mode, end to end through the
+ * application: the configuration selects the partitioned item store, and a write lands in shard
+ * files rather than in a single items file, one set per account.
+ */
+@QuarkusTest
+@TestProfile(DynamoDbPartitionedStorageProfile.class)
+class DynamoDbPartitionedStorageIntegrationTest {
+
+ private static final String DYNAMODB_CONTENT_TYPE = "application/x-amz-json-1.0";
+ private static final String AUTH_ACCOUNT_1 =
+ "AWS4-HMAC-SHA256 Credential=000000000001/20260215/us-east-1/dynamodb/aws4_request, SignedHeaders=host, Signature=abc";
+ private static final String AUTH_ACCOUNT_2 =
+ "AWS4-HMAC-SHA256 Credential=000000000002/20260215/us-east-1/dynamodb/aws4_request, SignedHeaders=host, Signature=abc";
+
+ @BeforeAll
+ static void configureRestAssured() {
+ RestAssuredJsonUtils.configureAwsContentTypes();
+ }
+
+ @Test
+ void itemsAreWrittenToPerAccountShardFiles() throws IOException {
+ // The directory outlives a run, so each run uses its own table name.
+ String table = "partitioned-" + UUID.randomUUID();
+ Path itemsDirectory = Path.of(STORAGE_DIR, "dynamodb-items");
+ try {
+ for (String auth : List.of(AUTH_ACCOUNT_1, AUTH_ACCOUNT_2)) {
+ call(auth, "CreateTable", """
+ {"TableName": "%s", "BillingMode": "PAY_PER_REQUEST",
+ "KeySchema": [{"AttributeName": "pk", "KeyType": "HASH"}],
+ "AttributeDefinitions": [{"AttributeName": "pk", "AttributeType": "S"}]}
+ """.formatted(table));
+ }
+ call(AUTH_ACCOUNT_1, "PutItem", item(table, "one"));
+ call(AUTH_ACCOUNT_2, "PutItem", item(table, "two"));
+
+ assertEquals(2, shardFilesOf(itemsDirectory, table), "one shard file per account's table");
+ assertFalse(Files.exists(Path.of(STORAGE_DIR, "dynamodb-items.json")));
+ getItem(AUTH_ACCOUNT_1, table, "one");
+ getItem(AUTH_ACCOUNT_2, table, "two");
+ } finally {
+ for (String auth : List.of(AUTH_ACCOUNT_1, AUTH_ACCOUNT_2)) {
+ given()
+ .header("X-Amz-Target", "DynamoDB_20120810.DeleteTable")
+ .header("Authorization", auth)
+ .contentType(DYNAMODB_CONTENT_TYPE)
+ .body("{\"TableName\": \"%s\"}".formatted(table))
+ .when().post("/");
+ }
+ }
+ }
+
+ private static String item(String table, String value) {
+ return """
+ {"TableName": "%s", "Item": {"pk": {"S": "item"}, "v": {"S": "%s"}}}
+ """.formatted(table, value);
+ }
+
+ private static void call(String auth, String action, String body) {
+ given()
+ .header("X-Amz-Target", "DynamoDB_20120810." + action)
+ .header("Authorization", auth)
+ .contentType(DYNAMODB_CONTENT_TYPE)
+ .body(body)
+ .when().post("/")
+ .then().statusCode(200);
+ }
+
+ private static void getItem(String auth, String table, String expected) {
+ given()
+ .header("X-Amz-Target", "DynamoDB_20120810.GetItem")
+ .header("Authorization", auth)
+ .contentType(DYNAMODB_CONTENT_TYPE)
+ .body("{\"TableName\": \"%s\", \"Key\": {\"pk\": {\"S\": \"item\"}}}".formatted(table))
+ .when().post("/")
+ .then()
+ .statusCode(200)
+ .body("Item.v.S", equalTo(expected));
+ }
+
+ /** Shard files whose readable name prefix carries the table name, under any account. */
+ private static long shardFilesOf(Path directory, String table) throws IOException {
+ String safeName = table.replaceAll("[^A-Za-z0-9._-]", "_");
+ try (Stream<Path> files = Files.list(directory)) {
+ return files.map(f -> f.getFileName().toString())
+ .filter(name -> name.endsWith(".json") && name.contains(safeName))
+ .count();
+ }
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/testing/DynamoDbPartitionedStorageProfile.java b/src/test/java/io/github/hectorvent/floci/testing/DynamoDbPartitionedStorageProfile.java
new file mode 100644
index 0000000000..3f93a0184d
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/testing/DynamoDbPartitionedStorageProfile.java
@@ -0,0 +1,25 @@
+package io.github.hectorvent.floci.testing;
+
+import io.quarkus.test.junit.QuarkusTestProfile;
+
+import java.util.Map;
+
+/**
+ * DynamoDB in {@code persistent} mode with its items in the {@code per-table} layout, split into
+ * four shards, on a directory of its own. Every other service stays on the test default, memory.
+ * {@code DynamoDbPartitionedStorageIntegrationTest} relies on all of it: it checks that the
+ * configuration reaches the item store and that a write lands in shard files.
+ */
+public class DynamoDbPartitionedStorageProfile implements QuarkusTestProfile {
+
+ public static final String STORAGE_DIR = "target/dynamodb-partitioned-it";
+
+ @Override
+ public Map<String, String> getConfigOverrides() {
+ return Map.of(
+ "floci.storage.services.dynamodb.mode", "persistent",
+ "floci.storage.services.dynamodb.item-layout", "per-table",
+ "floci.storage.services.dynamodb.item-shards", "4",
+ "floci.storage.persistent-path", STORAGE_DIR);
+ }
+}