Light, fluffy, and always free - The AWS Local Emulator alternative
feat(lambda): make reactive S3 sync toggleable and optionally async (#6548)
floverfelt89be5c174290
diff --git a/docs/configuration/advanced/application-yml.md b/docs/configuration/advanced/application-yml.mdindex f0a75991e7..aa97f7a8fb 100644--- a/docs/configuration/advanced/application-yml.md+++ b/docs/configuration/advanced/application-yml.md@@ -182,6 +182,9 @@ floci:# allowed-paths: # Optional allowlist of host paths that may be bind-mounted# - /home/user/projects# - /tmp+ reactive-s3-sync:+ enabled: true # Re-deploy a function when the S3 object it was deployed from is overwritten+ async: false # true = re-deploy in the background so the upload does not waitapigateway:enabled: truediff --git a/docs/configuration/environment-variables.md b/docs/configuration/environment-variables.mdindex 795e30b7c3..d9dd2ad970 100644--- a/docs/configuration/environment-variables.md+++ b/docs/configuration/environment-variables.md@@ -262,6 +262,8 @@ See [Initialization Hooks](./initialization-hooks.md) for lifecycle phases and s| `FLOCI_SERVICES_LAMBDA_CODE_VOLUME_POPULATE_CONCURRENCY` | `max(2, cpus/2)` | Maximum concurrent first-time code-volume populates (functions whose unpacked code is at least 32 MB). The default is derived from the CPU count the JVM sees, so a CPU-constrained Floci container collapses it to 2 and concurrent cold starts of distinct functions serialise into pairs; set this to decouple the cap from the CPU allocation || `FLOCI_SERVICES_LAMBDA_HOT_RELOAD_ENABLED` | `false` | Watch Lambda code directories for changes and reload without redeployment || `FLOCI_SERVICES_LAMBDA_HOT_RELOAD_ALLOWED_PATHS` | _(none)_ | Comma-separated host paths that hot-reload is allowed to watch |+| `FLOCI_SERVICES_LAMBDA_REACTIVE_S3_SYNC_ENABLED` | `true` | Re-deploy a function when the S3 object it was deployed from is overwritten. See [Reactive S3 Sync](../services/lambda.md#reactive-s3-sync-configuration) |+| `FLOCI_SERVICES_LAMBDA_REACTIVE_S3_SYNC_ASYNC` | `false` | Run that re-deploy in the background so the upload returns at once; the function reports `LastUpdateStatus` `InProgress` until it finishes || `FLOCI_SERVICES_LAMBDA_DOCKER_NETWORK` | _(none)_ | Docker network for Lambda containers (overrides `FLOCI_SERVICES_DOCKER_NETWORK`) || `FLOCI_SERVICES_LAMBDA_DOCKER_FLAGS` | _(none)_ | Additional Docker flags applied to Lambda containers, such as `--env`, `--volume`, `--publish`, `--add-host`, `--dns`, `--label`, `--network`, `--user`, `--privileged`, and `--platform`. Explicit `--dns` entries replace automatic embedded DNS and public fallback resolvers. Published ports support `host:container` and `127.0.0.1:host:container` forms || `FLOCI_SERVICES_LAMBDA_CONTAINER_NAME_PREFIX` | `floci` | Base name prefix for Lambda-spawned containers and code volumes (must match `[A-Za-z0-9][A-Za-z0-9_.-]*`) |diff --git a/docs/services/lambda.md b/docs/services/lambda.mdindex 79f583f2bf..613ef99f6f 100644--- a/docs/services/lambda.md+++ b/docs/services/lambda.md@@ -285,6 +285,32 @@ aws lambda invoke --function-name my-function out.json!!! note "Local development behavior"This mechanism requires no custom configuration or special strings. It uses standard AWS SDK and CLI calls; automatic redeployment after an S3 upload is a Floci convenience.+### Configuration {#reactive-s3-sync-configuration}++| Variable | Default | Description |+|---|---|---|+| `FLOCI_SERVICES_LAMBDA_REACTIVE_S3_SYNC_ENABLED` | `true` | Re-deploy a function when the S3 object it was deployed from is overwritten. `false` only stores the object, as AWS does |+| `FLOCI_SERVICES_LAMBDA_REACTIVE_S3_SYNC_ASYNC` | `false` | Re-deploy in the background, so the upload returns without waiting for it |++By default the re-deploy runs inside the upload: `PutObject`, `CopyObject` or+`CompleteMultipartUpload` returns only once the function has its new code. With+`FLOCI_SERVICES_LAMBDA_REACTIVE_S3_SYNC_ASYNC=true` the upload returns at once, and the+function reports `LastUpdateStatus` `InProgress` until the re-deploy finishes. Wait for it+before invoking, or the invocation may still run the previous code:++```bash+aws s3 cp updated-function.zip s3://my-bucket/function.zip+aws lambda wait function-updated --function-name my-function+aws lambda invoke --function-name my-function out.json+```++Uploads that arrive while a re-deploy is running are coalesced: the next re-deploy reads the+object as it is when it starts, so a burst of uploads costs at most one more. A re-deploy that+fails, for example because the object is not a valid ZIP, leaves the previous code in place and+reports `LastUpdateStatus` `Failed` with a `LastUpdateStatusReason` until the function's next+successful update: a re-deploy, `UpdateFunctionCode` or `UpdateFunctionConfiguration`, since the+status describes the last update of either kind. The status is kept in memory and is not persisted.+## Hot-Reload via Bind MountFor the tightest inner-loop development cycle, Floci supports a **bind-mount hot-reload** mode. Instead of packaging code into a ZIP and uploading it to S3, you point Floci directly at a directory on your host machine. The directory is bind-mounted read-only into `/var/task` inside the container, as `/var/task` is read-only in AWS Lambda, so every invocation runs the files as they currently exist on disk, with no upload or redeploy.@@ -579,6 +605,8 @@ These AWS Lambda operations have no handler in Floci. Calls will return `404` or| `FLOCI_SERVICES_LAMBDA_UNRESERVED_CONCURRENCY_MIN` | `100` | Minimum unreserved capacity `PutFunctionConcurrency` must leave || `FLOCI_SERVICES_LAMBDA_HOT_RELOAD_ENABLED` | `false` | Enable bind-mount hot-reload via `S3Bucket=hot-reload` || `FLOCI_SERVICES_LAMBDA_HOT_RELOAD_ALLOWED_PATHS` | *(unset)* | Comma-separated allowlist of host paths that may be bind-mounted |+| `FLOCI_SERVICES_LAMBDA_REACTIVE_S3_SYNC_ENABLED` | `true` | Re-deploy a function when its S3 deployment object is overwritten. See [Reactive S3 Sync](#reactive-s3-sync-configuration) |+| `FLOCI_SERVICES_LAMBDA_REACTIVE_S3_SYNC_ASYNC` | `false` | Run that re-deploy in the background so the upload does not wait for it || `FLOCI_SERVICES_LAMBDA_DOCKER_NETWORK` | *(unset)* | Docker network to attach Lambda containers to (overrides `FLOCI_SERVICES_DOCKER_NETWORK`) || `FLOCI_SERVICES_LAMBDA_DOCKER_FLAGS` | *(unset)* | Additional Docker flags applied to Lambda containers, including `--env`, `--volume`, `--publish`, `--add-host`, `--dns`, `--label`, `--network`, `--user`, `--privileged`, and `--platform`. Explicit `--dns` entries replace automatic embedded DNS and public fallback resolvers. Published ports support `host:container` and `127.0.0.1:host:container` forms || `FLOCI_SERVICES_LAMBDA_EXTRA_HOSTS` | *(unset)* | Comma-separated `hostname:ip` entries added to each Lambda container's `/etc/hosts`; `ip` may be `host-gateway`, mirroring `docker run --add-host` |diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.javaindex 21f30ff7b1..f8642be3af 100644--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java@@ -3272,6 +3272,30 @@ public interface EmulatorConfig {*/Optional<List<String>> allowedPaths();}++ ReactiveS3Sync reactiveS3Sync();++ interface ReactiveS3Sync {+ /**+ * When true, uploading a new object to the S3 bucket and key a function was deployed+ * from re-deploys that function's {@code $LATEST} and drains its warm containers. When+ * false, the object is only stored, as in AWS.+ *+ * Env var: FLOCI_SERVICES_LAMBDA_REACTIVE_S3_SYNC_ENABLED+ */+ @WithDefault("true")+ boolean enabled();++ /**+ * When true, the re-deploy runs in the background so the S3 upload returns at once;+ * the function reports {@code LastUpdateStatus=InProgress} until it finishes. When+ * false, the upload returns only after the function has been re-deployed.+ *+ * Env var: FLOCI_SERVICES_LAMBDA_REACTIVE_S3_SYNC_ASYNC+ */+ @WithDefault("false")+ boolean async();+ }}interface Ec2ServiceConfig {diff --git a/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaController.java b/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaController.javaindex c42324ae48..f471b4e693 100644--- a/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaController.java+++ b/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaController.java@@ -746,7 +746,11 @@ public class LambdaController {.format(Instant.ofEpochMilli(fn.getLastModified()).atOffset(ZoneOffset.UTC)));node.put("RevisionId", fn.getRevisionId());node.put("Version", fn.getVersion());- node.put("LastUpdateStatus", "Successful");+ node.put("LastUpdateStatus", lambdaService.lastUpdateStatus(fn));+ String lastUpdateStatusReason = lambdaService.lastUpdateStatusReason(fn);+ if (lastUpdateStatusReason != null) {+ node.put("LastUpdateStatusReason", lastUpdateStatusReason);+ }// Architectures — always present; default x86_64ArrayNode archNode = node.putArray("Architectures");diff --git a/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaService.java b/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaService.javaindex e6e388e953..f9fb80efa7 100644--- a/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaService.java+++ b/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaService.java@@ -67,6 +67,7 @@ import java.util.Set;import java.util.UUID;import java.util.concurrent.ConcurrentHashMap;import java.util.concurrent.ExecutionException;+import java.util.concurrent.Executor;import java.util.concurrent.TimeUnit;import java.util.concurrent.TimeoutException;import java.util.function.Consumer;@@ -165,6 +166,9 @@ public class LambdaService implements ResourceProvider {"Environment", "EphemeralStorage", "TracingConfig", "DeadLetterConfig","VpcConfig", "SnapStart", "LoggingConfig", "ImageConfig", "DurableConfig");+ private static final Executor REACTIVE_SYNC_THREADS =+ task -> Thread.ofVirtual().name("lambda-reactive-s3-sync").start(task);+private final LambdaFunctionStore functionStore;private final LambdaExecutorService executorService;private final LambdaConcurrencyLimiter concurrencyLimiter;@@ -183,6 +187,9 @@ public class LambdaService implements ResourceProvider {private final DynamoDbStreamsEventSourcePoller dynamodbStreamsPoller;private final StorageFactory storageFactory;private final LambdaLayerService layerService;+ private final Map<String, PendingReactiveSync> pendingReactiveSyncs = new ConcurrentHashMap<>();+ private final Map<String, Integer> runningReactiveSyncs = new ConcurrentHashMap<>();+ private final Map<String, String> failedReactiveSyncs = new ConcurrentHashMap<>();private final Ec2Service ec2Service;/** Null in the constructors tests use, exactly as the other optional collaborators above are. */private final CustomResourceLiveness customResourceLiveness;@@ -775,6 +782,7 @@ public class LambdaService implements ResourceProvider {// Drain warm containers — they have stale code mountedwarmPool.drainEnvironment(fn);+ failedReactiveSyncs.remove(fn.getFunctionArn());functionStore.save(region, fn);LOG.infov("Updated code for function: {0}", functionName);@@ -996,6 +1004,8 @@ public class LambdaService implements ResourceProvider {warmPool.drainEnvironment(fn);functionStore.save(region, fn);+ // LastUpdateStatus reports the last update of either kind, so this one replaces a failed sync's.+ failedReactiveSyncs.remove(fn.getFunctionArn());LOG.infov("Updated configuration for function: {0}", functionName);return fn;}@@ -1116,6 +1126,7 @@ public class LambdaService implements ResourceProvider {reclaimLegacyCodeDirectoryIfUnused(functionName);versionCounters.remove(versionCounterKey(region, fn));versionCounters.remove(legacyVersionCounterKey(region, functionName));+ failedReactiveSyncs.remove(arn);if (aliasStore != null) {for (LambdaAlias alias : aliasStore.list(region, functionName)) {aliasStore.delete(region, functionName, alias.getName());@@ -4140,6 +4151,9 @@ public class LambdaService implements ResourceProvider {* functions linked to the updated object.*/public void onS3ObjectUpdated(@Observes S3ObjectUpdatedEvent event) {+ if (!reactiveS3SyncEnabled()) {+ return;+ }LOG.debugv("Observing S3 update: {0}/{1}", event.bucketName(), event.key());// For simplicity, we scan all functions in the default region// Most local dev setups use a single region.@@ -4148,32 +4162,118 @@ public class LambdaService implements ResourceProvider {// other account would silently never hot-reload from S3.String region = regionResolver.getDefaultRegion();List<LambdaFunction> functions = functionStore.listAllAccounts(region);+ boolean async = reactiveS3SyncAsync();for (LambdaFunction fn : functions) {if (!event.bucketName().equals(fn.getS3Bucket()) || !event.key().equals(fn.getS3Key())) {continue;}- synchronized (lockForConcurrencyOp(fn.getFunctionArn())) {- if (fn.isHotReload()) {- continue;+ if (!async) {+ syncFromS3(fn, event.bucketName(), event.key(), region);+ continue;+ }+ // The newest upload replaces any source still waiting, and a queued sync reads its object+ // when it starts, so one queued sync per function covers every upload before it. The+ // uploader's account goes with it: S3 resolves the bucket in the caller's account, and+ // the background thread has no request of its own.+ String functionArn = fn.getFunctionArn();+ PendingReactiveSync source = new PendingReactiveSync(regionResolver.getAccountId(),+ event.bucketName(), event.key());+ if (pendingReactiveSyncs.put(functionArn, source) == null) {+ String ownerAccountId = ownerAccount(fn);+ String functionName = fn.getFunctionName();+ REACTIVE_SYNC_THREADS.execute(() -> syncQueued(ownerAccountId, region, functionName, functionArn));+ }+ }+ }++ /** Re-reads the function, which may have been redeployed or deleted while the sync was queued. */+ private void syncQueued(String ownerAccountId, String region, String functionName, String functionArn) {+ // Marked running before the pending source is taken, so the function never reports neither.+ runningReactiveSyncs.merge(functionArn, 1, Integer::sum);+ try {+ synchronized (lockForConcurrencyOp(functionArn)) {+ PendingReactiveSync source = pendingReactiveSyncs.remove(functionArn);+ if (source == null) {+ return;}- if (event.bucketName().equals(fn.getS3Bucket()) && event.key().equals(fn.getS3Key())) {- LOG.infov("Reactive S3 Sync: updating function {0} from s3://{1}/{2}",- fn.getFunctionName(), event.bucketName(), event.key());- try {- try (DeploymentPackage zip = readS3Package(fn, event.bucketName(), event.key(), region)) {- extractZipCodePackage(fn, zip, region);- }- fn.setLastModified(Instant.now().toEpochMilli());- fn.setRevisionId(UUID.randomUUID().toString());- functionStore.saveForAccount(ownerAccount(fn), region, fn);-- // Push to warm workers- warmPool.pushCodeUpdate(fn);- } catch (Exception e) {- LOG.warnv("Failed reactive sync for function {0}: {1}", fn.getFunctionName(), e.getMessage());- }+ RequestScopes.runAs(source.accountId(), () ->+ functionStore.getForAccount(ownerAccountId, region, functionName)+ .ifPresent(fn -> deployFromS3(fn, source.bucketName(), source.key(), region)));+ }+ } finally {+ runningReactiveSyncs.computeIfPresent(functionArn, (arn, count) -> count == 1 ? null : count - 1);+ }+ }++ private void syncFromS3(LambdaFunction fn, String bucketName, String key, String region) {+ String functionArn = fn.getFunctionArn();+ runningReactiveSyncs.merge(functionArn, 1, Integer::sum);+ try {+ synchronized (lockForConcurrencyOp(functionArn)) {+ deployFromS3(fn, bucketName, key, region);+ }+ } finally {+ runningReactiveSyncs.computeIfPresent(functionArn, (arn, count) -> count == 1 ? null : count - 1);+ }+ }++ /** Callers hold the function's lock. */+ private void deployFromS3(LambdaFunction fn, String bucketName, String key, String region) {+ if (fn.isHotReload()) {+ return;+ }+ if (bucketName.equals(fn.getS3Bucket()) && key.equals(fn.getS3Key())) {+ LOG.infov("Reactive S3 Sync: updating function {0} from s3://{1}/{2}",+ fn.getFunctionName(), bucketName, key);+ try {+ try (DeploymentPackage zip = readS3Package(fn, bucketName, key, region)) {+ extractZipCodePackage(fn, zip, region);}+ fn.setLastModified(Instant.now().toEpochMilli());+ fn.setRevisionId(UUID.randomUUID().toString());+ functionStore.saveForAccount(ownerAccount(fn), region, fn);++ // Push to warm workers+ warmPool.pushCodeUpdate(fn);+ failedReactiveSyncs.remove(fn.getFunctionArn());+ } catch (Exception e) {+ LOG.warnv("Failed reactive sync for function {0}: {1}", fn.getFunctionName(), e.getMessage());+ failedReactiveSyncs.put(fn.getFunctionArn(), "Reactive S3 sync from s3://" + bucketName+ + "/" + key + " failed: " + e.getMessage());}}}++ /**+ * The function's LastUpdateStatus, which only a reactive S3 sync moves off Successful:+ * InProgress from the upload until the re-deploy of {@code $LATEST} finishes, Failed when the+ * last one could not. Held in memory, so a restart cannot leave an update reported as running.+ */+ public String lastUpdateStatus(LambdaFunction fn) {+ if (!"$LATEST".equals(fn.getVersion())) {+ return "Successful";+ }+ String functionArn = fn.getFunctionArn();+ // Pending first: a sync marks itself running before it takes the pending source.+ if (pendingReactiveSyncs.containsKey(functionArn) || runningReactiveSyncs.containsKey(functionArn)) {+ return "InProgress";+ }+ return failedReactiveSyncs.containsKey(functionArn) ? "Failed" : "Successful";+ }++ /** Why the last reactive S3 sync of {@code $LATEST} failed, or null when it did not. */+ public String lastUpdateStatusReason(LambdaFunction fn) {+ return "Failed".equals(lastUpdateStatus(fn)) ? failedReactiveSyncs.get(fn.getFunctionArn()) : null;+ }++ private boolean reactiveS3SyncEnabled() {+ return config == null || config.services().lambda().reactiveS3Sync().enabled();+ }++ private boolean reactiveS3SyncAsync() {+ return config != null && config.services().lambda().reactiveS3Sync().async();+ }++ private record PendingReactiveSync(String accountId, String bucketName, String key) {+ }}diff --git a/src/main/resources/application.yml b/src/main/resources/application.ymlindex d8d8b6d4ed..cd10683868 100644--- a/src/main/resources/application.yml+++ b/src/main/resources/application.yml@@ -467,6 +467,9 @@ floci:# init-image: busybox:1.36 # Init container image (needs sh, wget, unzip)hot-reload:enabled: false+ reactive-s3-sync:+ enabled: true # Re-deploy a function when the S3 object it was deployed from is overwritten+ async: false # Re-deploy in the background so the S3 upload does not wait for itapigateway:enabled: truevtl-max-loops: 10000 # Maximum #foreach iterations a VTL mapping template may execute (truncates the loop, does not fail the template)diff --git a/src/test/java/io/github/hectorvent/floci/services/lambda/LambdaAsyncReactiveSyncIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/lambda/LambdaAsyncReactiveSyncIntegrationTest.javanew file mode 100644index 0000000000..8e9962f690--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/lambda/LambdaAsyncReactiveSyncIntegrationTest.java@@ -0,0 +1,77 @@+package io.github.hectorvent.floci.services.lambda;++import io.github.hectorvent.floci.services.lambdaweb.LambdaWebZips;+import io.github.hectorvent.floci.testing.LambdaCodeSourcesProfile;+import io.quarkus.test.junit.QuarkusTest;+import io.quarkus.test.junit.TestProfile;+import org.junit.jupiter.api.Test;++import java.security.MessageDigest;+import java.time.Duration;+import java.util.Base64;++import static io.restassured.RestAssured.given;+import static org.awaitility.Awaitility.await;+import static org.hamcrest.Matchers.equalTo;++/**+ * With asynchronous reactive S3 sync, the re-deploy runs on a background thread with no request of+ * its own, so it must read the object in the uploader's account rather than the default one.+ */+@QuarkusTest+@TestProfile(LambdaCodeSourcesProfile.class)+class LambdaAsyncReactiveSyncIntegrationTest {++ private static final String ACCOUNT = "000000000021";+ private static final String S3_AUTH =+ "AWS4-HMAC-SHA256 Credential=" + ACCOUNT + "/20260215/us-east-1/s3/aws4_request, SignedHeaders=host, Signature=abc";+ private static final String LAMBDA_AUTH =+ "AWS4-HMAC-SHA256 Credential=" + ACCOUNT + "/20260215/us-east-1/lambda/aws4_request, SignedHeaders=host, Signature=abc";+ private static final String BUCKET = "async-reactive-sync-bucket";+ private static final String KEY = "function.zip";+ private static final String FN = "async-reactive-sync-fn";++ @Test+ void uploadFromANonDefaultAccountRedeploysTheFunctionInTheBackground() throws Exception {+ byte[] v1 = LambdaWebZips.indexJs("exports.handler = async () => 'v1';");+ byte[] v2 = LambdaWebZips.indexJs("exports.handler = async () => 'v2';");+ given().header("Authorization", S3_AUTH).when().put("/" + BUCKET).then().statusCode(200);+ given().header("Authorization", S3_AUTH).body(v1).when().put("/" + BUCKET + "/" + KEY).then().statusCode(200);+ given()+ .header("Authorization", LAMBDA_AUTH)+ .contentType("application/json")+ .body("""+ {+ "FunctionName": "%s",+ "Runtime": "nodejs20.x",+ "Role": "arn:aws:iam::%s:role/role",+ "Handler": "index.handler",+ "Code": {+ "S3Bucket": "%s",+ "S3Key": "%s"+ }+ }+ """.formatted(FN, ACCOUNT, BUCKET, KEY))+ .when()+ .post("/2015-03-31/functions")+ .then()+ .statusCode(201)+ .body("CodeSha256", equalTo(sha256(v1)));++ given().header("Authorization", S3_AUTH).body(v2).when().put("/" + BUCKET + "/" + KEY).then().statusCode(200);++ await().atMost(Duration.ofSeconds(10)).pollInterval(Duration.ofMillis(100)).untilAsserted(() ->+ given()+ .header("Authorization", LAMBDA_AUTH)+ .when()+ .get("/2015-03-31/functions/" + FN + "/configuration")+ .then()+ .statusCode(200)+ .body("LastUpdateStatus", equalTo("Successful"))+ .body("CodeSha256", equalTo(sha256(v2))));+ }++ private static String sha256(byte[] data) throws Exception {+ return Base64.getEncoder().encodeToString(MessageDigest.getInstance("SHA-256").digest(data));+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/lambda/LambdaServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/lambda/LambdaServiceTest.javaindex 2baadd841e..a82bfaeebf 100644--- a/src/test/java/io/github/hectorvent/floci/services/lambda/LambdaServiceTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/lambda/LambdaServiceTest.java@@ -2354,13 +2354,18 @@ class LambdaServiceTest {}private static LambdaService s3BackedService(S3Service s3, WarmPool warmPool, ZipExtractor extractor) {+ return s3BackedService(s3, warmPool, extractor, null);+ }++ private static LambdaService s3BackedService(S3Service s3, WarmPool warmPool, ZipExtractor extractor,+ EmulatorConfig config) {StorageFactory storageFactory = mock(StorageFactory.class);when(storageFactory.create(anyString(), anyString(), any())).thenAnswer(inv -> AccountAwareStorageBackend.inMemory("000000000000"));LambdaFunctionStore functionStore = new LambdaFunctionStore(new InMemoryStorage<String, LambdaFunction>());LambdaAliasStore aliasStore = new LambdaAliasStore(storageFactory);return new LambdaService(functionStore, null, new LambdaConcurrencyLimiter(),- warmPool, new CodeStore(Path.of("target/test-data/lambda-code")), extractor, null,+ warmPool, new CodeStore(Path.of("target/test-data/lambda-code")), extractor, config,new RegionResolver(REGION, "000000000000"), null, aliasStore,new LambdaTargetResolver(functionStore, aliasStore), s3, null, null, null, null,storageFactory, null, null, null, null, new ObjectMapper());@@ -2420,6 +2425,201 @@ class LambdaServiceTest {}}+ @Test+ void reactiveSyncDisabled_leavesTheFunctionOnItsDeployedCode() throws Exception {+ S3Service s3 = mock(S3Service.class);+ WarmPool warmPool = mock(WarmPool.class);+ LambdaService syncingService = s3BackedService(s3, warmPool, new ZipExtractor(),+ reactiveS3SyncConfig(false, false));+ String name = "reactive-disabled-" + UUID.randomUUID();+ when(s3.openObject("code-bucket", "function.zip", null)).thenReturn(heldObject(zipWithBody("v1")));+ Map<String, Object> request = baseRequest(name);+ request.put("Code", Map.of("S3Bucket", "code-bucket", "S3Key", "function.zip"));+ LambdaFunction latest = syncingService.createFunction(REGION, request);++ syncingService.onS3ObjectUpdated(new S3ObjectUpdatedEvent("code-bucket", "function.zip"));++ verify(s3, times(1)).openObject("code-bucket", "function.zip", null);+ verify(warmPool, never()).pushCodeUpdate(any());+ assertEquals("v1", Files.readString(Path.of(latest.getCodeLocalPath()).resolve("index.js")).trim());+ }++ @Test+ void asyncReactiveSync_returnsBeforeTheRedeployAndReportsInProgressUntilItFinishes() throws Exception {+ S3Service s3 = mock(S3Service.class);+ WarmPool warmPool = mock(WarmPool.class);+ ZipExtractor extractor = spy(new ZipExtractor());+ LambdaService syncingService = s3BackedService(s3, warmPool, extractor, reactiveS3SyncConfig(true, true));+ String name = "reactive-async-" + UUID.randomUUID();+ when(s3.openObject("code-bucket", "function.zip", null))+ .thenReturn(heldObject(zipWithBody("v1")))+ .thenReturn(heldObject(zipWithBody("v2")));+ Map<String, Object> request = baseRequest(name);+ request.put("Code", Map.of("S3Bucket", "code-bucket", "S3Key", "function.zip"));+ LambdaFunction latest = syncingService.createFunction(REGION, request);+ CountDownLatch release = new CountDownLatch(1);+ doAnswer(inv -> {+ assertTrue(release.await(5, TimeUnit.SECONDS));+ return inv.callRealMethod();+ }).when(extractor).extractTo(any(byte[].class), any(Path.class), anyInt(), any(Runnable.class));+ try {+ syncingService.onS3ObjectUpdated(new S3ObjectUpdatedEvent("code-bucket", "function.zip"));++ assertEquals("InProgress", syncingService.lastUpdateStatus(latest));+ verify(warmPool, never()).pushCodeUpdate(any());+ } finally {+ release.countDown();+ }++ verify(warmPool, timeout(5000)).pushCodeUpdate(latest);+ awaitLastUpdateStatus(syncingService, latest, "Successful");+ assertEquals("v2", Files.readString(Path.of(latest.getCodeLocalPath()).resolve("index.js")).trim());+ }++ @Test+ void asyncReactiveSync_coalescesUploadsThatArriveWhileOneIsRunning() throws Exception {+ S3Service s3 = mock(S3Service.class);+ WarmPool warmPool = mock(WarmPool.class);+ ZipExtractor extractor = spy(new ZipExtractor());+ LambdaService syncingService = s3BackedService(s3, warmPool, extractor, reactiveS3SyncConfig(true, true));+ String name = "reactive-coalesce-" + UUID.randomUUID();+ when(s3.openObject("code-bucket", "function.zip", null))+ .thenReturn(heldObject(zipWithBody("v1")))+ .thenReturn(heldObject(zipWithBody("v2")))+ .thenReturn(heldObject(zipWithBody("v5")));+ Map<String, Object> request = baseRequest(name);+ request.put("Code", Map.of("S3Bucket", "code-bucket", "S3Key", "function.zip"));+ LambdaFunction latest = syncingService.createFunction(REGION, request);+ CountDownLatch extracting = new CountDownLatch(1);+ CountDownLatch release = new CountDownLatch(1);+ doAnswer(inv -> {+ extracting.countDown();+ assertTrue(release.await(5, TimeUnit.SECONDS));+ return inv.callRealMethod();+ }).when(extractor).extractTo(any(byte[].class), any(Path.class), anyInt(), any(Runnable.class));+ try {+ syncingService.onS3ObjectUpdated(new S3ObjectUpdatedEvent("code-bucket", "function.zip"));+ assertTrue(extracting.await(5, TimeUnit.SECONDS));+ for (int upload = 3; upload <= 5; upload++) {+ syncingService.onS3ObjectUpdated(new S3ObjectUpdatedEvent("code-bucket", "function.zip"));+ }+ } finally {+ release.countDown();+ }++ verify(warmPool, timeout(5000).times(2)).pushCodeUpdate(latest);+ awaitLastUpdateStatus(syncingService, latest, "Successful");+ // Create, the running sync, and one queued sync for the three uploads behind it.+ verify(s3, times(3)).openObject("code-bucket", "function.zip", null);+ assertEquals("v5", Files.readString(Path.of(latest.getCodeLocalPath()).resolve("index.js")).trim());+ }++ @Test+ void asyncReactiveSync_deploysAnUploadToANewSourceWhileTheOldOneIsStillQueued() throws Exception {+ S3Service s3 = mock(S3Service.class);+ WarmPool warmPool = mock(WarmPool.class);+ LambdaService syncingService = s3BackedService(s3, warmPool, new ZipExtractor(),+ reactiveS3SyncConfig(true, true));+ String name = "reactive-new-source-" + UUID.randomUUID();+ when(s3.openObject("code-bucket", "old.zip", null)).thenReturn(heldObject(zipWithBody("v1")));+ when(s3.openObject("code-bucket", "new.zip", null))+ .thenReturn(heldObject(zipWithBody("v2")))+ .thenReturn(heldObject(zipWithBody("v3")));+ Map<String, Object> request = baseRequest(name);+ request.put("Code", Map.of("S3Bucket", "code-bucket", "S3Key", "old.zip"));+ LambdaFunction latest = syncingService.createFunction(REGION, request);++ // Holding the function's lock keeps the old source's sync queued behind it.+ synchronized (syncingService.lockForConcurrencyOp(latest.getFunctionArn())) {+ syncingService.onS3ObjectUpdated(new S3ObjectUpdatedEvent("code-bucket", "old.zip"));+ syncingService.updateFunctionCode(REGION, name, Map.of("S3Bucket", "code-bucket", "S3Key", "new.zip"));+ syncingService.onS3ObjectUpdated(new S3ObjectUpdatedEvent("code-bucket", "new.zip"));+ assertEquals("InProgress", syncingService.lastUpdateStatus(latest));+ }++ verify(warmPool, timeout(5000)).pushCodeUpdate(latest);+ awaitLastUpdateStatus(syncingService, latest, "Successful");+ verify(s3, times(1)).openObject("code-bucket", "old.zip", null);+ verify(s3, times(2)).openObject("code-bucket", "new.zip", null);+ assertEquals("v3", Files.readString(Path.of(latest.getCodeLocalPath()).resolve("index.js")).trim());+ }++ @Test+ void failedReactiveSync_reportsFailedUntilTheNextCodeUpdate() throws Exception {+ S3Service s3 = mock(S3Service.class);+ WarmPool warmPool = mock(WarmPool.class);+ LambdaService syncingService = s3BackedService(s3, warmPool, new ZipExtractor(),+ reactiveS3SyncConfig(true, false));+ LambdaFunction latest = functionWhoseLastSyncFailed(syncingService, s3);++ assertEquals("Failed", syncingService.lastUpdateStatus(latest));+ assertEquals("Reactive S3 sync from s3://code-bucket/function.zip failed: The specified key does not exist.",+ syncingService.lastUpdateStatusReason(latest));+ verify(warmPool, never()).pushCodeUpdate(any());++ syncingService.updateFunctionCode(REGION, latest.getFunctionName(), Map.of("ZipFile", zipWithBody("v2")));++ assertEquals("Successful", syncingService.lastUpdateStatus(latest));+ assertNull(syncingService.lastUpdateStatusReason(latest));+ }++ @Test+ void failedReactiveSync_isClearedByAConfigurationUpdate() throws Exception {+ S3Service s3 = mock(S3Service.class);+ LambdaService syncingService = s3BackedService(s3, mock(WarmPool.class), new ZipExtractor(),+ reactiveS3SyncConfig(true, false));+ LambdaFunction latest = functionWhoseLastSyncFailed(syncingService, s3);++ syncingService.updateFunctionConfiguration(REGION, latest.getFunctionName(),+ Map.of("Environment", Map.of("Variables", Map.of("STAGE", "ci"))));++ assertEquals("Successful", syncingService.lastUpdateStatus(latest));+ assertNull(syncingService.lastUpdateStatusReason(latest));+ }++ /**+ * A function deployed from {@code s3://code-bucket/function.zip} whose sync after the object was+ * overwritten failed because the object had gone. {@code s3} is the mock {@code service} reads.+ */+ private LambdaFunction functionWhoseLastSyncFailed(LambdaService service, S3Service s3)+ throws Exception {+ when(s3.openObject("code-bucket", "function.zip", null))+ .thenReturn(heldObject(zipWithBody("v1")))+ .thenThrow(new AwsException("NoSuchKey", "The specified key does not exist.", 404));+ Map<String, Object> request = baseRequest("reactive-failed-" + UUID.randomUUID());+ request.put("Code", Map.of("S3Bucket", "code-bucket", "S3Key", "function.zip"));+ LambdaFunction latest = service.createFunction(REGION, request);+ service.onS3ObjectUpdated(new S3ObjectUpdatedEvent("code-bucket", "function.zip"));+ return latest;+ }++ private static EmulatorConfig reactiveS3SyncConfig(boolean enabled, boolean async) {+ EmulatorConfig cfg = mock(EmulatorConfig.class);+ EmulatorConfig.ServicesConfig svc = mock(EmulatorConfig.ServicesConfig.class);+ EmulatorConfig.LambdaServiceConfig lambdaCfg = mock(EmulatorConfig.LambdaServiceConfig.class);+ EmulatorConfig.LambdaServiceConfig.ReactiveS3Sync sync =+ mock(EmulatorConfig.LambdaServiceConfig.ReactiveS3Sync.class);+ when(cfg.services()).thenReturn(svc);+ when(svc.lambda()).thenReturn(lambdaCfg);+ when(lambdaCfg.reactiveS3Sync()).thenReturn(sync);+ when(lambdaCfg.defaultTimeoutSeconds()).thenReturn(3);+ when(lambdaCfg.defaultMemoryMb()).thenReturn(128);+ when(lambdaCfg.zipMaxEntries()).thenReturn(ZipExtractor.DEFAULT_MAX_ENTRIES);+ when(lambdaCfg.executor()).thenReturn("docker");+ when(sync.enabled()).thenReturn(enabled);+ when(sync.async()).thenReturn(async);+ return cfg;+ }++ private static void awaitLastUpdateStatus(LambdaService service, LambdaFunction fn, String expected)+ throws InterruptedException {+ long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5);+ while (!expected.equals(service.lastUpdateStatus(fn)) && System.nanoTime() < deadline) {+ Thread.sleep(10);+ }+ assertEquals(expected, service.lastUpdateStatus(fn));+ }+@Testvoid deletingOneVersionPreservesAnotherVersionAndLatest() throws Exception {String name = "shared-version-code";diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebZips.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebZips.javaindex 75901934ce..b1647925c9 100644--- a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebZips.java+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebZips.java@@ -10,7 +10,7 @@ import java.util.Arrays;import java.util.zip.ZipEntry;import java.util.zip.ZipOutputStream;-/** Code packages for tests that create Web Function revisions. */+/** Code packages for tests that deploy function code, Web Function revisions among them. */public final class LambdaWebZips {private LambdaWebZips() {diff --git a/src/test/java/io/github/hectorvent/floci/testing/LambdaCodeSourcesProfile.java b/src/test/java/io/github/hectorvent/floci/testing/LambdaCodeSourcesProfile.javaindex 43755d425b..562067aac5 100644--- a/src/test/java/io/github/hectorvent/floci/testing/LambdaCodeSourcesProfile.java+++ b/src/test/java/io/github/hectorvent/floci/testing/LambdaCodeSourcesProfile.java@@ -6,10 +6,11 @@ import java.util.Map;/*** Sets where a Lambda function's code may come from, for the classes that test it: foreign-account- * layer ARNs are accepted (recorded verbatim, never mounted), and hot-reload mounts are limited to- * an allow-list under {@code /home/ci/code}. Each setting only changes the requests its own class- * makes, so the two classes share this profile, and Quarkus builds one application for both- * rather than one each.+ * layer ARNs are accepted (recorded verbatim, never mounted), hot-reload mounts are limited to an+ * allow-list under {@code /home/ci/code}, and a reactive S3 sync re-deploys a function in the+ * background rather than inside the upload of its code. Each setting only changes the requests its+ * own class makes, so the three classes share this profile, and Quarkus builds one application for+ * all of them rather than one each.*/public class LambdaCodeSourcesProfile implements QuarkusTestProfile {@@ -17,6 +18,7 @@ public class LambdaCodeSourcesProfile implements QuarkusTestProfile {public Map<String, String> getConfigOverrides() {return Map.of("floci.services.lambda.accept-external-layer-arns", "true",- "floci.services.lambda.hot-reload.allowed-paths", "/home/ci/code");+ "floci.services.lambda.hot-reload.allowed-paths", "/home/ci/code",+ "floci.services.lambda.reactive-s3-sync.async", "true");}}diff --git a/src/test/resources/application.yml b/src/test/resources/application.ymlindex 4bfc0e91e5..cba3a9c43a 100644--- a/src/test/resources/application.yml+++ b/src/test/resources/application.yml@@ -168,6 +168,9 @@ floci:# aws-config-path:hot-reload:enabled: true+ reactive-s3-sync:+ enabled: true+ async: falseapigateway:enabled: truevtl-max-loops: 10000