Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(lambda): make reactive S3 sync toggleable and optionally async (#6548)

floverfelt89be5c174290
diff --git a/docs/configuration/advanced/application-yml.md b/docs/configuration/advanced/application-yml.md
index f0a75991e7..aa97f7a8fb 100644
--- a/docs/configuration/advanced/application-yml.md
+++ b/docs/configuration/advanced/application-yml.md
@@ -182,6 +182,9 @@ floci:
# allowed-paths: # Optional allowlist of host paths that may be bind-mounted
# - /home/user/projects
# - /tmp
+ reactive-s3-sync:
+ enabled: true # Re-deploy a function when the S3 object it was deployed from is overwritten
+ async: false # true = re-deploy in the background so the upload does not wait
apigateway:
enabled: true
diff --git a/docs/configuration/environment-variables.md b/docs/configuration/environment-variables.md
index 795e30b7c3..d9dd2ad970 100644
--- a/docs/configuration/environment-variables.md
+++ b/docs/configuration/environment-variables.md
@@ -262,6 +262,8 @@ See [Initialization Hooks](./initialization-hooks.md) for lifecycle phases and s
| `FLOCI_SERVICES_LAMBDA_CODE_VOLUME_POPULATE_CONCURRENCY` | `max(2, cpus/2)` | Maximum concurrent first-time code-volume populates (functions whose unpacked code is at least 32 MB). The default is derived from the CPU count the JVM sees, so a CPU-constrained Floci container collapses it to 2 and concurrent cold starts of distinct functions serialise into pairs; set this to decouple the cap from the CPU allocation |
| `FLOCI_SERVICES_LAMBDA_HOT_RELOAD_ENABLED` | `false` | Watch Lambda code directories for changes and reload without redeployment |
| `FLOCI_SERVICES_LAMBDA_HOT_RELOAD_ALLOWED_PATHS` | _(none)_ | Comma-separated host paths that hot-reload is allowed to watch |
+| `FLOCI_SERVICES_LAMBDA_REACTIVE_S3_SYNC_ENABLED` | `true` | Re-deploy a function when the S3 object it was deployed from is overwritten. See [Reactive S3 Sync](../services/lambda.md#reactive-s3-sync-configuration) |
+| `FLOCI_SERVICES_LAMBDA_REACTIVE_S3_SYNC_ASYNC` | `false` | Run that re-deploy in the background so the upload returns at once; the function reports `LastUpdateStatus` `InProgress` until it finishes |
| `FLOCI_SERVICES_LAMBDA_DOCKER_NETWORK` | _(none)_ | Docker network for Lambda containers (overrides `FLOCI_SERVICES_DOCKER_NETWORK`) |
| `FLOCI_SERVICES_LAMBDA_DOCKER_FLAGS` | _(none)_ | Additional Docker flags applied to Lambda containers, such as `--env`, `--volume`, `--publish`, `--add-host`, `--dns`, `--label`, `--network`, `--user`, `--privileged`, and `--platform`. Explicit `--dns` entries replace automatic embedded DNS and public fallback resolvers. Published ports support `host:container` and `127.0.0.1:host:container` forms |
| `FLOCI_SERVICES_LAMBDA_CONTAINER_NAME_PREFIX` | `floci` | Base name prefix for Lambda-spawned containers and code volumes (must match `[A-Za-z0-9][A-Za-z0-9_.-]*`) |
diff --git a/docs/services/lambda.md b/docs/services/lambda.md
index 79f583f2bf..613ef99f6f 100644
--- a/docs/services/lambda.md
+++ b/docs/services/lambda.md
@@ -285,6 +285,32 @@ aws lambda invoke --function-name my-function out.json
!!! note "Local development behavior"
This mechanism requires no custom configuration or special strings. It uses standard AWS SDK and CLI calls; automatic redeployment after an S3 upload is a Floci convenience.
+### Configuration {#reactive-s3-sync-configuration}
+
+| Variable | Default | Description |
+|---|---|---|
+| `FLOCI_SERVICES_LAMBDA_REACTIVE_S3_SYNC_ENABLED` | `true` | Re-deploy a function when the S3 object it was deployed from is overwritten. `false` only stores the object, as AWS does |
+| `FLOCI_SERVICES_LAMBDA_REACTIVE_S3_SYNC_ASYNC` | `false` | Re-deploy in the background, so the upload returns without waiting for it |
+
+By default the re-deploy runs inside the upload: `PutObject`, `CopyObject` or
+`CompleteMultipartUpload` returns only once the function has its new code. With
+`FLOCI_SERVICES_LAMBDA_REACTIVE_S3_SYNC_ASYNC=true` the upload returns at once, and the
+function reports `LastUpdateStatus` `InProgress` until the re-deploy finishes. Wait for it
+before invoking, or the invocation may still run the previous code:
+
+```bash
+aws s3 cp updated-function.zip s3://my-bucket/function.zip
+aws lambda wait function-updated --function-name my-function
+aws lambda invoke --function-name my-function out.json
+```
+
+Uploads that arrive while a re-deploy is running are coalesced: the next re-deploy reads the
+object as it is when it starts, so a burst of uploads costs at most one more. A re-deploy that
+fails, for example because the object is not a valid ZIP, leaves the previous code in place and
+reports `LastUpdateStatus` `Failed` with a `LastUpdateStatusReason` until the function's next
+successful update: a re-deploy, `UpdateFunctionCode` or `UpdateFunctionConfiguration`, since the
+status describes the last update of either kind. The status is kept in memory and is not persisted.
+
## Hot-Reload via Bind Mount
For the tightest inner-loop development cycle, Floci supports a **bind-mount hot-reload** mode. Instead of packaging code into a ZIP and uploading it to S3, you point Floci directly at a directory on your host machine. The directory is bind-mounted read-only into `/var/task` inside the container, as `/var/task` is read-only in AWS Lambda, so every invocation runs the files as they currently exist on disk, with no upload or redeploy.
@@ -579,6 +605,8 @@ These AWS Lambda operations have no handler in Floci. Calls will return `404` or
| `FLOCI_SERVICES_LAMBDA_UNRESERVED_CONCURRENCY_MIN` | `100` | Minimum unreserved capacity `PutFunctionConcurrency` must leave |
| `FLOCI_SERVICES_LAMBDA_HOT_RELOAD_ENABLED` | `false` | Enable bind-mount hot-reload via `S3Bucket=hot-reload` |
| `FLOCI_SERVICES_LAMBDA_HOT_RELOAD_ALLOWED_PATHS` | *(unset)* | Comma-separated allowlist of host paths that may be bind-mounted |
+| `FLOCI_SERVICES_LAMBDA_REACTIVE_S3_SYNC_ENABLED` | `true` | Re-deploy a function when its S3 deployment object is overwritten. See [Reactive S3 Sync](#reactive-s3-sync-configuration) |
+| `FLOCI_SERVICES_LAMBDA_REACTIVE_S3_SYNC_ASYNC` | `false` | Run that re-deploy in the background so the upload does not wait for it |
| `FLOCI_SERVICES_LAMBDA_DOCKER_NETWORK` | *(unset)* | Docker network to attach Lambda containers to (overrides `FLOCI_SERVICES_DOCKER_NETWORK`) |
| `FLOCI_SERVICES_LAMBDA_DOCKER_FLAGS` | *(unset)* | Additional Docker flags applied to Lambda containers, including `--env`, `--volume`, `--publish`, `--add-host`, `--dns`, `--label`, `--network`, `--user`, `--privileged`, and `--platform`. Explicit `--dns` entries replace automatic embedded DNS and public fallback resolvers. Published ports support `host:container` and `127.0.0.1:host:container` forms |
| `FLOCI_SERVICES_LAMBDA_EXTRA_HOSTS` | *(unset)* | Comma-separated `hostname:ip` entries added to each Lambda container's `/etc/hosts`; `ip` may be `host-gateway`, mirroring `docker run --add-host` |
diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
index 21f30ff7b1..f8642be3af 100644
--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
@@ -3272,6 +3272,30 @@ public interface EmulatorConfig {
*/
Optional<List<String>> allowedPaths();
}
+
+ ReactiveS3Sync reactiveS3Sync();
+
+ interface ReactiveS3Sync {
+ /**
+ * When true, uploading a new object to the S3 bucket and key a function was deployed
+ * from re-deploys that function's {@code $LATEST} and drains its warm containers. When
+ * false, the object is only stored, as in AWS.
+ *
+ * Env var: FLOCI_SERVICES_LAMBDA_REACTIVE_S3_SYNC_ENABLED
+ */
+ @WithDefault("true")
+ boolean enabled();
+
+ /**
+ * When true, the re-deploy runs in the background so the S3 upload returns at once;
+ * the function reports {@code LastUpdateStatus=InProgress} until it finishes. When
+ * false, the upload returns only after the function has been re-deployed.
+ *
+ * Env var: FLOCI_SERVICES_LAMBDA_REACTIVE_S3_SYNC_ASYNC
+ */
+ @WithDefault("false")
+ boolean async();
+ }
}
interface Ec2ServiceConfig {
diff --git a/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaController.java b/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaController.java
index c42324ae48..f471b4e693 100644
--- a/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaController.java
+++ b/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaController.java
@@ -746,7 +746,11 @@ public class LambdaController {
.format(Instant.ofEpochMilli(fn.getLastModified()).atOffset(ZoneOffset.UTC)));
node.put("RevisionId", fn.getRevisionId());
node.put("Version", fn.getVersion());
- node.put("LastUpdateStatus", "Successful");
+ node.put("LastUpdateStatus", lambdaService.lastUpdateStatus(fn));
+ String lastUpdateStatusReason = lambdaService.lastUpdateStatusReason(fn);
+ if (lastUpdateStatusReason != null) {
+ node.put("LastUpdateStatusReason", lastUpdateStatusReason);
+ }
// Architectures — always present; default x86_64
ArrayNode archNode = node.putArray("Architectures");
diff --git a/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaService.java b/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaService.java
index e6e388e953..f9fb80efa7 100644
--- a/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaService.java
@@ -67,6 +67,7 @@ import java.util.Set;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Executor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.function.Consumer;
@@ -165,6 +166,9 @@ public class LambdaService implements ResourceProvider {
"Environment", "EphemeralStorage", "TracingConfig", "DeadLetterConfig",
"VpcConfig", "SnapStart", "LoggingConfig", "ImageConfig", "DurableConfig");
+ private static final Executor REACTIVE_SYNC_THREADS =
+ task -> Thread.ofVirtual().name("lambda-reactive-s3-sync").start(task);
+
private final LambdaFunctionStore functionStore;
private final LambdaExecutorService executorService;
private final LambdaConcurrencyLimiter concurrencyLimiter;
@@ -183,6 +187,9 @@ public class LambdaService implements ResourceProvider {
private final DynamoDbStreamsEventSourcePoller dynamodbStreamsPoller;
private final StorageFactory storageFactory;
private final LambdaLayerService layerService;
+ private final Map<String, PendingReactiveSync> pendingReactiveSyncs = new ConcurrentHashMap<>();
+ private final Map<String, Integer> runningReactiveSyncs = new ConcurrentHashMap<>();
+ private final Map<String, String> failedReactiveSyncs = new ConcurrentHashMap<>();
private final Ec2Service ec2Service;
/** Null in the constructors tests use, exactly as the other optional collaborators above are. */
private final CustomResourceLiveness customResourceLiveness;
@@ -775,6 +782,7 @@ public class LambdaService implements ResourceProvider {
// Drain warm containers — they have stale code mounted
warmPool.drainEnvironment(fn);
+ failedReactiveSyncs.remove(fn.getFunctionArn());
functionStore.save(region, fn);
LOG.infov("Updated code for function: {0}", functionName);
@@ -996,6 +1004,8 @@ public class LambdaService implements ResourceProvider {
warmPool.drainEnvironment(fn);
functionStore.save(region, fn);
+ // LastUpdateStatus reports the last update of either kind, so this one replaces a failed sync's.
+ failedReactiveSyncs.remove(fn.getFunctionArn());
LOG.infov("Updated configuration for function: {0}", functionName);
return fn;
}
@@ -1116,6 +1126,7 @@ public class LambdaService implements ResourceProvider {
reclaimLegacyCodeDirectoryIfUnused(functionName);
versionCounters.remove(versionCounterKey(region, fn));
versionCounters.remove(legacyVersionCounterKey(region, functionName));
+ failedReactiveSyncs.remove(arn);
if (aliasStore != null) {
for (LambdaAlias alias : aliasStore.list(region, functionName)) {
aliasStore.delete(region, functionName, alias.getName());
@@ -4140,6 +4151,9 @@ public class LambdaService implements ResourceProvider {
* functions linked to the updated object.
*/
public void onS3ObjectUpdated(@Observes S3ObjectUpdatedEvent event) {
+ if (!reactiveS3SyncEnabled()) {
+ return;
+ }
LOG.debugv("Observing S3 update: {0}/{1}", event.bucketName(), event.key());
// For simplicity, we scan all functions in the default region
// Most local dev setups use a single region.
@@ -4148,32 +4162,118 @@ public class LambdaService implements ResourceProvider {
// other account would silently never hot-reload from S3.
String region = regionResolver.getDefaultRegion();
List<LambdaFunction> functions = functionStore.listAllAccounts(region);
+ boolean async = reactiveS3SyncAsync();
for (LambdaFunction fn : functions) {
if (!event.bucketName().equals(fn.getS3Bucket()) || !event.key().equals(fn.getS3Key())) {
continue;
}
- synchronized (lockForConcurrencyOp(fn.getFunctionArn())) {
- if (fn.isHotReload()) {
- continue;
+ if (!async) {
+ syncFromS3(fn, event.bucketName(), event.key(), region);
+ continue;
+ }
+ // The newest upload replaces any source still waiting, and a queued sync reads its object
+ // when it starts, so one queued sync per function covers every upload before it. The
+ // uploader's account goes with it: S3 resolves the bucket in the caller's account, and
+ // the background thread has no request of its own.
+ String functionArn = fn.getFunctionArn();
+ PendingReactiveSync source = new PendingReactiveSync(regionResolver.getAccountId(),
+ event.bucketName(), event.key());
+ if (pendingReactiveSyncs.put(functionArn, source) == null) {
+ String ownerAccountId = ownerAccount(fn);
+ String functionName = fn.getFunctionName();
+ REACTIVE_SYNC_THREADS.execute(() -> syncQueued(ownerAccountId, region, functionName, functionArn));
+ }
+ }
+ }
+
+ /** Re-reads the function, which may have been redeployed or deleted while the sync was queued. */
+ private void syncQueued(String ownerAccountId, String region, String functionName, String functionArn) {
+ // Marked running before the pending source is taken, so the function never reports neither.
+ runningReactiveSyncs.merge(functionArn, 1, Integer::sum);
+ try {
+ synchronized (lockForConcurrencyOp(functionArn)) {
+ PendingReactiveSync source = pendingReactiveSyncs.remove(functionArn);
+ if (source == null) {
+ return;
}
- if (event.bucketName().equals(fn.getS3Bucket()) && event.key().equals(fn.getS3Key())) {
- LOG.infov("Reactive S3 Sync: updating function {0} from s3://{1}/{2}",
- fn.getFunctionName(), event.bucketName(), event.key());
- try {
- try (DeploymentPackage zip = readS3Package(fn, event.bucketName(), event.key(), region)) {
- extractZipCodePackage(fn, zip, region);
- }
- fn.setLastModified(Instant.now().toEpochMilli());
- fn.setRevisionId(UUID.randomUUID().toString());
- functionStore.saveForAccount(ownerAccount(fn), region, fn);
-
- // Push to warm workers
- warmPool.pushCodeUpdate(fn);
- } catch (Exception e) {
- LOG.warnv("Failed reactive sync for function {0}: {1}", fn.getFunctionName(), e.getMessage());
- }
+ RequestScopes.runAs(source.accountId(), () ->
+ functionStore.getForAccount(ownerAccountId, region, functionName)
+ .ifPresent(fn -> deployFromS3(fn, source.bucketName(), source.key(), region)));
+ }
+ } finally {
+ runningReactiveSyncs.computeIfPresent(functionArn, (arn, count) -> count == 1 ? null : count - 1);
+ }
+ }
+
+ private void syncFromS3(LambdaFunction fn, String bucketName, String key, String region) {
+ String functionArn = fn.getFunctionArn();
+ runningReactiveSyncs.merge(functionArn, 1, Integer::sum);
+ try {
+ synchronized (lockForConcurrencyOp(functionArn)) {
+ deployFromS3(fn, bucketName, key, region);
+ }
+ } finally {
+ runningReactiveSyncs.computeIfPresent(functionArn, (arn, count) -> count == 1 ? null : count - 1);
+ }
+ }
+
+ /** Callers hold the function's lock. */
+ private void deployFromS3(LambdaFunction fn, String bucketName, String key, String region) {
+ if (fn.isHotReload()) {
+ return;
+ }
+ if (bucketName.equals(fn.getS3Bucket()) && key.equals(fn.getS3Key())) {
+ LOG.infov("Reactive S3 Sync: updating function {0} from s3://{1}/{2}",
+ fn.getFunctionName(), bucketName, key);
+ try {
+ try (DeploymentPackage zip = readS3Package(fn, bucketName, key, region)) {
+ extractZipCodePackage(fn, zip, region);
}
+ fn.setLastModified(Instant.now().toEpochMilli());
+ fn.setRevisionId(UUID.randomUUID().toString());
+ functionStore.saveForAccount(ownerAccount(fn), region, fn);
+
+ // Push to warm workers
+ warmPool.pushCodeUpdate(fn);
+ failedReactiveSyncs.remove(fn.getFunctionArn());
+ } catch (Exception e) {
+ LOG.warnv("Failed reactive sync for function {0}: {1}", fn.getFunctionName(), e.getMessage());
+ failedReactiveSyncs.put(fn.getFunctionArn(), "Reactive S3 sync from s3://" + bucketName
+ + "/" + key + " failed: " + e.getMessage());
}
}
}
+
+ /**
+ * The function's LastUpdateStatus, which only a reactive S3 sync moves off Successful:
+ * InProgress from the upload until the re-deploy of {@code $LATEST} finishes, Failed when the
+ * last one could not. Held in memory, so a restart cannot leave an update reported as running.
+ */
+ public String lastUpdateStatus(LambdaFunction fn) {
+ if (!"$LATEST".equals(fn.getVersion())) {
+ return "Successful";
+ }
+ String functionArn = fn.getFunctionArn();
+ // Pending first: a sync marks itself running before it takes the pending source.
+ if (pendingReactiveSyncs.containsKey(functionArn) || runningReactiveSyncs.containsKey(functionArn)) {
+ return "InProgress";
+ }
+ return failedReactiveSyncs.containsKey(functionArn) ? "Failed" : "Successful";
+ }
+
+ /** Why the last reactive S3 sync of {@code $LATEST} failed, or null when it did not. */
+ public String lastUpdateStatusReason(LambdaFunction fn) {
+ return "Failed".equals(lastUpdateStatus(fn)) ? failedReactiveSyncs.get(fn.getFunctionArn()) : null;
+ }
+
+ private boolean reactiveS3SyncEnabled() {
+ return config == null || config.services().lambda().reactiveS3Sync().enabled();
+ }
+
+ private boolean reactiveS3SyncAsync() {
+ return config != null && config.services().lambda().reactiveS3Sync().async();
+ }
+
+ private record PendingReactiveSync(String accountId, String bucketName, String key) {
+ }
}
diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml
index d8d8b6d4ed..cd10683868 100644
--- a/src/main/resources/application.yml
+++ b/src/main/resources/application.yml
@@ -467,6 +467,9 @@ floci:
# init-image: busybox:1.36 # Init container image (needs sh, wget, unzip)
hot-reload:
enabled: false
+ reactive-s3-sync:
+ enabled: true # Re-deploy a function when the S3 object it was deployed from is overwritten
+ async: false # Re-deploy in the background so the S3 upload does not wait for it
apigateway:
enabled: true
vtl-max-loops: 10000 # Maximum #foreach iterations a VTL mapping template may execute (truncates the loop, does not fail the template)
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambda/LambdaAsyncReactiveSyncIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/lambda/LambdaAsyncReactiveSyncIntegrationTest.java
new file mode 100644
index 0000000000..8e9962f690
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/lambda/LambdaAsyncReactiveSyncIntegrationTest.java
@@ -0,0 +1,77 @@
+package io.github.hectorvent.floci.services.lambda;
+
+import io.github.hectorvent.floci.services.lambdaweb.LambdaWebZips;
+import io.github.hectorvent.floci.testing.LambdaCodeSourcesProfile;
+import io.quarkus.test.junit.QuarkusTest;
+import io.quarkus.test.junit.TestProfile;
+import org.junit.jupiter.api.Test;
+
+import java.security.MessageDigest;
+import java.time.Duration;
+import java.util.Base64;
+
+import static io.restassured.RestAssured.given;
+import static org.awaitility.Awaitility.await;
+import static org.hamcrest.Matchers.equalTo;
+
+/**
+ * With asynchronous reactive S3 sync, the re-deploy runs on a background thread with no request of
+ * its own, so it must read the object in the uploader's account rather than the default one.
+ */
+@QuarkusTest
+@TestProfile(LambdaCodeSourcesProfile.class)
+class LambdaAsyncReactiveSyncIntegrationTest {
+
+ private static final String ACCOUNT = "000000000021";
+ private static final String S3_AUTH =
+ "AWS4-HMAC-SHA256 Credential=" + ACCOUNT + "/20260215/us-east-1/s3/aws4_request, SignedHeaders=host, Signature=abc";
+ private static final String LAMBDA_AUTH =
+ "AWS4-HMAC-SHA256 Credential=" + ACCOUNT + "/20260215/us-east-1/lambda/aws4_request, SignedHeaders=host, Signature=abc";
+ private static final String BUCKET = "async-reactive-sync-bucket";
+ private static final String KEY = "function.zip";
+ private static final String FN = "async-reactive-sync-fn";
+
+ @Test
+ void uploadFromANonDefaultAccountRedeploysTheFunctionInTheBackground() throws Exception {
+ byte[] v1 = LambdaWebZips.indexJs("exports.handler = async () => 'v1';");
+ byte[] v2 = LambdaWebZips.indexJs("exports.handler = async () => 'v2';");
+ given().header("Authorization", S3_AUTH).when().put("/" + BUCKET).then().statusCode(200);
+ given().header("Authorization", S3_AUTH).body(v1).when().put("/" + BUCKET + "/" + KEY).then().statusCode(200);
+ given()
+ .header("Authorization", LAMBDA_AUTH)
+ .contentType("application/json")
+ .body("""
+ {
+ "FunctionName": "%s",
+ "Runtime": "nodejs20.x",
+ "Role": "arn:aws:iam::%s:role/role",
+ "Handler": "index.handler",
+ "Code": {
+ "S3Bucket": "%s",
+ "S3Key": "%s"
+ }
+ }
+ """.formatted(FN, ACCOUNT, BUCKET, KEY))
+ .when()
+ .post("/2015-03-31/functions")
+ .then()
+ .statusCode(201)
+ .body("CodeSha256", equalTo(sha256(v1)));
+
+ given().header("Authorization", S3_AUTH).body(v2).when().put("/" + BUCKET + "/" + KEY).then().statusCode(200);
+
+ await().atMost(Duration.ofSeconds(10)).pollInterval(Duration.ofMillis(100)).untilAsserted(() ->
+ given()
+ .header("Authorization", LAMBDA_AUTH)
+ .when()
+ .get("/2015-03-31/functions/" + FN + "/configuration")
+ .then()
+ .statusCode(200)
+ .body("LastUpdateStatus", equalTo("Successful"))
+ .body("CodeSha256", equalTo(sha256(v2))));
+ }
+
+ private static String sha256(byte[] data) throws Exception {
+ return Base64.getEncoder().encodeToString(MessageDigest.getInstance("SHA-256").digest(data));
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambda/LambdaServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/lambda/LambdaServiceTest.java
index 2baadd841e..a82bfaeebf 100644
--- a/src/test/java/io/github/hectorvent/floci/services/lambda/LambdaServiceTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/lambda/LambdaServiceTest.java
@@ -2354,13 +2354,18 @@ class LambdaServiceTest {
}
private static LambdaService s3BackedService(S3Service s3, WarmPool warmPool, ZipExtractor extractor) {
+ return s3BackedService(s3, warmPool, extractor, null);
+ }
+
+ private static LambdaService s3BackedService(S3Service s3, WarmPool warmPool, ZipExtractor extractor,
+ EmulatorConfig config) {
StorageFactory storageFactory = mock(StorageFactory.class);
when(storageFactory.create(anyString(), anyString(), any()))
.thenAnswer(inv -> AccountAwareStorageBackend.inMemory("000000000000"));
LambdaFunctionStore functionStore = new LambdaFunctionStore(new InMemoryStorage<String, LambdaFunction>());
LambdaAliasStore aliasStore = new LambdaAliasStore(storageFactory);
return new LambdaService(functionStore, null, new LambdaConcurrencyLimiter(),
- warmPool, new CodeStore(Path.of("target/test-data/lambda-code")), extractor, null,
+ warmPool, new CodeStore(Path.of("target/test-data/lambda-code")), extractor, config,
new RegionResolver(REGION, "000000000000"), null, aliasStore,
new LambdaTargetResolver(functionStore, aliasStore), s3, null, null, null, null,
storageFactory, null, null, null, null, new ObjectMapper());
@@ -2420,6 +2425,201 @@ class LambdaServiceTest {
}
}
+ @Test
+ void reactiveSyncDisabled_leavesTheFunctionOnItsDeployedCode() throws Exception {
+ S3Service s3 = mock(S3Service.class);
+ WarmPool warmPool = mock(WarmPool.class);
+ LambdaService syncingService = s3BackedService(s3, warmPool, new ZipExtractor(),
+ reactiveS3SyncConfig(false, false));
+ String name = "reactive-disabled-" + UUID.randomUUID();
+ when(s3.openObject("code-bucket", "function.zip", null)).thenReturn(heldObject(zipWithBody("v1")));
+ Map<String, Object> request = baseRequest(name);
+ request.put("Code", Map.of("S3Bucket", "code-bucket", "S3Key", "function.zip"));
+ LambdaFunction latest = syncingService.createFunction(REGION, request);
+
+ syncingService.onS3ObjectUpdated(new S3ObjectUpdatedEvent("code-bucket", "function.zip"));
+
+ verify(s3, times(1)).openObject("code-bucket", "function.zip", null);
+ verify(warmPool, never()).pushCodeUpdate(any());
+ assertEquals("v1", Files.readString(Path.of(latest.getCodeLocalPath()).resolve("index.js")).trim());
+ }
+
+ @Test
+ void asyncReactiveSync_returnsBeforeTheRedeployAndReportsInProgressUntilItFinishes() throws Exception {
+ S3Service s3 = mock(S3Service.class);
+ WarmPool warmPool = mock(WarmPool.class);
+ ZipExtractor extractor = spy(new ZipExtractor());
+ LambdaService syncingService = s3BackedService(s3, warmPool, extractor, reactiveS3SyncConfig(true, true));
+ String name = "reactive-async-" + UUID.randomUUID();
+ when(s3.openObject("code-bucket", "function.zip", null))
+ .thenReturn(heldObject(zipWithBody("v1")))
+ .thenReturn(heldObject(zipWithBody("v2")));
+ Map<String, Object> request = baseRequest(name);
+ request.put("Code", Map.of("S3Bucket", "code-bucket", "S3Key", "function.zip"));
+ LambdaFunction latest = syncingService.createFunction(REGION, request);
+ CountDownLatch release = new CountDownLatch(1);
+ doAnswer(inv -> {
+ assertTrue(release.await(5, TimeUnit.SECONDS));
+ return inv.callRealMethod();
+ }).when(extractor).extractTo(any(byte[].class), any(Path.class), anyInt(), any(Runnable.class));
+ try {
+ syncingService.onS3ObjectUpdated(new S3ObjectUpdatedEvent("code-bucket", "function.zip"));
+
+ assertEquals("InProgress", syncingService.lastUpdateStatus(latest));
+ verify(warmPool, never()).pushCodeUpdate(any());
+ } finally {
+ release.countDown();
+ }
+
+ verify(warmPool, timeout(5000)).pushCodeUpdate(latest);
+ awaitLastUpdateStatus(syncingService, latest, "Successful");
+ assertEquals("v2", Files.readString(Path.of(latest.getCodeLocalPath()).resolve("index.js")).trim());
+ }
+
+ @Test
+ void asyncReactiveSync_coalescesUploadsThatArriveWhileOneIsRunning() throws Exception {
+ S3Service s3 = mock(S3Service.class);
+ WarmPool warmPool = mock(WarmPool.class);
+ ZipExtractor extractor = spy(new ZipExtractor());
+ LambdaService syncingService = s3BackedService(s3, warmPool, extractor, reactiveS3SyncConfig(true, true));
+ String name = "reactive-coalesce-" + UUID.randomUUID();
+ when(s3.openObject("code-bucket", "function.zip", null))
+ .thenReturn(heldObject(zipWithBody("v1")))
+ .thenReturn(heldObject(zipWithBody("v2")))
+ .thenReturn(heldObject(zipWithBody("v5")));
+ Map<String, Object> request = baseRequest(name);
+ request.put("Code", Map.of("S3Bucket", "code-bucket", "S3Key", "function.zip"));
+ LambdaFunction latest = syncingService.createFunction(REGION, request);
+ CountDownLatch extracting = new CountDownLatch(1);
+ CountDownLatch release = new CountDownLatch(1);
+ doAnswer(inv -> {
+ extracting.countDown();
+ assertTrue(release.await(5, TimeUnit.SECONDS));
+ return inv.callRealMethod();
+ }).when(extractor).extractTo(any(byte[].class), any(Path.class), anyInt(), any(Runnable.class));
+ try {
+ syncingService.onS3ObjectUpdated(new S3ObjectUpdatedEvent("code-bucket", "function.zip"));
+ assertTrue(extracting.await(5, TimeUnit.SECONDS));
+ for (int upload = 3; upload <= 5; upload++) {
+ syncingService.onS3ObjectUpdated(new S3ObjectUpdatedEvent("code-bucket", "function.zip"));
+ }
+ } finally {
+ release.countDown();
+ }
+
+ verify(warmPool, timeout(5000).times(2)).pushCodeUpdate(latest);
+ awaitLastUpdateStatus(syncingService, latest, "Successful");
+ // Create, the running sync, and one queued sync for the three uploads behind it.
+ verify(s3, times(3)).openObject("code-bucket", "function.zip", null);
+ assertEquals("v5", Files.readString(Path.of(latest.getCodeLocalPath()).resolve("index.js")).trim());
+ }
+
+ @Test
+ void asyncReactiveSync_deploysAnUploadToANewSourceWhileTheOldOneIsStillQueued() throws Exception {
+ S3Service s3 = mock(S3Service.class);
+ WarmPool warmPool = mock(WarmPool.class);
+ LambdaService syncingService = s3BackedService(s3, warmPool, new ZipExtractor(),
+ reactiveS3SyncConfig(true, true));
+ String name = "reactive-new-source-" + UUID.randomUUID();
+ when(s3.openObject("code-bucket", "old.zip", null)).thenReturn(heldObject(zipWithBody("v1")));
+ when(s3.openObject("code-bucket", "new.zip", null))
+ .thenReturn(heldObject(zipWithBody("v2")))
+ .thenReturn(heldObject(zipWithBody("v3")));
+ Map<String, Object> request = baseRequest(name);
+ request.put("Code", Map.of("S3Bucket", "code-bucket", "S3Key", "old.zip"));
+ LambdaFunction latest = syncingService.createFunction(REGION, request);
+
+ // Holding the function's lock keeps the old source's sync queued behind it.
+ synchronized (syncingService.lockForConcurrencyOp(latest.getFunctionArn())) {
+ syncingService.onS3ObjectUpdated(new S3ObjectUpdatedEvent("code-bucket", "old.zip"));
+ syncingService.updateFunctionCode(REGION, name, Map.of("S3Bucket", "code-bucket", "S3Key", "new.zip"));
+ syncingService.onS3ObjectUpdated(new S3ObjectUpdatedEvent("code-bucket", "new.zip"));
+ assertEquals("InProgress", syncingService.lastUpdateStatus(latest));
+ }
+
+ verify(warmPool, timeout(5000)).pushCodeUpdate(latest);
+ awaitLastUpdateStatus(syncingService, latest, "Successful");
+ verify(s3, times(1)).openObject("code-bucket", "old.zip", null);
+ verify(s3, times(2)).openObject("code-bucket", "new.zip", null);
+ assertEquals("v3", Files.readString(Path.of(latest.getCodeLocalPath()).resolve("index.js")).trim());
+ }
+
+ @Test
+ void failedReactiveSync_reportsFailedUntilTheNextCodeUpdate() throws Exception {
+ S3Service s3 = mock(S3Service.class);
+ WarmPool warmPool = mock(WarmPool.class);
+ LambdaService syncingService = s3BackedService(s3, warmPool, new ZipExtractor(),
+ reactiveS3SyncConfig(true, false));
+ LambdaFunction latest = functionWhoseLastSyncFailed(syncingService, s3);
+
+ assertEquals("Failed", syncingService.lastUpdateStatus(latest));
+ assertEquals("Reactive S3 sync from s3://code-bucket/function.zip failed: The specified key does not exist.",
+ syncingService.lastUpdateStatusReason(latest));
+ verify(warmPool, never()).pushCodeUpdate(any());
+
+ syncingService.updateFunctionCode(REGION, latest.getFunctionName(), Map.of("ZipFile", zipWithBody("v2")));
+
+ assertEquals("Successful", syncingService.lastUpdateStatus(latest));
+ assertNull(syncingService.lastUpdateStatusReason(latest));
+ }
+
+ @Test
+ void failedReactiveSync_isClearedByAConfigurationUpdate() throws Exception {
+ S3Service s3 = mock(S3Service.class);
+ LambdaService syncingService = s3BackedService(s3, mock(WarmPool.class), new ZipExtractor(),
+ reactiveS3SyncConfig(true, false));
+ LambdaFunction latest = functionWhoseLastSyncFailed(syncingService, s3);
+
+ syncingService.updateFunctionConfiguration(REGION, latest.getFunctionName(),
+ Map.of("Environment", Map.of("Variables", Map.of("STAGE", "ci"))));
+
+ assertEquals("Successful", syncingService.lastUpdateStatus(latest));
+ assertNull(syncingService.lastUpdateStatusReason(latest));
+ }
+
+ /**
+ * A function deployed from {@code s3://code-bucket/function.zip} whose sync after the object was
+ * overwritten failed because the object had gone. {@code s3} is the mock {@code service} reads.
+ */
+ private LambdaFunction functionWhoseLastSyncFailed(LambdaService service, S3Service s3)
+ throws Exception {
+ when(s3.openObject("code-bucket", "function.zip", null))
+ .thenReturn(heldObject(zipWithBody("v1")))
+ .thenThrow(new AwsException("NoSuchKey", "The specified key does not exist.", 404));
+ Map<String, Object> request = baseRequest("reactive-failed-" + UUID.randomUUID());
+ request.put("Code", Map.of("S3Bucket", "code-bucket", "S3Key", "function.zip"));
+ LambdaFunction latest = service.createFunction(REGION, request);
+ service.onS3ObjectUpdated(new S3ObjectUpdatedEvent("code-bucket", "function.zip"));
+ return latest;
+ }
+
+ private static EmulatorConfig reactiveS3SyncConfig(boolean enabled, boolean async) {
+ EmulatorConfig cfg = mock(EmulatorConfig.class);
+ EmulatorConfig.ServicesConfig svc = mock(EmulatorConfig.ServicesConfig.class);
+ EmulatorConfig.LambdaServiceConfig lambdaCfg = mock(EmulatorConfig.LambdaServiceConfig.class);
+ EmulatorConfig.LambdaServiceConfig.ReactiveS3Sync sync =
+ mock(EmulatorConfig.LambdaServiceConfig.ReactiveS3Sync.class);
+ when(cfg.services()).thenReturn(svc);
+ when(svc.lambda()).thenReturn(lambdaCfg);
+ when(lambdaCfg.reactiveS3Sync()).thenReturn(sync);
+ when(lambdaCfg.defaultTimeoutSeconds()).thenReturn(3);
+ when(lambdaCfg.defaultMemoryMb()).thenReturn(128);
+ when(lambdaCfg.zipMaxEntries()).thenReturn(ZipExtractor.DEFAULT_MAX_ENTRIES);
+ when(lambdaCfg.executor()).thenReturn("docker");
+ when(sync.enabled()).thenReturn(enabled);
+ when(sync.async()).thenReturn(async);
+ return cfg;
+ }
+
+ private static void awaitLastUpdateStatus(LambdaService service, LambdaFunction fn, String expected)
+ throws InterruptedException {
+ long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5);
+ while (!expected.equals(service.lastUpdateStatus(fn)) && System.nanoTime() < deadline) {
+ Thread.sleep(10);
+ }
+ assertEquals(expected, service.lastUpdateStatus(fn));
+ }
+
@Test
void deletingOneVersionPreservesAnotherVersionAndLatest() throws Exception {
String name = "shared-version-code";
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebZips.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebZips.java
index 75901934ce..b1647925c9 100644
--- a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebZips.java
+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebZips.java
@@ -10,7 +10,7 @@ import java.util.Arrays;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
-/** Code packages for tests that create Web Function revisions. */
+/** Code packages for tests that deploy function code, Web Function revisions among them. */
public final class LambdaWebZips {
private LambdaWebZips() {
diff --git a/src/test/java/io/github/hectorvent/floci/testing/LambdaCodeSourcesProfile.java b/src/test/java/io/github/hectorvent/floci/testing/LambdaCodeSourcesProfile.java
index 43755d425b..562067aac5 100644
--- a/src/test/java/io/github/hectorvent/floci/testing/LambdaCodeSourcesProfile.java
+++ b/src/test/java/io/github/hectorvent/floci/testing/LambdaCodeSourcesProfile.java
@@ -6,10 +6,11 @@ import java.util.Map;
/**
* Sets where a Lambda function's code may come from, for the classes that test it: foreign-account
- * layer ARNs are accepted (recorded verbatim, never mounted), and hot-reload mounts are limited to
- * an allow-list under {@code /home/ci/code}. Each setting only changes the requests its own class
- * makes, so the two classes share this profile, and Quarkus builds one application for both
- * rather than one each.
+ * layer ARNs are accepted (recorded verbatim, never mounted), hot-reload mounts are limited to an
+ * allow-list under {@code /home/ci/code}, and a reactive S3 sync re-deploys a function in the
+ * background rather than inside the upload of its code. Each setting only changes the requests its
+ * own class makes, so the three classes share this profile, and Quarkus builds one application for
+ * all of them rather than one each.
*/
public class LambdaCodeSourcesProfile implements QuarkusTestProfile {
@@ -17,6 +18,7 @@ public class LambdaCodeSourcesProfile implements QuarkusTestProfile {
public Map<String, String> getConfigOverrides() {
return Map.of(
"floci.services.lambda.accept-external-layer-arns", "true",
- "floci.services.lambda.hot-reload.allowed-paths", "/home/ci/code");
+ "floci.services.lambda.hot-reload.allowed-paths", "/home/ci/code",
+ "floci.services.lambda.reactive-s3-sync.async", "true");
}
}
diff --git a/src/test/resources/application.yml b/src/test/resources/application.yml
index 4bfc0e91e5..cba3a9c43a 100644
--- a/src/test/resources/application.yml
+++ b/src/test/resources/application.yml
@@ -168,6 +168,9 @@ floci:
# aws-config-path:
hot-reload:
enabled: true
+ reactive-s3-sync:
+ enabled: true
+ async: false
apigateway:
enabled: true
vtl-max-loops: 10000