Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(lambdaweb): serve each region of a PerRegion endpoint in that region (#6423)

TATSUNO “Taz” Yasuhiro72bbc809b435
diff --git a/docs/services/lambda-web-functions.md b/docs/services/lambda-web-functions.md
index bf7e6785d5..d61a43d7ff 100644
--- a/docs/services/lambda-web-functions.md
+++ b/docs/services/lambda-web-functions.md
@@ -43,8 +43,8 @@ The `Lambda Web` service model (apiVersion 2025-03-07, `aws lambda-web` in the A
- **Function names** are unique across the regions of an account, as on AWS. A name that another region already has is `409 ConflictException` "A function named <name> already exists in the account. Function names must be unique across all regions in an account.".
- **ARNs** given as identifiers must name the request's region and the caller's account, as on AWS. A revision or endpoint ARN resolves by its id or name only.
- **Page tokens** are base64 cursors that carry the request and the position of the page, not the UUIDs AWS issues. A token reused with other parameters gets the AWS "does not match the request" error, and any other token the "invalid or has expired" error.
-- **Domain names** take the form `<id>.001.lambda-web.<region>.<floci host>` and `<id>.lambda-web.global.<floci host>`, with the host and port of `floci.base-url`. A `PerRegion` endpoint gets one domain per region in `regionalEndpoints`.
-- **Not implemented yet:** SigV4a on `MultiRegion` endpoints, the check that a policy's principal exists, the automatic deletion of revisions past 50, the metadata API, and serving a `MultiRegion` or `PerRegion` endpoint in its other regions.
+- **Domain names** take the form `<id>.001.lambda-web.<region>.<floci host>` and `<id>.lambda-web.global.<floci host>`, with the host and port of `floci.base-url`. A `PerRegion` endpoint gets one domain per region in `regionalEndpoints`, all with the same id, as on AWS.
+- **Not implemented yet:** SigV4a on `MultiRegion` endpoints, the check that a policy's principal exists, and the automatic deletion of revisions past 50.
### `aws lambda-web deploy`
@@ -73,9 +73,11 @@ An execution environment is a `public.ecr.aws/lambda/nodejs:24` container that a
Each endpoint scales on its own, as on AWS. An environment takes up to the revision's `maxConcurrencyPerEnvironment` requests at once, the least busy one first, so requests rotate over the environments. A request that finds every environment full starts another one, or waits for one that is starting and has room. When the endpoint already runs `maxEnvironments`, an idle environment of another revision makes room, and without one the request is refused with `429` and `Platform.EndpointMaxEnvThrottle`. A `rateLimit` admits a burst of up to `rateLimit` requests that refills at `rateLimit` per second, and refuses the rest with `429` and `Platform.EndpointRequestThrottle`. AWS adjusts the concurrency of each environment under `maxConcurrencyPerEnvironment` from the measured load, so a burst there can spread over more environments than in Floci, and the account quotas are not applied.
+Each regional domain of a `PerRegion` endpoint is served in its own region, as on AWS. Its environments see that region in `AWS_REGION` and `AWS_DEFAULT_REGION` and get that region's zone from the metadata endpoint, and `maxEnvironments` and `rateLimit` hold in each region on its own. Floci serves the global domain of a `MultiRegion` endpoint in the home region, where AWS serves it from a region near the caller.
+
Requests are forwarded over HTTP/1.1 with the headers AWS adds and drops: `lambda-web-request-id`, `lambda-web-endpoint-name`, `x-forwarded-proto`, `x-forwarded-port` and `Connection: close` are set, the client address is appended as an `X-Forwarded-For` line, and `Connection`, `Keep-Alive`, `Upgrade`, `Expect`, `Trailer`, the `Proxy-*` headers, `X-Real-IP`, the method override headers and any `lambda-web-*` header the client sent are dropped. A body without a `Content-Length` reaches the app chunked. The path reaches the app without dot segments, as on AWS, and repeated slashes stay. Responses carry `lambda-web-request-id` and `lambda-web-logstream`, lose their connection headers and trailers, and stream as the app writes them. Requests with two revisions in `revisionWeights` are split by weight.
-An `IamAuth` endpoint takes a request signed with SigV4 for the service `lambda` in the endpoint's region, in the `Authorization` header or in a presigned query. The request must carry `X-Amz-Content-SHA256`. Lambda takes its value as the payload hash and does not read the body, so `UNSIGNED-PAYLOAD` works. The signing time must be within five minutes. Floci verifies keys whose secret it holds: `test` with the secret `test`, IAM user keys and STS credentials with their session token. Any other key is refused. When `floci.services.iam.enforcement-enabled` is on, the signer also needs `lambda:InvokeWebFunctionEndpoint` on the endpoint ARN. Within one account the identity policy or the function's resource policy can allow it, and across accounts both must. `lambda:Resource/WebFunctionAuthType` and the function's tags as `aws:ResourceTag/<key>` are in the context. The `test` key is not evaluated, as elsewhere in Floci. A `MultiRegion` endpoint also takes SigV4 for the region that serves the request, which in Floci is always the home region. Its SigV4a requests are refused, because Floci does not verify SigV4a yet. On every endpoint, an `Authorization` that is an AWS signature is removed before the request reaches the app, together with the `X-Amz-Security-Token` header. Other `Authorization` values pass. A presigned query reaches the app unchanged, as on AWS.
+An `IamAuth` endpoint takes a request signed with SigV4 for the service `lambda` in the region of the domain, in the `Authorization` header or in a presigned query. The request must carry `X-Amz-Content-SHA256`. Lambda takes its value as the payload hash and does not read the body, so `UNSIGNED-PAYLOAD` works. The signing time must be within five minutes. Floci verifies keys whose secret it holds: `test` with the secret `test`, IAM user keys and STS credentials with their session token. Any other key is refused. When `floci.services.iam.enforcement-enabled` is on, the signer also needs `lambda:InvokeWebFunctionEndpoint` on the endpoint ARN. Within one account the identity policy or the function's resource policy can allow it, and across accounts both must. `lambda:Resource/WebFunctionAuthType` and the function's tags as `aws:ResourceTag/<key>` are in the context, and `aws:RequestedRegion` is the region of the domain, as on AWS. The `test` key is not evaluated, as elsewhere in Floci. A `MultiRegion` endpoint also takes SigV4 for the region that serves the request, which in Floci is always the home region. Its SigV4a requests are refused, because Floci does not verify SigV4a yet. On every endpoint, an `Authorization` that is an AWS signature is removed before the request reaches the app, together with the `X-Amz-Security-Token` header. Other `Authorization` values pass. A presigned query reaches the app unchanged, as on AWS.
The revision's `timeoutSeconds` bounds the whole exchange, as on AWS: a response that has not started by then is the 504 page, and one that is still streaming is cut. The request body is taken in full before the answer, so a slow upload gets its 504 once the body is in.
@@ -103,9 +105,9 @@ A URI of 65,535 bytes or more gets an empty `414`, as on AWS. Headers or a reque
## Logs
-What an environment writes goes to the revision's `logGroup`, by default `/aws/lambda/web/<function>`. Each event goes to the stream that `lambda-web-logstream` names in the `<logGroup>.local` group, and to the same stream with a `-<account>-<region>` suffix in `<logGroup>`, where AWS centralizes the regions an endpoint serves. A group and a stream are created when their first event arrives, with no retention.
+What an environment writes goes to the revision's `logGroup`, by default `/aws/lambda/web/<function>`. Each event goes to the stream that `lambda-web-logstream` names in the `<logGroup>.local` group of the region that served, and to the same stream with a `-<account>-<region>` suffix in `<logGroup>` of the home region, where AWS centralizes the regions an endpoint serves. Floci writes both at once, where AWS copies the events of another region to the home region within minutes. A group and a stream are created when their first event arrives, with no retention.
-Console output becomes the JSON events AWS writes, with `timestamp`, `level`, `message`, `region`, and `requestId` and `endpointName` for output made while a request runs. `console.log` and `console.info` are `INFO`, `console.debug` is `DEBUG`, `console.trace` is `TRACE` with the stack, `console.warn` is `WARN` and `console.error` is `ERROR`. One argument that is not a string goes into `message` as JSON, and an `Error` as `errorType`, `errorMessage` and `stackTrace`. Several arguments make one string, formatted as Node prints them. Output written to stdout or stderr directly, and `console.dir`, stays as it is, one event per line. Events below the revision's `applicationLogLevel` are dropped, and direct output counts as `INFO`. `FATAL` drops all console output, as on AWS. Bytes that are not UTF-8 become U+FFFD. A line longer than 262,118 bytes in UTF-8 is cut before the character that would pass it, and a line of only spaces makes no event.
+Console output becomes the JSON events AWS writes, with `timestamp`, `level`, `message`, `region` (the region that served), and `requestId` and `endpointName` for output made while a request runs. `console.log` and `console.info` are `INFO`, `console.debug` is `DEBUG`, `console.trace` is `TRACE` with the stack, `console.warn` is `WARN` and `console.error` is `ERROR`. One argument that is not a string goes into `message` as JSON, and an `Error` as `errorType`, `errorMessage` and `stackTrace`. Several arguments make one string, formatted as Node prints them. Output written to stdout or stderr directly, and `console.dir`, stays as it is, one event per line. Events below the revision's `applicationLogLevel` are dropped, and direct output counts as `INFO`. `FATAL` drops all console output, as on AWS. Bytes that are not UTF-8 become U+FFFD. A line longer than 262,118 bytes in UTF-8 is cut before the character that would pass it, and a line of only spaces makes no event.
The output of an environment that fails before it listens is written too, as on AWS. A stack trace shows the frames of Floci's runtime where AWS shows its own.
@@ -126,7 +128,7 @@ CPU time and memory come from the container's cgroup, read at each report. Floci
## Metrics
-Floci publishes the [Web Function metrics](https://docs.aws.amazon.com/lambda/latest/dg/web-functions-monitoring-metrics.html) to CloudWatch in the function's account and region:
+Floci publishes the [Web Function metrics](https://docs.aws.amazon.com/lambda/latest/dg/web-functions-monitoring-metrics.html) to CloudWatch in the function's account, in the region that served. `AWS/Lambda/Web` also goes to the home region, where AWS centralizes it, and `AWS/Lambda/Web/Usage` does not, as on AWS:
| Namespace | Metrics | Dimensions | Datapoints |
| --- | --- | --- | --- |
@@ -136,7 +138,7 @@ Floci publishes the [Web Function metrics](https://docs.aws.amazon.com/lambda/la
| `AWS/Lambda/Web/Usage` | `ExecutedRequests`, `EndpointActiveEnvironments` | `FunctionName`, `EndpointName`, `ExecutionRegion` | One for each request that reached an environment. `EndpointActiveEnvironments` is the number of environments of the endpoint, those that are starting included |
| `AWS/Lambda/Web/Usage` | `EndpointRequestThrottles`, `EndpointActiveEnvironmentThrottles` | `FunctionName`, `EndpointName`, `ExecutionRegion` | 1 for each `429` of the `rateLimit` or of `maxEnvironments` |
-The account quotas are not applied, so `AccountRequestThrottles` and `AccountActiveVCPUArm64Throttles` never appear. CPU time and memory have the gaps of the platform events. The datapoints are stored on their own thread, and when 10,000 writes are waiting new ones are dropped with a warning in Floci's log.
+The account quotas are not applied, so `AccountRequestThrottles` and `AccountActiveVCPUArm64Throttles` never appear. AWS starts to centralize another region some minutes after the endpoint is `Active`, and the datapoints from before that never reach the home region. Floci centralizes every datapoint at once. CPU time and memory have the gaps of the platform events. The datapoints are stored on their own thread, and when 10,000 writes are waiting new ones are dropped with a warning in Floci's log.
## CloudFormation
diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlane.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlane.java
index 6d01ddab26..77ef506bc5 100644
--- a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlane.java
+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlane.java
@@ -228,9 +228,12 @@ public class LambdaWebDataPlane {
acquire(request, host, target, revision, requestId, null);
}
- /** The domain is minted for each creation, so a recreated endpoint shares no state with the deleted one. */
+ /**
+ * The domain is minted for each creation, so a recreated endpoint shares no state with the deleted one.
+ * Each region of an endpoint has its own environments and rate limit.
+ */
private static String endpointKey(WebFunctionTarget target) {
- return String.join("|", target.accountId(), target.region(), target.endpoint().domainName);
+ return String.join("|", target.accountId(), target.servingRegion(), target.endpoint().domainName);
}
/** Starts or reuses the revision's environment off the event loop, then forwards. */
@@ -242,7 +245,7 @@ public class LambdaWebDataPlane {
}
Integer maxEnvironments = target.endpoint().scalingConfig == null
? null : target.endpoint().scalingConfig.maxEnvironments;
- return environments.acquire(target.accountId(), target.region(), target.function(),
+ return environments.acquire(target.accountId(), target.servingRegion(), target.function(),
endpointKey(target), maxEnvironments, revision);
}, false)
.onSuccess(environment -> forward(request, host, target, revision, environment, requestId, dead == null))
diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java
index 198a175794..13a35306c6 100644
--- a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebEnvironmentManager.java
@@ -8,6 +8,7 @@ import com.github.dockerjava.api.exception.NotFoundException;
import com.github.dockerjava.api.model.Frame;
import com.github.dockerjava.api.model.StreamType;
import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.core.common.AwsArnUtils;
import io.github.hectorvent.floci.core.common.ContainerTeardown;
import io.github.hectorvent.floci.core.common.FileTrees;
import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;
@@ -313,8 +314,9 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
* for it. The least busy one is taken, the one idle longest on a tie, so requests rotate. When
* all are full a new one starts. At {@code maxEnvironments} an idle environment of another
* revision makes room, and without one the request is {@link Throttled}. {@code endpointKey}
- * names one creation of the endpoint in one region, so a recreated endpoint starts afresh. The
- * caller releases the environment when the request is over. Blocks on Docker and S3.
+ * names one creation of the endpoint in one region, so a recreated endpoint starts afresh.
+ * {@code region} is the region the environment serves. The caller releases the environment
+ * when the request is over. Blocks on Docker and S3.
*/
public Environment acquire(String accountId, String region, WebFunction fn, String endpointKey,
Integer maxEnvironments, WebFunctionRevision revision) {
@@ -547,7 +549,8 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
}
private void start(Environment environment, String region, WebFunction fn, WebFunctionRevision revision) {
- if (!executionRole.assumableFor(region, fn.functionArn, revision.serviceConfig.executionRoleArn)) {
+ String homeRegion = AwsArnUtils.parse(fn.functionArn).region();
+ if (!executionRole.assumableFor(homeRegion, fn.functionArn, revision.serviceConfig.executionRoleArn)) {
throw new StartFailure(LambdaWebErrorPages.CANNOT_ASSUME_ROLE);
}
// Before the session and the token, which a failed start must not leave behind.
@@ -594,17 +597,17 @@ public class LambdaWebEnvironmentManager implements ContainerTeardown {
}
LambdaWebPlatformLog platform = null;
try {
- copyCode(containerId, accountId, region, fn, revision);
+ copyCode(containerId, accountId, homeRegion, fn, revision);
RetryingTarCopier.copyBytes(lifecycleManager.getDockerClient(), containerId, RUNTIME_DIR, "index.js",
runtime, 0644);
environment.containerId = containerId;
environment.logStream = LOG_STREAM_DATE.format(Instant.now()) + "/" + fn.functionName + "["
+ revision.revisionId + "]" + UUID.randomUUID().toString().replace("-", "");
- environment.logs = logs.open(accountId, region, revision.serviceConfig.logGroup, environment.logStream,
- revision.serviceConfig.systemLogLevel);
+ environment.logs = logs.open(accountId, region, homeRegion, revision.serviceConfig.logGroup,
+ environment.logStream, revision.serviceConfig.systemLogLevel);
platform = new LambdaWebPlatformLog(environment.logs, region, fn.functionName, revision.revisionId,
revision.buildConfig.runtime, revision.serviceConfig.applicationLogLevel,
- metrics.environment(accountId, region, fn.functionName));
+ metrics.environment(accountId, region, homeRegion, fn.functionName));
environment.platformLog = platform;
ContainerInfo info = lifecycleManager.startCreated(containerId, spec);
platform.initStart();
diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebIamAuth.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebIamAuth.java
index 04e7bdd8a4..59f84286f7 100644
--- a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebIamAuth.java
+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebIamAuth.java
@@ -112,7 +112,7 @@ public class LambdaWebIamAuth {
if (distributionArn != null) {
return authorizeCloudFront(distributionArn, target);
}
- String accessKeyId = authenticate(request, target.region());
+ String accessKeyId = authenticate(request, target.servingRegion());
return accessKeyId != null && authorize(accessKeyId, target);
}
@@ -251,7 +251,7 @@ public class LambdaWebIamAuth {
if (callerAccount.isEmpty()) {
return false;
}
- boolean allowed = RequestScopes.callAs(callerAccount.get(), target.region(),
+ boolean allowed = RequestScopes.callAs(callerAccount.get(), target.servingRegion(),
() -> evaluate(accessKeyId, callerAccount.get(), target));
if (!allowed) {
LOG.infov("IAM enforcement DENY: akid={0} action={1} resource={2}",
@@ -283,7 +283,7 @@ public class LambdaWebIamAuth {
}
String endpointArn = target.endpoint().endpointArn;
Map<String, List<String>> effective = IamConditionContextResolver.withGlobalContext(
- context, endpointArn, target.region(), callerAccount, target.accountId());
+ context, endpointArn, target.servingRegion(), callerAccount, target.accountId());
String policy = target.function().policy;
ResourcePolicyDecision grant = evaluator.evaluateResourcePolicyFor(
isBlank(policy) ? null : List.of(policy), RequestPrincipal.caller(callerArns.orElse(null)),
diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogs.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogs.java
index caee8e6a74..03e80880fc 100644
--- a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogs.java
+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogs.java
@@ -24,10 +24,11 @@ import java.util.concurrent.atomic.AtomicInteger;
/**
* Delivers the log events of Web Function environments. Each event goes to the stream of its
- * environment in the {@code .local} companion of the revision's log group, and to the same stream
- * with an {@code -<account>-<region>} suffix in the log group itself, where AWS centralizes the
- * regions an endpoint serves. A group or stream is created when its first event arrives. The
- * {@code platform.*} events below the revision's {@code systemLogLevel} are dropped.
+ * environment in the {@code .local} companion of the revision's log group in the region the
+ * environment serves, and to the same stream with an {@code -<account>-<region>} suffix in the log
+ * group itself in the home region, where AWS centralizes the regions an endpoint serves. A group or
+ * stream is created when its first event arrives. The {@code platform.*} events below the
+ * revision's {@code systemLogLevel} are dropped.
*/
@ApplicationScoped
public class LambdaWebLogs {
@@ -50,10 +51,11 @@ public class LambdaWebLogs {
this.logs = logs;
}
- public Sink open(String accountId, String region, String logGroup, String logStream, String systemLogLevel) {
+ public Sink open(String accountId, String region, String homeRegion, String logGroup, String logStream,
+ String systemLogLevel) {
return new Sink(accountId, region, SYSTEM_LEVELS.indexOf(systemLogLevel), List.of(
- new Destination(logGroup + ".local", logStream),
- new Destination(logGroup, logStream + "-" + accountId + "-" + region)));
+ new Destination(region, logGroup + ".local", logStream),
+ new Destination(homeRegion, logGroup, logStream + "-" + accountId + "-" + region)));
}
/** A time as AWS writes it in a platform event, without the trailing zeros of the fraction. */
@@ -68,7 +70,7 @@ public class LambdaWebLogs {
writers.shutdownNow();
}
- private record Destination(String logGroup, String logStream) {
+ private record Destination(String region, String logGroup, String logStream) {
}
private record Event(long timestamp, String message) {
@@ -198,20 +200,22 @@ public class LambdaWebLogs {
/** The group and the stream are created when an event finds one of them missing, also after a delete. */
private void putTo(Destination destination, List<Map<String, Object>> batch) {
try {
- logs.putLogEventsForAccount(accountId, destination.logGroup(), destination.logStream(), batch, region);
+ logs.putLogEventsForAccount(accountId, destination.logGroup(), destination.logStream(), batch,
+ destination.region());
} catch (AwsException e) {
if (!"ResourceNotFoundException".equals(e.getErrorCode())) {
throw e;
}
createLogGroup(destination);
createLogStream(destination);
- logs.putLogEventsForAccount(accountId, destination.logGroup(), destination.logStream(), batch, region);
+ logs.putLogEventsForAccount(accountId, destination.logGroup(), destination.logStream(), batch,
+ destination.region());
}
}
private void createLogGroup(Destination destination) {
try {
- logs.createLogGroupForAccount(accountId, destination.logGroup(), null, null, region);
+ logs.createLogGroupForAccount(accountId, destination.logGroup(), null, null, destination.region());
} catch (AwsException e) {
ignoreAlreadyExists(e);
}
@@ -219,7 +223,8 @@ public class LambdaWebLogs {
private void createLogStream(Destination destination) {
try {
- logs.createLogStreamForAccount(accountId, destination.logGroup(), destination.logStream(), region);
+ logs.createLogStreamForAccount(accountId, destination.logGroup(), destination.logStream(),
+ destination.region());
} catch (AwsException e) {
ignoreAlreadyExists(e);
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebMetrics.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebMetrics.java
index 6dee2600be..b9fc9b2bbe 100644
--- a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebMetrics.java
+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebMetrics.java
@@ -26,8 +26,8 @@ import java.util.stream.Stream;
/**
* The CloudWatch metrics of Web Functions: one datapoint per request when it is answered, and one
- * per environment when it starts, each minute and when it ends. Floci serves an endpoint in its
- * home region only, so both namespaces go there.
+ * per environment when it starts, each minute and when it ends. Both namespaces go to the region
+ * that served, and {@code AWS/Lambda/Web} also to the home region, where AWS centralizes it.
*/
@ApplicationScoped
public class LambdaWebMetrics {
@@ -84,14 +84,14 @@ public class LambdaWebMetrics {
datapoints.add(new Datapoint(USAGE_NAMESPACE,
MetricDatum.of("EndpointActiveEnvironments", "Count", activeEnvironments, usage, now)));
statusClass(datapoints, status, request, now);
- publish(target.accountId(), target.region(), datapoints);
+ publish(target.accountId(), target.servingRegion(), target.region(), datapoints);
}
/** A request the platform answered without an environment. */
void answered(WebFunctionTarget target, int status) {
List<Datapoint> datapoints = new ArrayList<>();
statusClass(datapoints, status, requestDimensions(target), Instant.now().getEpochSecond());
- publish(target.accountId(), target.region(), datapoints);
+ publish(target.accountId(), target.servingRegion(), target.region(), datapoints);
}
/** A request refused with 429. {@code metric} names the limit it hit. */
@@ -101,14 +101,14 @@ public class LambdaWebMetrics {
datapoints.add(new Datapoint(USAGE_NAMESPACE,
MetricDatum.of(metric, "Count", 1, usageDimensions(target), now)));
statusClass(datapoints, 429, requestDimensions(target), now);
- publish(target.accountId(), target.region(), datapoints);
+ publish(target.accountId(), target.servingRegion(), target.region(), datapoints);
}
- EnvironmentMetrics environment(String accountId, String region, String functionName) {
+ EnvironmentMetrics environment(String accountId, String region, String homeRegion, String functionName) {
List<Dimension> dimensions = List.of(new Dimension("FunctionName", functionName));
return (activeMillis, cpuMillis, maxMemoryMb, concurrentRequests, responseBytes) -> {
long now = Instant.now().getEpochSecond();
- publish(accountId, region, Stream.of(
+ publish(accountId, region, homeRegion, Stream.of(
MetricDatum.of("EnvironmentActiveTime", "Milliseconds", activeMillis, dimensions, now),
MetricDatum.of("EnvironmentCPUTime", "Milliseconds", cpuMillis, dimensions, now),
MetricDatum.of("EnvironmentMaxMemoryUsage", "Megabytes", maxMemoryMb, dimensions, now),
@@ -157,10 +157,10 @@ public class LambdaWebMetrics {
private static List<Dimension> usageDimensions(WebFunctionTarget target) {
return List.of(new Dimension("FunctionName", target.function().functionName),
new Dimension("EndpointName", target.endpoint().endpointName),
- new Dimension("ExecutionRegion", target.region()));
+ new Dimension("ExecutionRegion", target.servingRegion()));
}
- private synchronized void publish(String accountId, String region, List<Datapoint> datapoints) {
+ private synchronized void publish(String accountId, String region, String homeRegion, List<Datapoint> datapoints) {
if (datapoints.isEmpty()) {
return;
}
@@ -168,6 +168,9 @@ public class LambdaWebMetrics {
lastWrite = publisher.submit(() -> {
for (Datapoint datapoint : datapoints) {
store(accountId, region, datapoint);
+ if (!region.equals(homeRegion) && NAMESPACE.equals(datapoint.namespace())) {
+ store(accountId, homeRegion, datapoint);
+ }
}
});
dropping = false;
diff --git a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebService.java b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebService.java
index c9dbeff56d..ea3cf68bb5 100644
--- a/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebService.java
@@ -191,11 +191,16 @@ public class LambdaWebService implements Resettable {
code.clear();
}
- /** Where a request to an endpoint domain goes. Domains are unique across accounts and regions. */
- public record WebFunctionTarget(String accountId, String region, WebFunction function, WebFunctionEndpoint endpoint) {
+ /**
+ * Where a request to an endpoint domain goes. Domains are unique across accounts and regions.
+ * {@code region} is the function's home region and {@code servingRegion} the region of the domain.
+ */
+ public record WebFunctionTarget(String accountId, String region, WebFunction function, WebFunctionEndpoint endpoint,
+ String servingRegion) {
}
- private record EndpointLocator(String accountId, String region, String functionName, String endpointName) {
+ private record EndpointLocator(String accountId, String region, String functionName, String endpointName,
+ String servingRegion) {
}
public Optional<WebFunctionTarget> locate(String host) {
@@ -213,7 +218,7 @@ public class LambdaWebService implements Resettable {
}
WebFunction fn = copy(stored.get(), WebFunction.class);
return Optional.of(new WebFunctionTarget(locator.accountId(), locator.region(), fn,
- fn.endpoints.get(locator.endpointName())));
+ fn.endpoints.get(locator.endpointName()), locator.servingRegion()));
}
});
}
@@ -226,11 +231,13 @@ public class LambdaWebService implements Resettable {
return Optional.ofNullable(originHosts.get(host));
}
+ /** Floci serves the MultiRegion global domain in the home region. */
private void indexDomains(WebFunction fn, WebFunctionEndpoint endpoint) {
AwsArnUtils.Arn arn = AwsArnUtils.parse(fn.functionArn);
- EndpointLocator locator = new EndpointLocator(arn.accountId(), arn.region(), fn.functionName, endpoint.endpointName);
- index(endpoint.domainName, locator);
- endpoint.regionalDomainNames.values().forEach(domain -> index(domain, locator));
+ index(endpoint.domainName, new EndpointLocator(arn.accountId(), arn.region(), fn.functionName,
+ endpoint.endpointName, arn.region()));
+ endpoint.regionalDomainNames.forEach((region, domain) -> index(domain, new EndpointLocator(arn.accountId(),
+ arn.region(), fn.functionName, endpoint.endpointName, region)));
}
private void index(String domain, EndpointLocator locator) {
@@ -924,8 +931,9 @@ public class LambdaWebService implements Resettable {
if ("MultiRegion".equals(endpoint.endpointType)) {
endpoint.domainName = randomLowercase(DOMAIN_ID_LENGTH) + ".lambda-web.global." + host;
} else {
+ String id = randomLowercase(DOMAIN_ID_LENGTH);
for (String each : endpoint.regions) {
- endpoint.regionalDomainNames.put(each, regionalDomain(each, host));
+ endpoint.regionalDomainNames.put(each, regionalDomain(id, each, host));
}
endpoint.domainName = endpoint.regionalDomainNames.get(region);
}
@@ -933,8 +941,8 @@ public class LambdaWebService implements Resettable {
endpoint.updatedAt = endpoint.createdAt;
}
- private static String regionalDomain(String region, String host) {
- return randomLowercase(DOMAIN_ID_LENGTH) + "." + DOMAIN_SHARD + ".lambda-web." + region + "." + host;
+ private static String regionalDomain(String id, String region, String host) {
+ return id + "." + DOMAIN_SHARD + ".lambda-web." + region + "." + host;
}
private static void requireModeAllowed(String mode, String endpointType) {
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java
index c4a41a3293..c3c42c2a09 100644
--- a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneDockerIntegrationTest.java
@@ -42,10 +42,12 @@ import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Predicate;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
+import static io.github.hectorvent.floci.testing.EnforcementFixtures.auth;
import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.contains;
import static org.hamcrest.Matchers.endsWith;
@@ -433,6 +435,75 @@ class LambdaWebDataPlaneDockerIntegrationTest {
assertTrue(answer.getString("token").matches("[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}"));
}
+ @Test
+ @Timeout(240)
+ void aPerRegionEndpointServesEachRegionFromItsOwnEnvironments() throws Exception {
+ createFunction(30, 1);
+ String endpoints = BASE + "/web-functions/" + functionName + "/endpoints";
+ String revisionId = given().when().get(endpoints + "/dev").then().statusCode(200)
+ .extract().path("revisionWeights[0].revisionId");
+ JsonPath created = given().contentType("application/json")
+ .body("""
+ {"endpointName": "per", "endpointType": "PerRegion", "regions": ["us-west-2"],
+ "authType": "ApplicationManaged", "scalingConfig": {"maxEnvironments": 2},
+ "revisionWeights": [{"revisionId": "%s", "weight": 100}]}""".formatted(revisionId))
+ .when().put(endpoints)
+ .then().statusCode(202)
+ .extract().jsonPath();
+ String east = created.getString("regionalEndpoints.us-east-1.domainName");
+ String west = created.getString("regionalEndpoints.us-west-2.domainName");
+
+ Response westEcho = given().header("Host", west).when().get("/");
+ westEcho.then().statusCode(200).body("env.region", equalTo("us-west-2"));
+ String westStream = westEcho.header("lambda-web-logstream");
+ given().header("Host", east).when().get("/").then().statusCode(200).body("env.region", equalTo("us-east-1"))
+ .header("lambda-web-logstream", not(equalTo(westStream)));
+ JsonPath zone = given().header("Host", west).when().get("/metadata").then().statusCode(200)
+ .extract().jsonPath();
+ assertEquals("{\"AvailabilityZoneID\":\"usw2-az1\"}\n", zone.getString("body"));
+
+ // maxEnvironments holds in each region on its own, as on AWS.
+ AtomicInteger next = new AtomicInteger();
+ CyclicBarrier together = new CyclicBarrier(4);
+ List<String> waited = parallel(4, () -> raw("GET /wait HTTP/1.1\r\nHost: "
+ + (next.getAndIncrement() % 2 == 0 ? east : west) + "\r\nConnection: close\r\n\r\n", together));
+ assertTrue(waited.stream().allMatch(response -> response.startsWith("HTTP/1.1 200 ")),
+ String.join("\n----\n", waited));
+
+ // The .local group is in the region that served and the centralized one in the home region. Each batch
+ // reaches the .local group first.
+ String group = "/aws/lambda/web/" + functionName;
+ Predicate<List<String>> started = events -> events.stream().anyMatch(event -> event.contains("\"starting\""));
+ assertTrue(started.test(awaitLogEvents(group, westStream + "-000000000000-us-west-2", started)));
+ List<String> local = logEvents("us-west-2", group + ".local", westStream);
+ JsonPath starting = JsonPath.from(local.stream().filter(event -> event.contains("\"starting\""))
+ .findFirst().orElseThrow());
+ assertEquals("us-west-2", starting.getString("region"));
+ JsonPath initStart = platformEvent(local, "platform.initStart");
+ assertEquals("us-west-2", initStart.getString("region"));
+ assertTrue(initStart.getString("record.runtimeVersionArn").startsWith(
+ "arn:aws:lambda:us-west-2::runtime/nodejs24.x/version/"), initStart.prettify());
+ assertEquals(List.of(), logEvents("us-east-1", group + ".local", westStream));
+
+ // AWS/Lambda/Web is also centralized in the home region. The usage metrics are not.
+ Map<String, String> endpoint = Map.of("FunctionName", functionName, "EndpointName", "per");
+ Map<String, String> westUsage = Map.of("FunctionName", functionName, "EndpointName", "per",
+ "ExecutionRegion", "us-west-2");
+ metrics.awaitPublished(Duration.ofSeconds(5));
+ assertEquals(4, LambdaWebMetricStatistics.of("us-west-2", LambdaWebMetrics.NAMESPACE, "ExecutedRequests",
+ endpoint, "Sum"));
+ assertEquals(7, LambdaWebMetricStatistics.of("us-east-1", LambdaWebMetrics.NAMESPACE, "ExecutedRequests",
+ endpoint, "Sum"));
+ assertEquals(4, LambdaWebMetricStatistics.of("us-west-2", LambdaWebMetrics.USAGE_NAMESPACE,
+ "ExecutedRequests", westUsage, "Sum"));
+ assertEquals(0, LambdaWebMetricStatistics.of("us-east-1", LambdaWebMetrics.USAGE_NAMESPACE,
+ "ExecutedRequests", westUsage, "Sum"));
+
+ given().when().delete(endpoints + "/per").then().statusCode(204);
+ given().header("Host", west).when().get("/").then().statusCode(404)
+ .header("lambda-web-error-type", "Platform.InvalidEndpoint");
+ }
+
@Test
@Timeout(180)
void anEnvironmentRunsTheCodeTheRevisionWasCreatedFrom() throws Exception {
@@ -786,9 +857,17 @@ class LambdaWebDataPlaneDockerIntegrationTest {
}
private static List<String> logEvents(String logGroup, String logStream) {
- List<String> messages = RestAssuredJsonUtils.awsAction("Logs_20140328", "GetLogEvents",
- "{\"logGroupName\": \"%s\", \"logStreamName\": \"%s\", \"startFromHead\": true}"
- .formatted(logGroup, logStream))
+ return logEvents("us-east-1", logGroup, logStream);
+ }
+
+ private static List<String> logEvents(String region, String logGroup, String logStream) {
+ List<String> messages = given()
+ .header("Authorization", auth("test", region, "logs"))
+ .header("X-Amz-Target", "Logs_20140328.GetLogEvents")
+ .contentType("application/x-amz-json-1.1")
+ .body("{\"logGroupName\": \"%s\", \"logStreamName\": \"%s\", \"startFromHead\": true}"
+ .formatted(logGroup, logStream))
+ .when().post("/")
.then().extract().jsonPath().getList("events.message", String.class);
return messages == null ? List.of() : messages;
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneIntegrationTest.java
index d75d2e31fc..fe7827dac7 100644
--- a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebDataPlaneIntegrationTest.java
@@ -301,6 +301,36 @@ class LambdaWebDataPlaneIntegrationTest {
assertTrue(raw("GET / HTTP/1.1\r\n" + headers + pad + "a\r\n\r\n").startsWith("HTTP/1.1 431 "));
}
+ @Test
+ void aPerRegionEndpointCountsAPageInTheRegionThatServedAndInTheHomeRegion() {
+ String functionName = "dp-regions-" + Long.toString(System.nanoTime(), 36);
+ createFunction(functionName);
+ String endpoints = BASE + "/web-functions/" + functionName + "/endpoints";
+ String revisionId = given().when().get(endpoints + "/dev").then().statusCode(200)
+ .extract().path("revisionWeights[0].revisionId");
+ String west = given().contentType("application/json")
+ .body("""
+ {"endpointName": "per", "endpointType": "PerRegion", "regions": ["us-west-2"],
+ "authType": "IamAuth", "revisionWeights": [{"revisionId": "%s", "weight": 100}]}"""
+ .formatted(revisionId))
+ .when().put(endpoints)
+ .then().statusCode(202)
+ .extract().path("regionalEndpoints.us-west-2.domainName");
+ given().header("Host", west).when().get("/").then().statusCode(403);
+
+ Map<String, String> endpoint = Map.of("FunctionName", functionName, "EndpointName", "per");
+ metrics.awaitPublished(Duration.ofSeconds(5));
+ assertEquals(1, LambdaWebMetricStatistics.of("us-west-2", LambdaWebMetrics.NAMESPACE,
+ "HTTPResponseStatusClass4XX", endpoint, "Sum"));
+ assertEquals(1, LambdaWebMetricStatistics.of("us-east-1", LambdaWebMetrics.NAMESPACE,
+ "HTTPResponseStatusClass4XX", endpoint, "Sum"));
+
+ given().when().delete(endpoints + "/per").then().statusCode(204);
+ given().header("Host", west).when().get("/")
+ .then().statusCode(404).header("lambda-web-error-type", "Platform.InvalidEndpoint");
+ given().when().delete(BASE + "/web-functions/" + functionName).then().statusCode(204);
+ }
+
@Test
void otherHostsStillReachTheControlPlane() {
given().when().get(BASE + "/web-account-settings").then().statusCode(200);
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebIamAuthTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebIamAuthTest.java
index 8913d0bee8..b83dfd9f51 100644
--- a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebIamAuthTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebIamAuthTest.java
@@ -40,6 +40,7 @@ class LambdaWebIamAuthTest {
private static final String ACCOUNT = "111111111111";
private static final String OTHER_ACCOUNT = "222222222222";
private static final String HOST = "abcdefghijklmnop.001.lambda-web.us-east-1.localhost:4566";
+ private static final String WEST_HOST = "abcdefghijklmnop.001.lambda-web.us-west-2.localhost:4566";
private static final String FUNCTION_ARN = "arn:aws:lambda:us-east-1:" + ACCOUNT + ":web-function/app";
private static final String ENDPOINT_ARN = FUNCTION_ARN + "/endpoint/prod";
private static final String USER_KEY = "AKIAUSERKEY000000001";
@@ -125,6 +126,12 @@ class LambdaWebIamAuthTest {
assertFalse(allows(signer().service("execute-api").sign("GET", HOST, "/")));
}
+ @Test
+ void aRegionalDomainOfAnotherRegionTakesOnlyASignatureForThatRegion() {
+ assertTrue(allows(signer().region("us-west-2").sign("GET", WEST_HOST, "/"), WEST_HOST, "us-west-2"));
+ assertFalse(allows(signer().sign("GET", WEST_HOST, "/"), WEST_HOST, "us-west-2"));
+ }
+
@Test
void theSigningTimeMustBeWithinFiveMinutes() {
Instant now = Instant.now();
@@ -201,6 +208,19 @@ class LambdaWebIamAuthTest {
",\"Condition\":{\"StringEquals\":{\"aws:ResourceTag/env\":\"prod\"}}")));
}
+ @Test
+ void underEnforcementTheRequestedRegionIsTheRegionOfTheDomain() {
+ enforceFor(ACCOUNT);
+ LambdaWebRequestSigner.Signed west = signer().credentials(USER_KEY, USER_SECRET, null).region("us-west-2")
+ .sign("GET", WEST_HOST, "/");
+ function.policy = denyRequestedRegion("us-west-2");
+ assertTrue(allowsUserWith(allow(ENDPOINT_ARN, "")));
+ assertFalse(allows(west, WEST_HOST, "us-west-2"));
+ function.policy = denyRequestedRegion("us-east-1");
+ assertFalse(allowsUserWith(allow(ENDPOINT_ARN, "")));
+ assertTrue(allows(west, WEST_HOST, "us-west-2"));
+ }
+
@Test
void underEnforcementTheResourcePolicyAllowsWithinTheAccountAndItsDenyWins() {
enforceFor(ACCOUNT);
@@ -263,7 +283,7 @@ class LambdaWebIamAuthTest {
private boolean allowsCloudFront(Map<String, String> signature) {
MultiMap headers = MultiMap.caseInsensitiveMultiMap().add("Host", HOST).addAll(signature);
return iamAuth.allows(new Inbound("POST", "/", null, headers, MultiMap.caseInsensitiveMultiMap()),
- new WebFunctionTarget(ACCOUNT, "us-east-1", function, endpoint));
+ new WebFunctionTarget(ACCOUNT, "us-east-1", function, endpoint, "us-east-1"));
}
private static String cloudFrontPolicy(String... statements) {
@@ -301,6 +321,13 @@ class LambdaWebIamAuthTest {
.formatted(effect, callerAccount, ENDPOINT_ARN);
}
+ private static String denyRequestedRegion(String region) {
+ return """
+ {"Version":"2012-10-17","Statement":[{"Effect":"Deny","Principal":"*",
+ "Action":"lambda:InvokeWebFunctionEndpoint","Resource":"%s",
+ "Condition":{"StringEquals":{"aws:RequestedRegion":"%s"}}}]}""".formatted(ENDPOINT_ARN, region);
+ }
+
private static String allow(String resource, String condition) {
return """
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"lambda:InvokeWebFunctionEndpoint",
@@ -312,11 +339,15 @@ class LambdaWebIamAuthTest {
}
private boolean allows(LambdaWebRequestSigner.Signed signed) {
+ return allows(signed, HOST, "us-east-1");
+ }
+
+ private boolean allows(LambdaWebRequestSigner.Signed signed, String host, String servingRegion) {
String pathAndQuery = signed.pathAndQuery();
int question = pathAndQuery.indexOf('?');
String path = question < 0 ? pathAndQuery : pathAndQuery.substring(0, question);
String query = question < 0 ? null : pathAndQuery.substring(question + 1);
- MultiMap headers = MultiMap.caseInsensitiveMultiMap().add("Host", HOST);
+ MultiMap headers = MultiMap.caseInsensitiveMultiMap().add("Host", host);
signed.headers().forEach(headers::add);
MultiMap params = MultiMap.caseInsensitiveMultiMap();
try {
@@ -325,6 +356,6 @@ class LambdaWebIamAuthTest {
// The data plane keeps going with no parameters when the query does not decode.
}
return iamAuth.allows(new Inbound(signed.method(), path, query, headers, params),
- new WebFunctionTarget(ACCOUNT, "us-east-1", function, endpoint));
+ new WebFunctionTarget(ACCOUNT, "us-east-1", function, endpoint, servingRegion));
}
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebIntegrationTest.java
index dfe20f59b9..1d4ee09a9b 100644
--- a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebIntegrationTest.java
@@ -364,6 +364,21 @@ class LambdaWebIntegrationTest {
.then().statusCode(404)
.header("X-Amzn-Errortype", "ResourceNotFoundException")
.body("message", equalTo("The endpoint global does not exist."));
+
+ // The regional domains of a PerRegion endpoint share one id, and the home one leads.
+ String home = given().contentType("application/json")
+ .body("{\"endpointName\": \"per\", \"endpointType\": \"PerRegion\", \"regions\": [\"us-west-2\"],"
+ + " \"authType\": \"ApplicationManaged\","
+ + " \"revisionWeights\": [{\"revisionId\": \"" + revision1 + "\", \"weight\": 100}]}")
+ .when().put(endpoints)
+ .then().statusCode(202)
+ .body("domainName", matchesPattern("[a-z0-9]{16}\\.001\\.lambda-web\\.us-east-1\\..+"))
+ .extract().path("domainName");
+ given().when().get(endpoints + "/per")
+ .then().statusCode(200)
+ .body("regionalEndpoints.us-east-1.domainName", equalTo(home))
+ .body("regionalEndpoints.us-west-2.domainName", equalTo(home.replace(".us-east-1.", ".us-west-2.")));
+ given().when().delete(endpoints + "/per").then().statusCode(204);
}
@Test
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogsTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogsTest.java
index b3950365d0..bdb7f5582a 100644
--- a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogsTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebLogsTest.java
@@ -1,5 +1,7 @@
package io.github.hectorvent.floci.services.lambdaweb;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
import io.github.hectorvent.floci.core.common.AwsException;
import io.github.hectorvent.floci.services.cloudwatch.logs.CloudWatchLogsService;
import org.junit.jupiter.api.AfterEach;
@@ -10,6 +12,7 @@ import org.mockito.ArgumentCaptor;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
+import java.time.Instant;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CountDownLatch;
@@ -58,7 +61,7 @@ class LambdaWebLogsTest {
doThrow(notFound()).doReturn("token").when(cloudWatchLogs)
.putLogEventsForAccount(eq(ACCOUNT), eq(GROUP), eq(CENTRAL_STREAM), anyList(), eq(REGION));
- logs.open(ACCOUNT, REGION, GROUP, STREAM, "INFO").accept("first\n");
+ logs.open(ACCOUNT, REGION, REGION, GROUP, STREAM, "INFO").accept("first\n");
ArgumentCaptor<List<Map<String, Object>>> local = ArgumentCaptor.forClass(List.class);
verify(cloudWatchLogs, timeout(5_000).times(2)).putLogEventsForAccount(eq(ACCOUNT), eq(GROUP + ".local"),
@@ -72,6 +75,24 @@ class LambdaWebLogsTest {
assertEquals("first\n", local.getValue().getFirst().get("message"));
}
+ @Test
+ @SuppressWarnings("unchecked")
+ void anEnvironmentInAnotherRegionWritesTheLocalGroupThereAndTheCentralizedOneInTheHomeRegion() {
+ String replica = "eu-central-1";
+ ObjectNode record = new ObjectMapper().createObjectNode().put("functionName", "web");
+
+ logs.open(ACCOUNT, replica, REGION, GROUP, STREAM, "INFO").platform(Instant.now(), "INFO",
+ "platform.initStart", record);
+
+ ArgumentCaptor<List<Map<String, Object>>> local = ArgumentCaptor.forClass(List.class);
+ verify(cloudWatchLogs, timeout(5_000)).putLogEventsForAccount(eq(ACCOUNT), eq(GROUP + ".local"), eq(STREAM),
+ local.capture(), eq(replica));
+ verify(cloudWatchLogs, timeout(5_000)).putLogEventsForAccount(eq(ACCOUNT), eq(GROUP),
+ eq(STREAM + "-" + ACCOUNT + "-" + replica), anyList(), eq(REGION));
+ String message = (String) local.getValue().getFirst().get("message");
+ assertTrue(message.contains("\"region\":\"eu-central-1\""), message);
+ }
+
@Test
void aGroupThatCannotBeCreatedTakesNoEventsAndTheOtherStillDoes() {
doThrow(notFound()).when(cloudWatchLogs)
@@ -79,7 +100,7 @@ class LambdaWebLogsTest {
doThrow(new AwsException("InvalidParameterException", "Invalid log group name.", 400)).when(cloudWatchLogs)
.createLogGroupForAccount(ACCOUNT, GROUP + ".local", null, null, REGION);
- logs.open(ACCOUNT, REGION, GROUP, STREAM, "INFO").accept("first\n");
+ logs.open(ACCOUNT, REGION, REGION, GROUP, STREAM, "INFO").accept("first\n");
verify(cloudWatchLogs, timeout(5_000)).putLogEventsForAccount(eq(ACCOUNT), eq(GROUP), eq(CENTRAL_STREAM),
anyList(), eq(REGION));
@@ -96,7 +117,7 @@ class LambdaWebLogsTest {
throw new IllegalStateException("the store failed");
}).doReturn("token").when(cloudWatchLogs)
.putLogEventsForAccount(eq(ACCOUNT), eq(GROUP + ".local"), eq(STREAM), anyList(), eq(REGION));
- LambdaWebLogs.Sink sink = logs.open(ACCOUNT, REGION, GROUP, STREAM, "INFO");
+ LambdaWebLogs.Sink sink = logs.open(ACCOUNT, REGION, REGION, GROUP, STREAM, "INFO");
sink.accept("first\n");
writing.await(5, TimeUnit.SECONDS);
@@ -118,7 +139,7 @@ class LambdaWebLogsTest {
return "token";
}).doReturn("token").when(cloudWatchLogs)
.putLogEventsForAccount(eq(ACCOUNT), eq(GROUP + ".local"), eq(STREAM), anyList(), eq(REGION));
- LambdaWebLogs.Sink sink = logs.open(ACCOUNT, REGION, GROUP, STREAM, "INFO");
+ LambdaWebLogs.Sink sink = logs.open(ACCOUNT, REGION, REGION, GROUP, STREAM, "INFO");
sink.accept("first\n");
writing.await(5, TimeUnit.SECONDS);
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebMetricStatistics.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebMetricStatistics.java
index 02ba0c4f60..a11d40a562 100644
--- a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebMetricStatistics.java
+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebMetricStatistics.java
@@ -8,6 +8,7 @@ import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
+import static io.github.hectorvent.floci.testing.EnforcementFixtures.auth;
import static io.restassured.RestAssured.given;
/** Reads Web Function metrics back with GetMetricStatistics, as a client does. */
@@ -18,12 +19,18 @@ final class LambdaWebMetricStatistics {
/** {@code Sum} and {@code SampleCount} add up over the periods, {@code Maximum} is the largest. */
static double of(String namespace, String metric, Map<String, String> dimensions, String statistic) {
+ return of("us-east-1", namespace, metric, dimensions, statistic);
+ }
+
+ static double of(String region, String namespace, String metric, Map<String, String> dimensions,
+ String statistic) {
RestAssuredJsonUtils.configureAwsContentTypes();
Instant now = Instant.now();
String filter = dimensions.entrySet().stream()
.map(each -> "{\"Name\":\"" + each.getKey() + "\",\"Value\":\"" + each.getValue() + "\"}")
.collect(Collectors.joining(","));
JsonPath result = given().contentType("application/x-amz-json-1.0")
+ .header("Authorization", auth("test", region, "monitoring"))
.header("X-Amz-Target", "GraniteServiceVersion20100801.GetMetricStatistics")
.body("{\"Namespace\":\"" + namespace + "\",\"MetricName\":\"" + metric + "\",\"Dimensions\":[" + filter
+ "],\"StartTime\":" + now.minusSeconds(600).getEpochSecond() + ",\"EndTime\":"
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebMetricsTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebMetricsTest.java
index 3944469aed..d01e715eba 100644
--- a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebMetricsTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebMetricsTest.java
@@ -49,7 +49,7 @@ class LambdaWebMetricsTest {
function.functionName = "web";
WebFunctionEndpoint endpoint = new WebFunctionEndpoint();
endpoint.endpointName = "dev";
- target = new WebFunctionTarget(ACCOUNT, REGION, function, endpoint);
+ target = new WebFunctionTarget(ACCOUNT, REGION, function, endpoint, REGION);
}
@AfterEach
@@ -99,7 +99,7 @@ class LambdaWebMetricsTest {
@Test
void anEnvironmentPublishesEachMetricByFunctionName() {
- metrics.environment(ACCOUNT, REGION, "web").publish(244, 178, 142, 2, 37);
+ metrics.environment(ACCOUNT, REGION, REGION, "web").publish(244, 178, 142, 2, 37);
assertEquals(List.of(
"AWS/Lambda/Web EnvironmentActiveTime FunctionName=web 244.0 Milliseconds",
@@ -110,6 +110,40 @@ class LambdaWebMetricsTest {
published(5));
}
+ @Test
+ void aRequestServedInAnotherRegionAlsoCountsInTheHomeRegionButItsUsageDoesNot() {
+ WebFunctionTarget replica = new WebFunctionTarget(ACCOUNT, REGION, target.function(), target.endpoint(),
+ "eu-central-1");
+ metrics.executed(replica, 404, 3);
+
+ assertEquals(List.of(
+ "AWS/Lambda/Web ExecutedRequests FunctionName=web,EndpointName=dev 1.0 Count",
+ "AWS/Lambda/Web HTTPResponseStatusClass4XX FunctionName=web,EndpointName=dev 1.0 Count",
+ "AWS/Lambda/Web/Usage EndpointActiveEnvironments"
+ + " FunctionName=web,EndpointName=dev,ExecutionRegion=eu-central-1 3.0 Count",
+ "AWS/Lambda/Web/Usage ExecutedRequests FunctionName=web,EndpointName=dev,ExecutionRegion=eu-central-1"
+ + " 1.0 Count"),
+ published("eu-central-1", 4));
+ assertEquals(List.of(
+ "AWS/Lambda/Web ExecutedRequests FunctionName=web,EndpointName=dev 1.0 Count",
+ "AWS/Lambda/Web HTTPResponseStatusClass4XX FunctionName=web,EndpointName=dev 1.0 Count"),
+ published(REGION, 2));
+ }
+
+ @Test
+ void anEnvironmentInAnotherRegionPublishesThereAndInTheHomeRegion() {
+ metrics.environment(ACCOUNT, "eu-central-1", REGION, "web").publish(244, 178, 142, 2, 37);
+
+ List<String> expected = List.of(
+ "AWS/Lambda/Web EnvironmentActiveTime FunctionName=web 244.0 Milliseconds",
+ "AWS/Lambda/Web EnvironmentCPUTime FunctionName=web 178.0 Milliseconds",
+ "AWS/Lambda/Web EnvironmentConcurrentRequests FunctionName=web 2.0 Count",
+ "AWS/Lambda/Web EnvironmentMaxMemoryUsage FunctionName=web 142.0 Megabytes",
+ "AWS/Lambda/Web ResponseBytes FunctionName=web 37.0 Bytes");
+ assertEquals(expected, published("eu-central-1", 5));
+ assertEquals(expected, published(REGION, 5));
+ }
+
@Test
void writesPastTheQueueLimitAreDroppedAndTheRestAreStoredBeforeTheWaitEnds() throws InterruptedException {
metrics.shutdown();
@@ -137,11 +171,15 @@ class LambdaWebMetricsTest {
}
private List<String> published(int count) {
+ return published(REGION, count);
+ }
+
+ private List<String> published(String region, int count) {
ArgumentCaptor<String> namespaces = ArgumentCaptor.forClass(String.class);
ArgumentCaptor<MetricDatum> datums = ArgumentCaptor.forClass(MetricDatum.class);
metrics.awaitPublished(Duration.ofSeconds(5));
verify(cloudWatch, times(count)).publishMetricForAccount(eq(ACCOUNT), namespaces.capture(),
- datums.capture(), eq(REGION), anyString());
+ datums.capture(), eq(region), anyString());
List<String> published = new ArrayList<>();
for (int i = 0; i < count; i++) {
MetricDatum datum = datums.getAllValues().get(i);
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebPlatformLogTest.java b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebPlatformLogTest.java
index cda7bc141e..f86f64c23d 100644
--- a/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebPlatformLogTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/lambdaweb/LambdaWebPlatformLogTest.java
@@ -55,8 +55,8 @@ class LambdaWebPlatformLogTest {
@Test
void aRequestIsReportedWithItsDurationInMillisecondsToThreeDecimals() {
LambdaWebPlatformLog platform = new LambdaWebPlatformLog(
- logs.open("111122223333", "eu-west-1", GROUP, STREAM, "DEBUG"), "eu-west-1", "web", "rev-abcde",
- "nodejs24.x", "INFO", this::datapoint);
+ logs.open("111122223333", "eu-west-1", "eu-west-1", GROUP, STREAM, "DEBUG"), "eu-west-1", "web",
+ "rev-abcde", "nodejs24.x", "INFO", this::datapoint);
platform.requestComplete("KXH25FBDZ33GP4TQ4ARAUYSTSA", "dev", 504, "Runtime.RequestTimeout", 3_000_716_400L, 0);
@@ -71,8 +71,8 @@ class LambdaWebPlatformLogTest {
@Test
void eventsBelowTheSystemLogLevelAreDroppedAndAFailedReportIsWarn() {
LambdaWebPlatformLog platform = new LambdaWebPlatformLog(
- logs.open("111122223333", "eu-west-1", GROUP, STREAM, "WARN"), "eu-west-1", "web", "rev-abcde",
- "nodejs24.x", "INFO", this::datapoint);
+ logs.open("111122223333", "eu-west-1", "eu-west-1", GROUP, STREAM, "WARN"), "eu-west-1", "web",
+ "rev-abcde", "nodejs24.x", "INFO", this::datapoint);
platform.initStart();
platform.initReport(null, new LambdaWebPlatformLog.Usage(178_400, 142L * 1024 * 1024 + 5));
platform.invokeStart(1);
@@ -92,8 +92,8 @@ class LambdaWebPlatformLogTest {
@Test
void theListenFailureIsDroppedAtFatal() {
LambdaWebPlatformLog platform = new LambdaWebPlatformLog(
- logs.open("111122223333", "eu-west-1", GROUP, STREAM, "DEBUG"), "eu-west-1", "web", "rev-abcde",
- "nodejs24.x", "FATAL", this::datapoint);
+ logs.open("111122223333", "eu-west-1", "eu-west-1", GROUP, STREAM, "DEBUG"), "eu-west-1", "web",
+ "rev-abcde", "nodejs24.x", "FATAL", this::datapoint);
platform.applicationError("Application.TimeoutListenError: nothing was listening");
platform.spinDownStart();
@@ -104,8 +104,8 @@ class LambdaWebPlatformLogTest {
@Test
void anEnvironmentPublishesItsMetricsAtInitAndForEachWindowWhateverTheSystemLogLevel() {
LambdaWebPlatformLog platform = new LambdaWebPlatformLog(
- logs.open("111122223333", "eu-west-1", GROUP, STREAM, "WARN"), "eu-west-1", "web", "rev-abcde",
- "nodejs24.x", "INFO", this::datapoint);
+ logs.open("111122223333", "eu-west-1", "eu-west-1", GROUP, STREAM, "WARN"), "eu-west-1", "web",
+ "rev-abcde", "nodejs24.x", "INFO", this::datapoint);
platform.initStart();
platform.initReport(null, new LambdaWebPlatformLog.Usage(178_400, 142L * 1024 * 1024 + 5));
platform.invokeStart(1);
@@ -125,8 +125,8 @@ class LambdaWebPlatformLogTest {
@Test
void aFailedInitPublishesOneDatapoint() {
LambdaWebPlatformLog platform = new LambdaWebPlatformLog(
- logs.open("111122223333", "eu-west-1", GROUP, STREAM, "INFO"), "eu-west-1", "web", "rev-abcde",
- "nodejs24.x", "INFO", this::datapoint);
+ logs.open("111122223333", "eu-west-1", "eu-west-1", GROUP, STREAM, "INFO"), "eu-west-1", "web",
+ "rev-abcde", "nodejs24.x", "INFO", this::datapoint);
platform.initStart();
platform.initReport("Runtime.ExitError", null);