Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

fix(partition): read strict-mode service availability from endpoint rulesets too (#5231)

Hector Ventura8a1d1ddcd394
diff --git a/docs/configuration/environment-variables.md b/docs/configuration/environment-variables.md
index d3d99adc4a..6c04a4d8b3 100644
--- a/docs/configuration/environment-variables.md
+++ b/docs/configuration/environment-variables.md
@@ -12,7 +12,7 @@ Floci is configured exclusively through environment variables. Every option belo
| `FLOCI_HOSTNAME` | _(none)_ | Overrides only the hostname part of `FLOCI_BASE_URL`. Set to the Compose service name (e.g. `floci`) so other containers can reach Floci by DNS |
| `FLOCI_DEFAULT_REGION` | `us-east-1` | AWS region used in ARNs and API responses when a request names none. Also selects the deployment's partition (`cn-north-1` means `aws-cn`), see [AWS Partitions](./partitions.md) |
| `FLOCI_PARTITIONS_ID` | _(derived)_ | Pins the partition (`aws`, `aws-cn`, `aws-us-gov`, `aws-iso`, `aws-iso-b`, `aws-iso-e`, `aws-iso-f`, `aws-eusc`). Startup refuses a value that contradicts `FLOCI_DEFAULT_REGION` |
-| `FLOCI_PARTITIONS_STRICT` | `false` | Reject requests signed for a service AWS does not publish in the request's partition (CloudFront in `aws-us-gov`, IAM in `aws-eusc`) with a 404 `UnknownOperationException`, mirroring the host that fails to resolve on AWS. See [AWS Partitions](./partitions.md#partition-absent-services) |
+| `FLOCI_PARTITIONS_STRICT` | `false` | Reject requests signed for a service AWS does not publish in the request's partition (CloudFront in `aws-us-gov`, Lightsail in `aws-cn`) with a 404 `UnknownOperationException`, mirroring the host that fails to resolve on AWS. See [AWS Partitions](./partitions.md#partition-absent-services) |
| `FLOCI_PARTITIONS_ALLOW_UNKNOWN_REGIONS` | `false` | Accept a request signed for a region no AWS partition publishes or admits by its region pattern (`polygondwanaland-west-1`), giving the label its own namespace. Off, such a request is refused with a 400, as moto and LocalStack refuse it. See [AWS Partitions](./partitions.md#which-partition-a-request-belongs-to) |
| `FLOCI_DEFAULT_ACCOUNT_ID` | `000000000000` | Fallback account ID used in ARNs when the request's access key is not exactly 12 digits. When the access key IS 12 digits, it is used directly as the account ID — see [Multi-Account Isolation](./multi-account.md) |
| `FLOCI_DEFAULT_AVAILABILITY_ZONE` | `us-east-1a` | Availability zone reported in EC2 and other responses |
diff --git a/docs/configuration/partitions.md b/docs/configuration/partitions.md
index b6c31db358..e2f4312812 100644
--- a/docs/configuration/partitions.md
+++ b/docs/configuration/partitions.md
@@ -127,8 +127,8 @@ back to `aws` for a label no partition publishes.
## Partition-absent services
-AWS publishes which services exist in each partition (CloudFront is not in GovCloud, IAM is
-not in `aws-eusc`). On AWS a request for an absent service never reaches an API: the SDK
+AWS publishes which services exist in each partition (CloudFront is not in GovCloud, Lightsail
+is not in China). On AWS a request for an absent service never reaches an API: the SDK
fails to resolve the host and reports an `UnknownHostException`. Floci serves every enabled
service in every partition by default.
@@ -142,6 +142,14 @@ while a GovCloud CloudFront client is refused. Only a service the data lists in
partition is refused: `endpoints.json` omits the newer services that ship an endpoint ruleset
alone (FIS, MWAA, S3 Tables), and those are served everywhere.
+A partition also offers a service when that service's endpoint ruleset names the partition in a
+branch with its own endpoint, even where `endpoints.json` leaves it out: IAM, Route 53, Budgets and
+Cost Explorer in `aws-eusc` (`iam.eusc-de-east-1.amazonaws.eu`), IAM and Cost Explorer in
+`aws-iso-e`. The vendored data records these under `servicesFromRulesets`. The `execute-api`
+signing name is never refused: API Gateway's invoke endpoints and its WebSocket management API
+(`apigatewaymanagementapi`) sign with it as well as Connect Participant, and `endpoints.json` lists
+the API Gateway side nowhere, so the data cannot say where the name is served.
+
STS is regionalized everywhere and its global host `sts.amazonaws.com` exists only in `aws`, so
IAM's `GetAccountSummary` reports `GlobalEndpointTokenVersion` only there.
diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
index 1f5245f6d0..77cc8f237c 100644
--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
@@ -77,7 +77,7 @@ public interface EmulatorConfig {
/**
* Refuse requests signed for a service AWS does not publish in the request's partition
- * (CloudFront in GovCloud, IAM in {@code aws-eusc}). On AWS such a request never reaches
+ * (CloudFront in GovCloud, Lightsail in China). On AWS such a request never reaches
* an API because its endpoint does not resolve; Floci serves every enabled service in
* every partition unless this is set.
*/
diff --git a/src/main/java/io/github/hectorvent/floci/core/common/AccountContextFilter.java b/src/main/java/io/github/hectorvent/floci/core/common/AccountContextFilter.java
index 25c0267b44..7a4c11f3e5 100644
--- a/src/main/java/io/github/hectorvent/floci/core/common/AccountContextFilter.java
+++ b/src/main/java/io/github/hectorvent/floci/core/common/AccountContextFilter.java
@@ -169,7 +169,7 @@ public class AccountContextFilter implements ContainerRequestFilter {
/**
* Strict partition mode. AWS publishes no endpoint for a service outside the partitions it
- * exists in (CloudFront in GovCloud, IAM in {@code aws-eusc}); a client there never reaches
+ * exists in (CloudFront in GovCloud, Lightsail in China); a client there never reaches
* an API because the host does not resolve, which the SDKs surface as an
* {@code UnknownHostException}. Floci cannot fail DNS, so it answers the request with the
* same 404 shape the unknown-service-scope guard uses. Only a service the published data
diff --git a/src/main/java/io/github/hectorvent/floci/core/common/AwsPartition.java b/src/main/java/io/github/hectorvent/floci/core/common/AwsPartition.java
index 5046a43caf..333f58ea4c 100644
--- a/src/main/java/io/github/hectorvent/floci/core/common/AwsPartition.java
+++ b/src/main/java/io/github/hectorvent/floci/core/common/AwsPartition.java
@@ -24,7 +24,8 @@ import java.util.regex.Pattern;
* @param globalPseudoRegion the {@code <partition>-global} label the SDKs accept as a region,
* or {@code null} where the partition has none ({@code aws-eusc})
* @param regions the published regions, in botocore order
- * @param services every service key {@code endpoints.json} lists for the partition
+ * @param services every service key {@code endpoints.json} lists for the partition,
+ * plus those whose endpoint ruleset names it explicitly
* @param globalServices services with a partition-wide endpoint, by service key
* @param supportsDualStack whether the partition publishes dual-stack endpoints
* @param supportsFips whether the partition publishes FIPS endpoints
@@ -86,7 +87,7 @@ public record AwsPartition(
return region != null && s3DualStackRegions.contains(region.trim().toLowerCase(Locale.ROOT));
}
- /** True when {@code endpoints.json} lists {@code service} for this partition. */
+ /** True when the published data lists {@code service} for this partition (see {@link #services()}). */
public boolean offers(String service) {
return service != null && services.contains(service);
}
diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml
index 91f5ae168f..a8efe279e4 100644
--- a/src/main/resources/application.yml
+++ b/src/main/resources/application.yml
@@ -147,7 +147,7 @@ floci:
# id: aws # Partition served (aws, aws-cn, aws-us-gov, aws-iso, aws-iso-b, aws-iso-e, aws-iso-f, aws-eusc).
# Derived from default-region when unset. Env: FLOCI_PARTITIONS_ID
strict: false # true rejects requests signed for a service AWS does not publish in the request's
- # partition (CloudFront in aws-us-gov, IAM in aws-eusc). Env: FLOCI_PARTITIONS_STRICT
+ # partition (CloudFront in aws-us-gov, Lightsail in aws-cn). Env: FLOCI_PARTITIONS_STRICT
allow-unknown-regions: false # true accepts a SigV4 scope region no partition publishes or admits by its
# region pattern, giving it its own namespace. Env: FLOCI_PARTITIONS_ALLOW_UNKNOWN_REGIONS
# Shared mock-response config for Textract/Comprehend/Rekognition/Translate. See AiMockConfigLoader.
diff --git a/src/main/resources/aws/partitions.json b/src/main/resources/aws/partitions.json
index e6279b2b6f..21a0060db7 100644
--- a/src/main/resources/aws/partitions.json
+++ b/src/main/resources/aws/partitions.json
@@ -10,9 +10,6 @@
"IoTSecuredTunneling": [
"api.tunneling.iot"
],
- "agent-registry": [
- "agent-registry-control"
- ],
"appconfig": [
"appconfigdata"
],
@@ -38,9 +35,6 @@
"bedrock-data-automation-runtime",
"bedrock-runtime"
],
- "bedrock-agentcore": [
- "bedrock-agentcore-control"
- ],
"cassandra": [
"cassandra-streams"
],
@@ -72,9 +66,6 @@
"ecr-public": [
"api.ecr-public"
],
- "execute-api": [
- "participant.connect"
- ],
"forecast": [
"forecastquery"
],
@@ -699,6 +690,7 @@
"workspaces-web",
"xray"
],
+ "servicesFromRulesets": [],
"globalServices": {
"account": {
"hostname": "account.us-east-1.amazonaws.com",
@@ -1028,6 +1020,7 @@
"workspaces",
"xray"
],
+ "servicesFromRulesets": [],
"globalServices": {
"account": {
"hostname": "account.cn-northwest-1.amazonaws.com.cn",
@@ -1116,6 +1109,8 @@
"backup",
"batch",
"bedrock",
+ "budgets",
+ "ce",
"cloudcontrolapi",
"cloudformation",
"cloudtrail",
@@ -1154,6 +1149,7 @@
"glue",
"guardduty",
"health",
+ "iam",
"identitystore",
"internetmonitor",
"kafka",
@@ -1184,6 +1180,7 @@
"redshift-serverless",
"resource-groups",
"rolesanywhere",
+ "route53",
"route53profiles",
"route53resolver",
"rum",
@@ -1213,6 +1210,12 @@
"wafv2",
"xray"
],
+ "servicesFromRulesets": [
+ "budgets",
+ "ce",
+ "iam",
+ "route53"
+ ],
"globalServices": {},
"s3DualStackRegions": []
},
@@ -1348,6 +1351,7 @@
"workspaces",
"xray"
],
+ "servicesFromRulesets": [],
"globalServices": {
"budgets": {
"hostname": "budgets.c2s.ic.gov",
@@ -1509,6 +1513,7 @@
"workspaces",
"xray"
],
+ "servicesFromRulesets": [],
"globalServices": {
"budgets": {
"hostname": "budgets.global.sc2s.sgov.gov",
@@ -1579,6 +1584,7 @@
"autoscaling",
"batch",
"budgets",
+ "ce",
"cloudcontrolapi",
"cloudformation",
"cloudtrail",
@@ -1606,6 +1612,7 @@
"firehose",
"glue",
"health",
+ "iam",
"kinesis",
"kms",
"lakeformation",
@@ -1649,6 +1656,10 @@
"trustedadvisor",
"xray"
],
+ "servicesFromRulesets": [
+ "ce",
+ "iam"
+ ],
"globalServices": {
"budgets": {
"hostname": "budgets.global.cloud.adc-e.uk",
@@ -1798,6 +1809,7 @@
"trustedadvisor",
"xray"
],
+ "servicesFromRulesets": [],
"globalServices": {
"budgets": {
"hostname": "budgets.global.csp.hci.ic.gov",
@@ -1985,6 +1997,7 @@
"osis",
"outposts",
"participant.connect",
+ "partnercentral-channel",
"pi",
"pinpoint",
"polly",
@@ -2049,6 +2062,9 @@
"workspaces",
"xray"
],
+ "servicesFromRulesets": [
+ "partnercentral-channel"
+ ],
"globalServices": {
"iam": {
"hostname": "iam.us-gov.amazonaws.com",
diff --git a/src/test/java/io/github/hectorvent/floci/core/common/AccountContextFilterTest.java b/src/test/java/io/github/hectorvent/floci/core/common/AccountContextFilterTest.java
index fee22db36a..b45a7f5e70 100644
--- a/src/test/java/io/github/hectorvent/floci/core/common/AccountContextFilterTest.java
+++ b/src/test/java/io/github/hectorvent/floci/core/common/AccountContextFilterTest.java
@@ -249,7 +249,7 @@ class AccountContextFilterTest {
@Test
void strictModeAlsoCoversPresignedCredentials() {
strictPartitions = true;
- ContainerRequestContext ctx = mockContext(null, "AKID/20260617/eusc-de-east-1/iam/aws4_request");
+ ContainerRequestContext ctx = mockContext(null, "AKID/20260617/cn-north-1/lightsail/aws4_request");
filter.filter(ctx);
ArgumentCaptor<Response> aborted = ArgumentCaptor.forClass(Response.class);
verify(ctx).abortWith(aborted.capture());
diff --git a/src/test/java/io/github/hectorvent/floci/core/common/AwsPartitionsTest.java b/src/test/java/io/github/hectorvent/floci/core/common/AwsPartitionsTest.java
index 3a0961b285..3c3dd7c581 100644
--- a/src/test/java/io/github/hectorvent/floci/core/common/AwsPartitionsTest.java
+++ b/src/test/java/io/github/hectorvent/floci/core/common/AwsPartitionsTest.java
@@ -134,12 +134,12 @@ class AwsPartitionsTest {
}
@Test
- void serviceAvailabilityFollowsEndpointsJson() {
+ void serviceAvailabilityFollowsThePublishedData() {
assertTrue(AwsPartitions.commercial().offers("cloudfront"));
assertTrue(AwsPartitions.byId("aws-cn").offers("cloudfront"));
assertFalse(AwsPartitions.byId("aws-us-gov").offers("cloudfront"));
- assertFalse(AwsPartitions.byId("aws-eusc").offers("iam"));
- assertFalse(AwsPartitions.byId("aws-iso-e").offers("iam"));
+ assertFalse(AwsPartitions.byId("aws-cn").offers("lightsail"));
+ assertTrue(AwsPartitions.byId("aws-eusc").offers("iam"), "IAM's endpoint ruleset names aws-eusc");
assertFalse(AwsPartitions.byId("aws-cn").offers("cognito-idp"));
assertFalse(AwsPartitions.commercial().offers(null));
}
@@ -156,7 +156,7 @@ class AwsPartitionsTest {
assertTrue(AwsPartitions.byId("aws-cn").offersSigningName("ecr"));
assertTrue(AwsPartitions.byId("aws-cn").offersSigningName("sqs"));
assertFalse(AwsPartitions.byId("aws-us-gov").offersSigningName("cloudfront"));
- assertFalse(AwsPartitions.byId("aws-eusc").offersSigningName("iam"));
+ assertFalse(AwsPartitions.byId("aws-cn").offersSigningName("lightsail"));
assertFalse(AwsPartitions.commercial().offersSigningName("no-such-service"));
assertFalse(AwsPartitions.commercial().offersSigningName(null));
@@ -167,6 +167,35 @@ class AwsPartitionsTest {
assertFalse(AwsPartitions.publishesSomewhere(null));
}
+ /**
+ * {@code endpoints.json} leaves IAM, Route 53, Budgets and Cost Explorer out of {@code aws-eusc},
+ * and IAM and Cost Explorer out of {@code aws-iso-e}, but each service's endpoint ruleset has an
+ * explicit branch for the partition ({@code iam.eusc-de-east-1.amazonaws.eu}).
+ */
+ @ParameterizedTest
+ @CsvSource({
+ "aws-eusc, iam",
+ "aws-eusc, route53",
+ "aws-eusc, budgets",
+ "aws-eusc, ce",
+ "aws-iso-e, iam",
+ "aws-iso-e, ce",
+ })
+ void partitionsAServiceRulesetNamesExplicitlyOfferIt(String partitionId, String signingName) {
+ assertTrue(AwsPartitions.byId(partitionId).offersSigningName(signingName));
+ }
+
+ /**
+ * API Gateway's invoke endpoints and its WebSocket management API sign {@code execute-api},
+ * as Connect Participant does, and the published data lists the API Gateway side nowhere, so
+ * the name must not be judged by where Connect Participant is offered.
+ */
+ @Test
+ void executeApiIsNotTiedToConnectParticipant() {
+ assertEquals(Set.of(), AwsPartitions.endpointPrefixes("execute-api"));
+ assertFalse(AwsPartitions.publishesSomewhere("execute-api"));
+ }
+
@Test
void optInAndWebsiteFormFlagsComeFromTheCdkRules() {
AwsPartition aws = AwsPartitions.commercial();
diff --git a/src/test/java/io/github/hectorvent/floci/core/common/PartitionStrictModeIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/core/common/PartitionStrictModeIntegrationTest.java
index cbe8899709..f4c964205c 100644
--- a/src/test/java/io/github/hectorvent/floci/core/common/PartitionStrictModeIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/core/common/PartitionStrictModeIntegrationTest.java
@@ -15,6 +15,7 @@ import java.util.TreeSet;
import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.not;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
@@ -44,17 +45,50 @@ class PartitionStrictModeIntegrationTest {
.body("message", containsString("cloudfront has no endpoint in partition aws-us-gov"));
}
- /** IAM speaks the Query protocol, so the refusal is the Query XML error a Query client parses. */
+ /** Elastic Beanstalk speaks the Query protocol, so the refusal is the Query XML error a Query client parses. */
@Test
- void iamIsRefusedInEuscWithAQueryError() {
+ void elasticBeanstalkIsRefusedInEuscWithAQueryError() {
given()
- .header("Authorization", PartitionMatrix.sigV4Auth("eusc-de-east-1", "iam"))
- .formParam("Action", "ListRoles")
- .formParam("Version", "2010-05-08")
+ .header("Authorization", PartitionMatrix.sigV4Auth("eusc-de-east-1", "elasticbeanstalk"))
+ .formParam("Action", "DescribeApplications")
+ .formParam("Version", "2010-12-01")
.when().post("/").then().statusCode(404)
.contentType(containsString("xml"))
.body("ErrorResponse.Error.Code", equalTo("UnknownOperationException"))
- .body("ErrorResponse.Error.Message", containsString("iam has no endpoint in partition aws-eusc"));
+ .body("ErrorResponse.Error.Message",
+ containsString("elasticbeanstalk has no endpoint in partition aws-eusc"));
+ }
+
+ /**
+ * {@code endpoints.json} leaves IAM out of {@code aws-eusc} and {@code aws-iso-e}, but IAM's
+ * endpoint ruleset publishes {@code iam.eusc-de-east-1.amazonaws.eu} and
+ * {@code iam.eu-isoe-west-1.cloud.adc-e.uk}, so a client there reaches IAM.
+ */
+ @Test
+ void iamIsServedInTheEuscAndIsoEPartitionsItsRulesetNames() {
+ for (String region : new String[] {"eusc-de-east-1", "eu-isoe-west-1"}) {
+ given()
+ .header("Authorization", PartitionMatrix.sigV4Auth(region, "iam"))
+ .formParam("Action", "ListRoles")
+ .formParam("Version", "2010-05-08")
+ .when().post("/").then().statusCode(200);
+ }
+ }
+
+ /**
+ * An IAM-authorized API Gateway invoke signs {@code execute-api}, a name Connect Participant
+ * shares. China offers API Gateway but not Connect Participant, so the invoke must reach API
+ * Gateway, which answers for the missing API itself.
+ */
+ @Test
+ void executeApiInvokesAreNotRefusedWhereConnectParticipantIsAbsent() {
+ given()
+ .header("Authorization", PartitionMatrix.sigV4Auth("cn-north-1", "execute-api"))
+ .when().get("/execute-api/nosuchapi/prod/items").then()
+ .statusCode(404)
+ .header("X-Amzn-Errortype", not(equalTo("UnknownOperationException")))
+ .body("message", equalTo("Invalid API id specified"))
+ .body(not(containsString("has no endpoint in partition")));
}
@Test
diff --git a/tools/aws/regen_partitions.py b/tools/aws/regen_partitions.py
index 11e936ea8a..112e4fa7be 100644
--- a/tools/aws/regen_partitions.py
+++ b/tools/aws/regen_partitions.py
@@ -19,6 +19,15 @@ from two sources and a rule:
differs from their `endpoints.json` key (`ecr` signs what `api.ecr` publishes, `bedrock`
covers `bedrock-runtime`). `signingNames` maps each such signing name to its endpoint
prefixes so a request's credential scope can be checked against a partition's service list.
+ A signing name is left out when a model whose own endpoint prefix is that signing name is
+ listed in no partition (`execute-api` is Connect Participant's signing name, but also the
+ endpoint prefix of API Gateway's WebSocket management API, which `endpoints.json` lists
+ nowhere): the data cannot say where such a name is served.
+- botocore's per-service `endpoint-rule-set-1.json`, for the partitions a ruleset names in an
+ explicit branch that resolves to an endpoint (IAM's `iam.eusc-de-east-1.amazonaws.eu`).
+ `endpoints.json` omits some of those, so each partition's `services` adds them, recorded again
+ under `servicesFromRulesets`. Only a service `endpoints.json` lists in some partition is added:
+ a service it lists nowhere stays unknown everywhere rather than present in a few partitions.
- The AWS CDK's `region-info/lib/aws-entities.ts` (Apache-2.0), whose ordered region list
carries two rule markers: regions before `RULE_S3_WEBSITE_REGIONAL_SUBDOMAIN` use the
legacy `s3-website-<region>` endpoint form, and commercial regions after
@@ -98,6 +107,14 @@ def service_model(version_dir: Path) -> Path | None:
return None
+def ruleset_file(version_dir: Path) -> Path | None:
+ """A version directory's `endpoint-rule-set-1.json`, plain or gzipped, or None."""
+ for name in ("endpoint-rule-set-1.json", "endpoint-rule-set-1.json.gz"):
+ if (version_dir / name).is_file():
+ return version_dir / name
+ return None
+
+
def load_model(path: Path) -> dict:
if path.suffix == ".gz":
with gzip.open(path, "rt", encoding="utf-8") as handle:
@@ -165,10 +182,81 @@ def collect_signing_names(botocore_data: Path) -> dict[str, list[str]]:
return {name: sorted(prefixes) for name, prefixes in sorted(by_signing_name.items())}
+def collect_self_signed_prefixes(botocore_data: Path) -> set[str]:
+ """Endpoint prefixes of the services that sign with their own endpoint prefix."""
+ prefixes: set[str] = set()
+ for service_dir in sorted(p for p in botocore_data.iterdir() if p.is_dir()):
+ versions = sorted(v for v in service_dir.iterdir() if service_model(v) is not None)
+ if not versions:
+ continue
+ metadata = load_model(service_model(versions[-1])).get("metadata", {})
+ endpoint_prefix = metadata.get("endpointPrefix")
+ if endpoint_prefix and (metadata.get("signingName") or endpoint_prefix) == endpoint_prefix:
+ prefixes.add(endpoint_prefix)
+ return prefixes
+
+
+def partition_name_condition(condition: dict) -> str | None:
+ """The partition id a `stringEquals(getAttr(<partition>, "name"), "<id>")` condition tests."""
+ if condition.get("fn") != "stringEquals":
+ return None
+ argv = condition.get("argv", [])
+ if len(argv) != 2:
+ return None
+ for attribute, value in ((argv[0], argv[1]), (argv[1], argv[0])):
+ if (isinstance(attribute, dict) and attribute.get("fn") == "getAttr"
+ and attribute.get("argv", [None, None])[1] == "name" and isinstance(value, str)):
+ return value
+ return None
+
+
+def resolves_to_endpoint(rule: dict) -> bool:
+ if rule.get("type") == "endpoint":
+ return True
+ return rule.get("type") == "tree" and any(resolves_to_endpoint(child) for child in rule.get("rules", []))
+
+
+def ruleset_partitions(rules: list[dict]) -> set[str]:
+ """Partitions a ruleset names in a branch that resolves to an endpoint. A branch that only
+ raises an error, a negated test and the generic `{PartitionResult#dnsSuffix}` template name
+ no partition."""
+ named: set[str] = set()
+ for rule in rules:
+ if resolves_to_endpoint(rule):
+ for condition in rule.get("conditions", []):
+ partition_id = partition_name_condition(condition)
+ if partition_id:
+ named.add(partition_id)
+ if rule.get("type") == "tree":
+ named |= ruleset_partitions(rule.get("rules", []))
+ return named
+
+
+def collect_ruleset_partitions(botocore_data: Path) -> dict[str, list[str]]:
+ """Endpoint prefix -> the partitions its endpoint ruleset names explicitly."""
+ by_prefix: dict[str, set[str]] = {}
+ for service_dir in sorted(p for p in botocore_data.iterdir() if p.is_dir()):
+ versions = sorted(v for v in service_dir.iterdir()
+ if service_model(v) is not None and ruleset_file(v) is not None)
+ if not versions:
+ continue
+ endpoint_prefix = load_model(service_model(versions[-1])).get("metadata", {}).get("endpointPrefix")
+ named = ruleset_partitions(load_model(ruleset_file(versions[-1])).get("rules", []))
+ if endpoint_prefix and named:
+ by_prefix.setdefault(endpoint_prefix, set()).update(named)
+ return {prefix: sorted(partitions) for prefix, partitions in sorted(by_prefix.items())}
+
+
def build(partitions_doc: dict, endpoints_doc: dict, entities: list[str] | None,
carried: dict | None, provenance: str,
- signing_names: dict[str, list[str]] | None = None) -> dict:
+ signing_names: dict[str, list[str]] | None = None,
+ rulesets: dict[str, list[str]] | None = None,
+ self_signed: set[str] | None = None) -> dict:
endpoint_partitions = {p["partition"]: p for p in endpoints_doc["partitions"]}
+ listed_anywhere = {service for p in endpoints_doc["partitions"] for service in p.get("services", {})}
+ if signing_names is not None and self_signed is not None:
+ signing_names = {name: prefixes for name, prefixes in signing_names.items()
+ if name not in self_signed or name in listed_anywhere}
carried_flags = {}
for partition in (carried or {}).get("partitions", []):
for region in partition.get("regions", []):
@@ -228,6 +316,9 @@ def build(partitions_doc: dict, endpoints_doc: dict, entities: list[str] | None,
"signingRegion": signing_region,
"regionalized": service.get("isRegionalized", True) is not False,
}
+ from_rulesets = sorted(
+ prefix for prefix, partitions in (rulesets or {}).items()
+ if partition_id in partitions and prefix in listed_anywhere and prefix not in services)
result_partitions.append({
"id": partition_id,
"name": endpoints_partition.get("partitionName", partition_id),
@@ -239,7 +330,8 @@ def build(partitions_doc: dict, endpoints_doc: dict, entities: list[str] | None,
"supportsDualStack": bool(outputs.get("supportsDualStack", False)),
"supportsFips": bool(outputs.get("supportsFIPS", False)),
"regions": regions,
- "services": sorted(services),
+ "services": sorted(set(services) | set(from_rulesets)),
+ "servicesFromRulesets": from_rulesets,
"globalServices": global_services,
"s3DualStackRegions": s3_dualstack_regions,
})
@@ -269,7 +361,8 @@ def generate(botocore_data: Path, provenance: str, cdk_root: Path | None, existi
if cdk_root is not None and (cdk_root / CDK_ENTITIES).exists():
entities = parse_cdk_entities((cdk_root / CDK_ENTITIES).read_text(encoding="utf-8"))
carried = load_json(existing) if existing.exists() else None
- return build(partitions_doc, endpoints_doc, entities, carried, provenance, collect_signing_names(botocore_data))
+ return build(partitions_doc, endpoints_doc, entities, carried, provenance, collect_signing_names(botocore_data),
+ collect_ruleset_partitions(botocore_data), collect_self_signed_prefixes(botocore_data))
def main(argv: list[str] | None = None) -> int:
diff --git a/tools/aws/test_regen_partitions.py b/tools/aws/test_regen_partitions.py
index 8b75d6d967..534db1e9cc 100644
--- a/tools/aws/test_regen_partitions.py
+++ b/tools/aws/test_regen_partitions.py
@@ -269,6 +269,78 @@ def test_collect_signing_names_reads_the_gzipped_models_the_botocore_wheel_ships
assert r.collect_signing_names(tmp_path) == {"ecr": ["api.ecr"]}
+def partition_is(partition_id):
+ return {"fn": "stringEquals",
+ "argv": [{"fn": "getAttr", "argv": [{"ref": "PartitionResult"}, "name"]}, partition_id]}
+
+
+def endpoint_rule(conditions, url="https://example"):
+ return {"type": "endpoint", "conditions": conditions, "endpoint": {"url": url}}
+
+
+def test_ruleset_partitions_counts_only_explicit_branches_that_resolve_to_an_endpoint():
+ rules = [
+ {"type": "tree", "conditions": [partition_is("aws-eusc")],
+ "rules": [endpoint_rule([], "https://iam.eusc-de-east-1.amazonaws.eu")]},
+ endpoint_rule([partition_is("aws-iso-e")]),
+ {"type": "error", "conditions": [partition_is("aws-iso-f")], "error": "not here"},
+ {"type": "tree", "conditions": [partition_is("aws-iso-b")],
+ "rules": [{"type": "error", "conditions": [], "error": "FIPS is not supported"}]},
+ endpoint_rule([{"fn": "not", "argv": [partition_is("aws-cn")]}]),
+ endpoint_rule([], "https://iam.{Region}.{PartitionResult#dnsSuffix}"),
+ ]
+
+ assert r.ruleset_partitions(rules) == {"aws-eusc", "aws-iso-e"}
+
+
+def write_service(root, service, version, metadata, rules=None, gzipped=False):
+ directory = root / service / version
+ directory.mkdir(parents=True)
+ (directory / "service-2.json").write_text(json.dumps({"metadata": metadata}))
+ if rules is not None:
+ document = {"version": "1.0", "parameters": {}, "rules": rules}
+ if gzipped:
+ with gzip.open(directory / "endpoint-rule-set-1.json.gz", "wt", encoding="utf-8") as handle:
+ json.dump(document, handle)
+ else:
+ (directory / "endpoint-rule-set-1.json").write_text(json.dumps(document))
+
+
+def test_collect_ruleset_partitions_keys_by_endpoint_prefix_and_reads_gzipped_rulesets(tmp_path):
+ write_service(tmp_path, "iam", "2010-05-08", {"endpointPrefix": "iam"},
+ [endpoint_rule([partition_is("aws-eusc")])], gzipped=True)
+ write_service(tmp_path, "sqs", "2012-11-05", {"endpointPrefix": "sqs"}, [endpoint_rule([])])
+ write_service(tmp_path, "nomodel", "2020-01-01", {"endpointPrefix": "nomodel"})
+
+ assert r.collect_ruleset_partitions(tmp_path) == {"iam": ["aws-eusc"]}
+
+
+def test_build_adds_ruleset_partitions_only_for_services_endpoints_json_lists_somewhere():
+ document = r.build(PARTITIONS, ENDPOINTS, r.parse_cdk_entities(ENTITIES), None, "test", {},
+ {"iam": ["aws-eusc"], "fis": ["aws-eusc"], "sqs": ["aws-eusc"]})
+ eusc = by_id(document, "aws-eusc")
+
+ assert eusc["services"] == ["iam", "sqs"]
+ assert eusc["servicesFromRulesets"] == ["iam"]
+ assert by_id(document, "aws")["servicesFromRulesets"] == []
+
+
+def test_build_drops_a_signing_name_whose_own_prefix_service_is_listed_nowhere():
+ signing_names = {"execute-api": ["participant.connect"], "ecr": ["api.ecr"], "iam": ["iam-toolbox"]}
+ document = r.build(PARTITIONS, ENDPOINTS, r.parse_cdk_entities(ENTITIES), None, "test",
+ signing_names, None, {"execute-api", "iam", "sqs"})
+
+ assert document["signingNames"] == {"ecr": ["api.ecr"], "iam": ["iam-toolbox"]}
+
+
+def test_collect_self_signed_prefixes_lists_services_signing_with_their_own_prefix(tmp_path):
+ write_service(tmp_path, "apigatewaymanagementapi", "2018-11-29", {"endpointPrefix": "execute-api"})
+ write_service(tmp_path, "connectparticipant", "2018-09-07",
+ {"endpointPrefix": "participant.connect", "signingName": "execute-api"})
+
+ assert r.collect_self_signed_prefixes(tmp_path) == {"execute-api"}
+
+
def test_build_refuses_a_partition_missing_from_endpoints_json():
endpoints = {"partitions": [p for p in ENDPOINTS["partitions"] if p["partition"] != "aws-eusc"], "version": 3}
with pytest.raises(ValueError, match="aws-eusc"):