Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(redshift): support default IAM roles for COPY and UNLOAD (#6304)

Vũ Thế Huy7e51e69c4495
diff --git a/docs/services/redshift.md b/docs/services/redshift.md
index f85a78aa75..5e45347ff1 100644
--- a/docs/services/redshift.md
+++ b/docs/services/redshift.md
@@ -56,7 +56,7 @@ For running SQL without a PostgreSQL wire connection (the way Lambda and Step Fu
| `ModifyCluster` | Update node type, parameter group, security groups, Multi-AZ flag, or the master password |
| `DescribeClusterVersions` | Return the single emulated engine version and its parameter group family |
| `DescribeOrderableClusterOptions` | Return the static node types and cluster types, optionally filtered by `NodeType` or `ClusterVersion` |
-| `ModifyClusterIamRoles` | Add or remove the IAM roles associated with a cluster; COPY and UNLOAD see the change on new connections. `DefaultIamRoleArn` is ignored |
+| `ModifyClusterIamRoles` | Add or remove associated IAM roles and set `DefaultIamRoleArn`; COPY and UNLOAD see the change on new connections |
| `DescribeLoggingStatus` | Return a cluster's stored audit-logging configuration |
| `EnableLogging` | Store audit-logging configuration (S3 bucket, CloudWatch, or S3 table via `LogDestinationType`); no logs are delivered |
| `DisableLogging` | Clear a cluster's audit-logging configuration |
@@ -99,6 +99,37 @@ Sources are persisted with parameter values. For older stored groups without a s
reports `engine-default` when the value matches its catalog and `user` otherwise. The old format
cannot distinguish an explicit assignment of a default value from an unchanged default.
+### Default IAM role for COPY and UNLOAD
+
+`CreateCluster` and `ModifyClusterIamRoles` accept `DefaultIamRoleArn`. It must name a role
+in the cluster's associated `IamRoles` list, including roles added in the same request.
+`DescribeClusters` returns the configured ARN. Removing that role clears the default.
+An explicit empty or whitespace-only `DefaultIamRoleArn` means no default: it creates a cluster
+without one or clears an existing default while retaining the associated roles. Omitting the field
+on an update otherwise keeps the existing default. The setting uses the
+cluster's configured storage backend and is reapplied when its proxy restarts.
+
+Both `COPY ... IAM_ROLE default` and `UNLOAD ... IAM_ROLE default` use that role, in Simple
+Query and zero-parameter Extended Query statements. A cluster without a default fails these
+statements with SQLSTATE `42501`; it never falls back to unsigned S3 access. Role association,
+trust and S3 permission checks are the same as for an explicit role ARN. Existing connections
+keep the role configuration captured when they connected; open a new connection after changing it.
+
+```bash
+aws redshift modify-cluster-iam-roles \
+ --cluster-identifier my-warehouse \
+ --add-iam-roles "$REDSHIFT_ROLE_ARN" \
+ --default-iam-role-arn "$REDSHIFT_ROLE_ARN"
+```
+
+```sql
+COPY people FROM 's3://my-data/people.csv' IAM_ROLE default CSV;
+UNLOAD ('SELECT * FROM people') TO 's3://my-data/export/' IAM_ROLE default CSV;
+```
+
+See AWS's [default IAM role guide](https://docs.aws.amazon.com/redshift/latest/mgmt/default-iam-role.html)
+and [COPY authorization parameters](https://docs.aws.amazon.com/redshift/latest/dg/copy-parameters-authorization.html).
+
### Snapshot copy grants
- **`KmsKeyId` defaults to an alias ARN.** Floci has no per-account AWS-managed key, so
@@ -449,7 +480,7 @@ order) through its own S3 service and streams the rows into the backing PostgreS
the local IAM service under that ARN, path included, and trust Redshift to assume it. With
`FLOCI_SERVICES_S3_ENFORCE_AUTH` off, S3 policy checks are skipped. With it on, the role's
identity policy must allow the required S3 actions, and any bucket policy must not deny the
- request. `IAM_ROLE default` is not supported.
+ request. `IAM_ROLE default` uses the cluster's default role with the same permission checks.
- `REGION '<region>'` is accepted and ignored: Floci has one S3.
- `CREDENTIALS 'aws_iam_role=<arn>'` behaves like `IAM_ROLE`. `ACCESS_KEY_ID` with `SECRET_ACCESS_KEY`
(and an optional `SESSION_TOKEN`), or `CREDENTIALS 'aws_access_key_id=...;aws_secret_access_key=...'`,
@@ -549,7 +580,7 @@ the result to S3 as one or more objects under `<prefix>`.
the local IAM service under that ARN, path included, and trust Redshift to assume it. With
`FLOCI_SERVICES_S3_ENFORCE_AUTH` off, S3 policy checks are skipped. With it on, the role's
identity policy must allow the required S3 actions, and any bucket policy must not deny the
- request. `IAM_ROLE default` is not supported.
+ request. `IAM_ROLE default` uses the cluster's default role with the same permission checks.
- Any other option (`PARQUET`, `JSON`, `FIXEDWIDTH`, `MASTER_SYMMETRIC_KEY`, `MANIFEST VERBOSE`,
`ZSTD`, `BZIP2`, `PARTITION`, and so on) is not intercepted; the
statement is forwarded and PostgreSQL reports its own error.
@@ -588,7 +619,7 @@ These views expose the documented Redshift column names, types, and ordering map
- Parameter groups apply no real engine settings; values are stored and echoed back only.
- Subnet groups, VPC routing, and security groups are metadata only.
- Resize, pause/resume, and IAM authentication. Snapshot schedules and cross-region snapshot copy are stored and reported, but no snapshot is taken on a schedule or copied to another region.
-- `IAM_ROLE default` and cross-account role ARNs are not supported for COPY or UNLOAD.
+- Cross-account role ARNs are not supported for COPY or UNLOAD.
- The auth proxy validates the master user's password and any live `GetClusterCredentials` credential. Other non-master users pass straight through to PostgreSQL, which remains the authority for their credentials.
- A cluster using `ManageMasterPassword` keeps its generated password in sync with the
Redshift-owned Secrets Manager secret. Updating `MasterUserPassword` through `ModifyCluster`
diff --git a/pom.xml b/pom.xml
index 68b5ba895b..79da85b607 100644
--- a/pom.xml
+++ b/pom.xml
@@ -342,6 +342,12 @@
<version>${aws.sdk.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>redshift</artifactId>
+ <version>${aws.sdk.version}</version>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>ssm</artifactId>
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftQueryHandler.java b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftQueryHandler.java
index 31c8544b50..94d7956a50 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftQueryHandler.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftQueryHandler.java
@@ -91,15 +91,13 @@ public class RedshiftQueryHandler {
List<String> iamRoleArns = memberList(params, "IamRoles");
String region = regionResolver.resolveRegionFromAuth(authorizationHeader);
+ String defaultIamRoleArn = params.getFirst("DefaultIamRoleArn");
Cluster cluster = manageMasterPassword
? service.createClusterWithManagedMasterPassword(identifier, nodeType, masterUsername,
clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns,
- params.getFirst("MasterPasswordSecretKmsKeyId"), region)
- : iamRoleArns.isEmpty()
- ? service.createCluster(identifier, nodeType, masterUsername, masterUserPassword,
- clusterSubnetGroupName, vpcSecurityGroupIds)
+ params.getFirst("MasterPasswordSecretKmsKeyId"), region, defaultIamRoleArn)
: service.createCluster(identifier, nodeType, masterUsername, masterUserPassword,
- clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns);
+ clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns, defaultIamRoleArn);
String parameterGroupName = params.getFirst("ClusterParameterGroupName");
String multiAz = params.getFirst("MultiAZ");
boolean hasParameterGroup = parameterGroupName != null && !parameterGroupName.isBlank();
@@ -748,7 +746,8 @@ public class RedshiftQueryHandler {
throw new AwsException("InvalidParameterValue", "ClusterIdentifier is required", 400);
}
Cluster cluster = service.modifyClusterIamRoles(clusterIdentifier,
- memberList(params, "AddIamRoles"), memberList(params, "RemoveIamRoles"));
+ memberList(params, "AddIamRoles"), memberList(params, "RemoveIamRoles"),
+ params.getFirst("DefaultIamRoleArn"));
String xml = new XmlBuilder()
.start("ModifyClusterIamRolesResponse")
.start("ModifyClusterIamRolesResult")
@@ -941,6 +940,9 @@ public class RedshiftQueryHandler {
builder.end("VpcSecurityGroups");
}
+ if (cluster.getDefaultIamRoleArn() != null) {
+ builder.elem("DefaultIamRoleArn", cluster.getDefaultIamRoleArn());
+ }
if (cluster.getIamRoleArns() != null && !cluster.getIamRoleArns().isEmpty()) {
builder.start("IamRoles");
for (String iamRoleArn : cluster.getIamRoleArns()) {
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftService.java b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftService.java
index fdd7fefe44..6c67db74b5 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftService.java
@@ -154,7 +154,7 @@ public class RedshiftService {
handle.getHost(), handle.getPort(), endpoint.getAddress(),
cluster.getMasterUsername(), password, CLUSTER_DB_NAME,
passwordValidatorFor(entry.accountId(), cluster.getClusterIdentifier()),
- cluster.getIamRoleArns());
+ cluster.getIamRoleArns(), cluster.getDefaultIamRoleArn());
cluster.setContainerHost(handle.getHost());
cluster.setContainerPort(handle.getPort());
cluster.setEndpoint(endpoint);
@@ -194,9 +194,18 @@ public class RedshiftService {
public synchronized Cluster createCluster(String identifier, String nodeType, String username, String password,
String clusterSubnetGroupName, List<String> vpcSecurityGroupIds,
List<String> iamRoleArns) {
+ return createCluster(identifier, nodeType, username, password, clusterSubnetGroupName,
+ vpcSecurityGroupIds, iamRoleArns, null);
+ }
+
+ public synchronized Cluster createCluster(String identifier, String nodeType, String username, String password,
+ String clusterSubnetGroupName, List<String> vpcSecurityGroupIds,
+ List<String> iamRoleArns, String defaultIamRoleArn) {
if (clusters.get(identifier).isPresent()) {
throw new AwsException("ClusterAlreadyExists", "Cluster " + identifier + " already exists", 400);
}
+ List<String> roles = iamRoleArns == null ? List.of() : List.copyOf(iamRoleArns);
+ String defaultRole = validateDefaultIamRole(defaultIamRoleArn, roles);
// A previous cluster with this identifier may have been deleted without its temp
// credentials being cleared; drop them so the new cluster starts with none.
credentialBroker.revokeCluster(clusters.accountId(), identifier);
@@ -208,7 +217,8 @@ public class RedshiftService {
cluster.setMasterPassword(password);
cluster.setClusterSubnetGroupName(clusterSubnetGroupName);
cluster.setVpcSecurityGroupIds(vpcSecurityGroupIds != null ? vpcSecurityGroupIds : List.of());
- cluster.setIamRoleArns(iamRoleArns != null ? List.copyOf(iamRoleArns) : List.of());
+ cluster.setIamRoleArns(roles);
+ cluster.setDefaultIamRoleArn(defaultRole);
// A group named at create time is applied afterwards via modifyCluster.
cluster.setClusterParameterGroupName(DEFAULT_PARAMETER_GROUP_NAME);
cluster.setClusterStatus("creating");
@@ -228,7 +238,7 @@ public class RedshiftService {
proxyManager.startProxy(relayKey(accountId, identifier), proxyPort,
handle.getHost(), handle.getPort(), endpoint.getAddress(),
username, password, CLUSTER_DB_NAME,
- passwordValidatorFor(accountId, identifier), cluster.getIamRoleArns());
+ passwordValidatorFor(accountId, identifier), cluster.getIamRoleArns(), cluster.getDefaultIamRoleArn());
cluster.setContainerHost(handle.getHost());
cluster.setContainerPort(handle.getPort());
cluster.setEndpoint(endpoint);
@@ -268,12 +278,20 @@ public class RedshiftService {
String identifier, String nodeType, String username, String clusterSubnetGroupName,
List<String> vpcSecurityGroupIds, List<String> iamRoleArns,
String kmsKeyId, String region) {
+ return createClusterWithManagedMasterPassword(identifier, nodeType, username, clusterSubnetGroupName,
+ vpcSecurityGroupIds, iamRoleArns, kmsKeyId, region, null);
+ }
+
+ public synchronized Cluster createClusterWithManagedMasterPassword(
+ String identifier, String nodeType, String username, String clusterSubnetGroupName,
+ List<String> vpcSecurityGroupIds, List<String> iamRoleArns,
+ String kmsKeyId, String region, String defaultIamRoleArn) {
if (secretsManagerService == null) {
throw new AwsException("InternalFailure", "Secrets Manager is unavailable", 500);
}
String password = RandomPasswordGenerator.generate(objectMapper.createObjectNode());
Cluster cluster = createCluster(identifier, nodeType, username, password,
- clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns);
+ clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns, defaultIamRoleArn);
String secretName = "redshift/" + identifier;
String secretString = managedMasterSecret(cluster, password);
Secret secret;
@@ -677,6 +695,11 @@ public class RedshiftService {
public synchronized Cluster modifyClusterIamRoles(String clusterIdentifier, List<String> addIamRoles,
List<String> removeIamRoles) {
+ return modifyClusterIamRoles(clusterIdentifier, addIamRoles, removeIamRoles, null);
+ }
+
+ public synchronized Cluster modifyClusterIamRoles(String clusterIdentifier, List<String> addIamRoles,
+ List<String> removeIamRoles, String defaultIamRoleArn) {
Cluster cluster = clusters.get(clusterIdentifier)
.orElseThrow(() -> new AwsException("ClusterNotFound", "Cluster " + clusterIdentifier + " not found", 404));
@@ -687,14 +710,32 @@ public class RedshiftService {
Set<String> roles = new LinkedHashSet<>(cluster.getIamRoleArns());
roles.addAll(addIamRoles);
roles.removeAll(removeIamRoles);
+ String defaultRole = defaultIamRoleArn == null ? cluster.getDefaultIamRoleArn() : defaultIamRoleArn;
+ if (defaultIamRoleArn == null && !roles.contains(defaultRole)) {
+ defaultRole = null;
+ }
+ defaultRole = validateDefaultIamRole(defaultRole, List.copyOf(roles));
cluster.setIamRoleArns(List.copyOf(roles));
+ cluster.setDefaultIamRoleArn(defaultRole);
- proxyManager.updateIamRoles(relayKey(clusters.accountId(), clusterIdentifier), cluster.getIamRoleArns());
+ proxyManager.updateIamRoles(relayKey(clusters.accountId(), clusterIdentifier), cluster.getIamRoleArns(), defaultRole);
clusters.put(clusterIdentifier, cluster);
clusters.flush();
return cluster;
}
+ private static String validateDefaultIamRole(String defaultIamRoleArn, List<String> iamRoleArns) {
+ if (defaultIamRoleArn == null || defaultIamRoleArn.isBlank()) {
+ return null;
+ }
+ requireIamRoleArn(defaultIamRoleArn);
+ if (!iamRoleArns.contains(defaultIamRoleArn)) {
+ throw new AwsException("InvalidParameterValue",
+ "Default IAM role must be associated with the cluster: " + defaultIamRoleArn, 400);
+ }
+ return defaultIamRoleArn;
+ }
+
private static void requireIamRoleArn(String arn) {
AwsArnUtils.Arn parsed;
try {
@@ -829,7 +870,7 @@ public class RedshiftService {
cluster.setProxyPort(proxyPort);
proxyManager.startProxy(key, proxyPort, handle.getHost(), handle.getPort(),
endpoint.getAddress(), cluster.getMasterUsername(), password, CLUSTER_DB_NAME,
- passwordValidatorFor(accountId, clusterIdentifier), cluster.getIamRoleArns());
+ passwordValidatorFor(accountId, clusterIdentifier), cluster.getIamRoleArns(), cluster.getDefaultIamRoleArn());
cluster.setContainerHost(handle.getHost());
cluster.setContainerPort(handle.getPort());
cluster.setEndpoint(endpoint);
@@ -1063,7 +1104,7 @@ public class RedshiftService {
proxyManager.startProxy(relayKey(accountId, clusterIdentifier), proxyPort,
handle.getHost(), handle.getPort(), endpoint.getAddress(),
username, password, CLUSTER_DB_NAME,
- passwordValidatorFor(accountId, clusterIdentifier), cluster.getIamRoleArns());
+ passwordValidatorFor(accountId, clusterIdentifier), cluster.getIamRoleArns(), cluster.getDefaultIamRoleArn());
cluster.setContainerHost(handle.getHost());
cluster.setContainerPort(handle.getPort());
cluster.setEndpoint(endpoint);
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/model/Cluster.java b/src/main/java/io/github/hectorvent/floci/services/redshift/model/Cluster.java
index 6629606339..55862c130c 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/model/Cluster.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/model/Cluster.java
@@ -21,6 +21,7 @@ public class Cluster {
private String clusterParameterGroupName;
private List<String> vpcSecurityGroupIds = new ArrayList<>();
private List<String> iamRoleArns = new ArrayList<>();
+ private String defaultIamRoleArn;
public String getClusterIdentifier() { return clusterIdentifier; }
public void setClusterIdentifier(String clusterIdentifier) { this.clusterIdentifier = clusterIdentifier; }
@@ -46,6 +47,8 @@ public class Cluster {
public void setVpcSecurityGroupIds(List<String> vpcSecurityGroupIds) { this.vpcSecurityGroupIds = vpcSecurityGroupIds; }
public List<String> getIamRoleArns() { return iamRoleArns; }
public void setIamRoleArns(List<String> iamRoleArns) { this.iamRoleArns = iamRoleArns; }
+ public String getDefaultIamRoleArn() { return defaultIamRoleArn; }
+ public void setDefaultIamRoleArn(String defaultIamRoleArn) { this.defaultIamRoleArn = defaultIamRoleArn; }
// Audit logging config as set by EnableLogging; no log delivery is emulated.
private boolean loggingEnabled = false;
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.java
index b30ac9ec3a..aeb09abea7 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.java
@@ -15,6 +15,9 @@ import java.util.regex.Pattern;
public final class CopyStatementParser {
public sealed interface S3Statement permits S3CopyFrom, S3Unload {
+ String iamRoleArn();
+
+ boolean defaultIamRoleRequested();
}
/** COPY options that change field content; {@link #NONE} keeps the plain byte-copy path. */
@@ -50,7 +53,8 @@ public final class CopyStatementParser {
boolean jsonAuto,
boolean jsonAutoIgnoreCase,
boolean manifest,
- CopyTransforms transforms) implements S3Statement {
+ CopyTransforms transforms,
+ boolean defaultIamRoleRequested) implements S3Statement {
public S3CopyFrom {
if (transforms == null) {
@@ -58,6 +62,25 @@ public final class CopyStatementParser {
}
}
+ public S3CopyFrom(
+ String targetTable,
+ List<String> columns,
+ String bucket,
+ String keyOrPrefix,
+ String delimiter,
+ int headerLines,
+ boolean gzip,
+ boolean csv,
+ String nullAs,
+ String iamRoleArn,
+ boolean jsonAuto,
+ boolean jsonAutoIgnoreCase,
+ boolean manifest,
+ CopyTransforms transforms) {
+ this(targetTable, columns, bucket, keyOrPrefix, delimiter, headerLines, gzip, csv,
+ nullAs, iamRoleArn, jsonAuto, jsonAutoIgnoreCase, manifest, transforms, false);
+ }
+
public S3CopyFrom(
String targetTable,
List<String> columns,
@@ -128,7 +151,33 @@ public final class CopyStatementParser {
boolean escape,
boolean cleanPath,
boolean encrypted,
- String sseKmsKeyId) implements S3Statement {
+ String sseKmsKeyId,
+ boolean defaultIamRoleRequested) implements S3Statement {
+
+ public S3Unload(
+ String selectQuery,
+ String bucket,
+ String prefix,
+ String delimiter,
+ boolean header,
+ boolean gzip,
+ boolean csv,
+ boolean addQuotes,
+ String nullAs,
+ boolean manifest,
+ boolean allowOverwrite,
+ boolean parallel,
+ long maxFileSizeBytes,
+ String iamRoleArn,
+ String extension,
+ boolean escape,
+ boolean cleanPath,
+ boolean encrypted,
+ String sseKmsKeyId) {
+ this(selectQuery, bucket, prefix, delimiter, header, gzip, csv, addQuotes, nullAs, manifest,
+ allowOverwrite, parallel, maxFileSizeBytes, iamRoleArn, extension,
+ escape, cleanPath, encrypted, sseKmsKeyId, false);
+ }
public S3Unload(
String selectQuery,
@@ -213,7 +262,7 @@ public final class CopyStatementParser {
private static final Pattern NULL_AS_PATTERN = Pattern.compile(
"(?i)\\bNULL\\s+(?:AS\\s+)?(?:'((?:[^']|'')*)'|\"([^\"]*)\")");
private static final Pattern IAM_ROLE_PATTERN = Pattern.compile(
- "(?i)\\bIAM_ROLE\\s+(?:'((?:[^']|'')*)'|\"([^\"]*)\")");
+ "(?i)\\bIAM_ROLE\\s+(?:'((?:[^']|'')*)'|\"([^\"]*)\"|(default)(?=\\s|;|$))");
private static final Pattern BLANKSASNULL_PATTERN = Pattern.compile("(?i)\\bBLANKSASNULL\\b");
private static final Pattern EMPTYASNULL_PATTERN = Pattern.compile("(?i)\\bEMPTYASNULL\\b");
private static final Pattern REMOVEQUOTES_PATTERN = Pattern.compile("(?i)\\bREMOVEQUOTES\\b");
@@ -264,6 +313,10 @@ public final class CopyStatementParser {
}
public static S3Statement parse(String sql) {
+ return parse(sql, null);
+ }
+
+ public static S3Statement parse(String sql, String defaultIamRoleArn) {
if (sql == null || sql.isBlank()) {
return null;
}
@@ -273,12 +326,12 @@ public final class CopyStatementParser {
}
Matcher unload = UNLOAD_PATTERN.matcher(cleaned);
if (unload.matches()) {
- return parseUnload(unload);
+ return parseUnload(unload, defaultIamRoleArn);
}
- return parseCopy(cleaned);
+ return parseCopy(cleaned, defaultIamRoleArn);
}
- private static S3Unload parseUnload(Matcher matcher) {
+ private static S3Unload parseUnload(Matcher matcher, String defaultIamRoleArn) {
String select = unescapeSingleQuotes(matcher.group(1).trim());
if (!isSafeUnloadSubquery(select)) {
return null;
@@ -348,7 +401,7 @@ public final class CopyStatementParser {
Matcher cleanPathM = CLEANPATH_PATTERN.matcher(options);
Matcher encryptedM = ENCRYPTED_PATTERN.matcher(options);
Matcher kmsKeyIdM = KMS_KEY_ID_PATTERN.matcher(options);
- AuthClauses auth = new AuthClauses(options);
+ AuthClauses auth = new AuthClauses(options, defaultIamRoleArn);
int next;
int offset = 0;
@@ -509,10 +562,10 @@ public final class CopyStatementParser {
}
return new S3Unload(select, bucket, prefix, delimiter, header, gzip, csv,
addQuotes, nullAs, manifest, allowOverwrite, parallel, maxFileSizeBytes, iamRoleArn, extension,
- escape, cleanPath, encrypted, kmsKeyId);
+ escape, cleanPath, encrypted, kmsKeyId, auth.defaultIamRoleRequested);
}
- private static S3CopyFrom parseCopy(String cleaned) {
+ private static S3CopyFrom parseCopy(String cleaned, String defaultIamRoleArn) {
Matcher matcher = COPY_PATTERN.matcher(cleaned);
if (!matcher.matches()) {
return null;
@@ -594,7 +647,7 @@ public final class CopyStatementParser {
Matcher removeQuotesMatcher = REMOVEQUOTES_PATTERN.matcher(options);
Matcher truncateColumnsMatcher = TRUNCATECOLUMNS_PATTERN.matcher(options);
Matcher acceptInvCharsMatcher = ACCEPTINVCHARS_PATTERN.matcher(options);
- AuthClauses auth = new AuthClauses(options);
+ AuthClauses auth = new AuthClauses(options, defaultIamRoleArn);
int next;
int offset = 0;
@@ -735,7 +788,7 @@ public final class CopyStatementParser {
}
return new S3CopyFrom(table, columns, bucket, keyOrPrefix, delimiter, headerLines, gzip, csv,
- nullAs, iamRoleArn, jsonAuto, jsonAutoIgnoreCase, manifest, transforms);
+ nullAs, iamRoleArn, jsonAuto, jsonAutoIgnoreCase, manifest, transforms, auth.defaultIamRoleRequested);
}
/**
@@ -753,10 +806,12 @@ public final class CopyStatementParser {
private final Matcher secretAccessKey;
private final Matcher sessionToken;
private final Matcher region;
+ private final String defaultIamRoleArn;
private String iamRoleArn;
private String credentialsRoleArn;
private boolean seenIamRole;
+ private boolean defaultIamRoleRequested;
private boolean seenCredentials;
private boolean seenAccessKeyId;
private boolean seenSecretAccessKey;
@@ -765,7 +820,8 @@ public final class CopyStatementParser {
private boolean credentialsHaveKeys;
private boolean credentialsMalformed;
- AuthClauses(String options) {
+ AuthClauses(String options, String defaultIamRoleArn) {
+ this.defaultIamRoleArn = defaultIamRoleArn;
iamRole = IAM_ROLE_PATTERN.matcher(options);
credentials = CREDENTIALS_PATTERN.matcher(options);
accessKeyId = ACCESS_KEY_ID_PATTERN.matcher(options);
@@ -780,8 +836,9 @@ public final class CopyStatementParser {
return REJECT;
}
seenIamRole = true;
- iamRoleArn = extractNullValue(iamRole);
- return isBlank(iamRoleArn) ? REJECT : iamRole.end();
+ defaultIamRoleRequested = iamRole.group(3) != null;
+ iamRoleArn = defaultIamRoleRequested ? defaultIamRoleArn : extractNullValue(iamRole);
+ return !defaultIamRoleRequested && isBlank(iamRoleArn) ? REJECT : iamRole.end();
}
if (matchClause(credentials, offset, end)) {
if (seenCredentials) {
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftAuthProxy.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftAuthProxy.java
index 57381b0619..faffad3fb8 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftAuthProxy.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftAuthProxy.java
@@ -41,7 +41,13 @@ public class RedshiftAuthProxy {
private final IamService iamService;
private final SpectrumInterceptor spectrumInterceptor;
private final String clusterAccountId;
- private volatile List<String> iamRoleArns;
+ private volatile RoleConfiguration roleConfiguration;
+
+ record RoleConfiguration(List<String> iamRoleArns, String defaultIamRoleArn) {
+ RoleConfiguration {
+ iamRoleArns = iamRoleArns == null ? List.of() : List.copyOf(iamRoleArns);
+ }
+ }
private final int handshakeTimeoutMillis;
private final int backendConnectTimeoutMillis;
private final Semaphore connectionPermits;
@@ -98,7 +104,7 @@ public class RedshiftAuthProxy {
this.iamService = iamService;
this.spectrumInterceptor = spectrumInterceptor;
this.clusterAccountId = clusterAccountId;
- this.iamRoleArns = iamRoleArns == null ? List.of() : List.copyOf(iamRoleArns);
+ this.roleConfiguration = new RoleConfiguration(iamRoleArns, null);
this.handshakeTimeoutMillis = handshakeTimeoutMillis;
this.backendConnectTimeoutMillis = backendConnectTimeoutMillis;
this.connectionPermits = new Semaphore(Math.max(1, maxConnections));
@@ -159,8 +165,16 @@ public class RedshiftAuthProxy {
}
/** Swap the associated-role snapshot; new connections authorize COPY and UNLOAD against it. */
- public void updateIamRoles(List<String> newIamRoleArns) {
- this.iamRoleArns = newIamRoleArns == null ? List.of() : List.copyOf(newIamRoleArns);
+ public synchronized void updateIamRoles(List<String> newIamRoleArns) {
+ updateIamRoles(newIamRoleArns, roleConfiguration.defaultIamRoleArn());
+ }
+
+ public synchronized void updateIamRoles(List<String> newIamRoleArns, String defaultIamRoleArn) {
+ this.roleConfiguration = new RoleConfiguration(newIamRoleArns, defaultIamRoleArn);
+ }
+
+ RoleConfiguration roleConfiguration() {
+ return roleConfiguration;
}
public void stop() {
@@ -221,8 +235,10 @@ public class RedshiftAuthProxy {
if (session != null) {
// Redshift-only DDL (DISTKEY/SORTKEY/ENCODE/...) is rewritten for the plain
// PostgreSQL backend on the way through; every other message is relayed verbatim.
+ RoleConfiguration roles = roleConfiguration;
new RedshiftInterceptingBridge(session.client(), session.backend(), s3Service, iamService,
- clusterAccountId, iamRoleArns, spectrumInterceptor, clusterKey, dbName).run();
+ clusterAccountId, roles.iamRoleArns(), spectrumInterceptor, clusterKey, dbName,
+ roles.defaultIamRoleArn()).run();
}
} catch (Exception e) {
LOG.debugv("Redshift connection error for cluster {0}: {1}", clusterKey, e.getMessage());
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftInterceptingBridge.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftInterceptingBridge.java
index 6edbd375bc..e623412779 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftInterceptingBridge.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftInterceptingBridge.java
@@ -61,6 +61,7 @@ public class RedshiftInterceptingBridge {
private final IamService iamService;
private final String clusterAccountId;
private final List<String> iamRoleArns;
+ private final String defaultIamRoleArn;
private final SpectrumInterceptor spectrumInterceptor;
private final String clusterKey;
private final String databaseName;
@@ -96,12 +97,21 @@ public class RedshiftInterceptingBridge {
public RedshiftInterceptingBridge(Socket client, Socket backend, S3Service s3Service, IamService iamService,
String clusterAccountId, List<String> iamRoleArns,
SpectrumInterceptor spectrumInterceptor, String clusterKey, String databaseName) {
+ this(client, backend, s3Service, iamService, clusterAccountId, iamRoleArns, spectrumInterceptor,
+ clusterKey, databaseName, null);
+ }
+
+ public RedshiftInterceptingBridge(Socket client, Socket backend, S3Service s3Service, IamService iamService,
+ String clusterAccountId, List<String> iamRoleArns,
+ SpectrumInterceptor spectrumInterceptor, String clusterKey, String databaseName,
+ String defaultIamRoleArn) {
this.client = client;
this.backend = backend;
this.s3Service = s3Service;
this.iamService = iamService;
this.clusterAccountId = clusterAccountId;
this.iamRoleArns = iamRoleArns == null ? List.of() : List.copyOf(iamRoleArns);
+ this.defaultIamRoleArn = defaultIamRoleArn;
this.spectrumInterceptor = spectrumInterceptor;
this.clusterKey = clusterKey;
this.databaseName = databaseName;
@@ -565,7 +575,7 @@ public class RedshiftInterceptingBridge {
private CopyStatementParser.S3Statement parseS3Statement(String sql) {
try {
- return CopyStatementParser.parse(sql);
+ return CopyStatementParser.parse(sql, defaultIamRoleArn);
} catch (RuntimeException e) {
LOG.warnv("CopyStatementParser failed, forwarding original statement: {0}", e.getMessage());
return null;
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManager.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManager.java
index cd09cb0a38..ed9ffb2bff 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManager.java
@@ -71,6 +71,15 @@ public class RedshiftProxyManager {
String backendHost, int backendPort, String advertisedHost,
String masterUsername, String masterPassword, String dbName,
PasswordValidator passwordValidator, List<String> iamRoleArns) {
+ startProxy(relayKey, proxyPort, backendHost, backendPort, advertisedHost, masterUsername,
+ masterPassword, dbName, passwordValidator, iamRoleArns, null);
+ }
+
+ public synchronized void startProxy(String relayKey, int proxyPort,
+ String backendHost, int backendPort, String advertisedHost,
+ String masterUsername, String masterPassword, String dbName,
+ PasswordValidator passwordValidator, List<String> iamRoleArns,
+ String defaultIamRoleArn) {
// A prior unclosable entry for this key is left in place: its listener may still
// be bound, and only a successful stop (never a fresh start) may drop it.
// Make sure the self-signed proxy certificate covers the host clients will connect to,
@@ -86,6 +95,7 @@ public class RedshiftProxyManager {
iamRoleArns,
redshiftConfig.proxyHandshakeTimeoutMillis(), redshiftConfig.proxyBackendConnectTimeoutMillis(),
redshiftConfig.proxyMaxConnections(), spectrumInterceptor);
+ proxy.updateIamRoles(iamRoleArns, defaultIamRoleArn);
try {
proxy.start(proxyPort);
} catch (IOException | RuntimeException e) {
@@ -124,6 +134,13 @@ public class RedshiftProxyManager {
}
}
+ public synchronized void updateIamRoles(String relayKey, List<String> iamRoleArns, String defaultIamRoleArn) {
+ RedshiftAuthProxy proxy = proxies.get(relayKey);
+ if (proxy != null) {
+ proxy.updateIamRoles(iamRoleArns, defaultIamRoleArn);
+ }
+ }
+
public synchronized void stopProxy(String relayKey) {
// Retry any listener a previous failed start/replace could not close. If it still
// cannot be closed this throws, and the entry stays for the next retry.
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.java
index 01752a1dc2..b3dfb0edc9 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.java
@@ -68,6 +68,13 @@ public final class S3CopySimulator {
private S3CopySimulator() {
}
+ private static void requireDefaultIamRole(CopyStatementParser.S3Statement statement) {
+ if (statement.defaultIamRoleRequested() && statement.iamRoleArn() == null) {
+ throw new S3TransferException(SQLSTATE_INSUFFICIENT_PRIVILEGE,
+ "No default IAM role is associated with the Redshift cluster", null);
+ }
+ }
+
record CopyInput(CopyStatementParser.S3CopyFrom spec, List<String> keys, S3Service s3,
IamService iamService, RedshiftRoleAccess.RoleSession roleSession) {
}
@@ -107,6 +114,7 @@ public final class S3CopySimulator {
static CopyInput prepareCopy(CopyStatementParser.S3CopyFrom spec, S3Service s3, IamService iamService,
String clusterAccountId, List<String> associatedRoleArns) {
+ requireDefaultIamRole(spec);
RedshiftRoleAccess.RoleSession roleSession = spec.iamRoleArn() != null
? RedshiftRoleAccess.resolveRoleSession(spec.iamRoleArn(), iamService, clusterAccountId, associatedRoleArns)
: null;
@@ -204,6 +212,7 @@ public final class S3CopySimulator {
static UnloadCollector prepareUnload(CopyStatementParser.S3Unload spec, S3Service s3, IamService iamService,
String clusterAccountId, List<String> associatedRoleArns) {
+ requireDefaultIamRole(spec);
RedshiftRoleAccess.RoleSession roleSession = spec.iamRoleArn() != null
? RedshiftRoleAccess.resolveRoleSession(spec.iamRoleArn(), iamService, clusterAccountId, associatedRoleArns)
: null;
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftDefaultIamRoleIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftDefaultIamRoleIntegrationTest.java
new file mode 100644
index 0000000000..8e4ea13825
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftDefaultIamRoleIntegrationTest.java
@@ -0,0 +1,91 @@
+package io.github.hectorvent.floci.services.redshift;
+
+import io.github.hectorvent.floci.services.redshift.container.RedshiftContainerHandle;
+import io.github.hectorvent.floci.services.redshift.container.RedshiftContainerManager;
+import io.quarkus.test.InjectMock;
+import io.quarkus.test.common.http.TestHTTPResource;
+import io.quarkus.test.junit.QuarkusTest;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.redshift.RedshiftClient;
+import software.amazon.awssdk.services.redshift.model.Cluster;
+import software.amazon.awssdk.services.redshift.model.RedshiftException;
+
+import java.net.URI;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.when;
+
+@QuarkusTest
+class RedshiftDefaultIamRoleIntegrationTest {
+
+ @InjectMock
+ RedshiftContainerManager containerManager;
+
+ @TestHTTPResource
+ URI endpoint;
+
+ @ParameterizedTest
+ @ValueSource(strings = {"", "arn:aws:iam::000000000000:role/first"})
+ void sdkCreatesChangesAndRemovesDefaultRole(String initialDefault) {
+ String clusterId = "sdk-default-role";
+ String first = "arn:aws:iam::000000000000:role/first";
+ String second = "arn:aws:iam::000000000000:role/second";
+ when(containerManager.start(anyString(), anyString(), anyString(), anyString()))
+ .thenReturn(new RedshiftContainerHandle("mock-container", clusterId, "localhost", 5432));
+ try (RedshiftClient client = RedshiftClient.builder()
+ .endpointOverride(endpoint)
+ .region(Region.US_EAST_1)
+ .credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create("test", "test")))
+ .build()) {
+ Cluster created = client.createCluster(request -> request.clusterIdentifier(clusterId)
+ .nodeType("dc2.large").masterUsername("admin").masterUserPassword("Secret123")
+ .iamRoles(first).defaultIamRoleArn(initialDefault)).cluster();
+ try {
+ assertEquals(initialDefault.isBlank() ? null : initialDefault, created.defaultIamRoleArn());
+ Cluster addedWithoutDefault = client.modifyClusterIamRoles(request -> request.clusterIdentifier(clusterId)
+ .addIamRoles(second).defaultIamRoleArn("")).cluster();
+ assertNull(addedWithoutDefault.defaultIamRoleArn());
+ assertEquals(2, addedWithoutDefault.iamRoles().size());
+ Cluster removedWithoutDefault = client.modifyClusterIamRoles(request -> request.clusterIdentifier(clusterId)
+ .removeIamRoles(second).defaultIamRoleArn("")).cluster();
+ assertNull(removedWithoutDefault.defaultIamRoleArn());
+ assertEquals(first, removedWithoutDefault.iamRoles().getFirst().iamRoleArn());
+ assertEquals(first, client.modifyClusterIamRoles(request -> request.clusterIdentifier(clusterId)
+ .defaultIamRoleArn(first)).cluster().defaultIamRoleArn());
+ assertEquals(first, client.describeClusters(request -> request.clusterIdentifier(clusterId))
+ .clusters().getFirst().defaultIamRoleArn());
+
+ RedshiftException error = assertThrows(RedshiftException.class,
+ () -> client.modifyClusterIamRoles(request -> request.clusterIdentifier(clusterId)
+ .defaultIamRoleArn(second)));
+ assertEquals("InvalidParameterValue", error.awsErrorDetails().errorCode());
+
+ Cluster updated = client.modifyClusterIamRoles(request -> request.clusterIdentifier(clusterId)
+ .addIamRoles(second).defaultIamRoleArn(second)).cluster();
+ assertEquals(second, updated.defaultIamRoleArn());
+
+ Cluster cleared = client.modifyClusterIamRoles(request -> request.clusterIdentifier(clusterId)
+ .defaultIamRoleArn("")).cluster();
+ assertNull(cleared.defaultIamRoleArn());
+ assertEquals(2, cleared.iamRoles().size());
+ client.modifyClusterIamRoles(request -> request.clusterIdentifier(clusterId).defaultIamRoleArn(second));
+
+ Cluster removed = client.modifyClusterIamRoles(request -> request.clusterIdentifier(clusterId)
+ .removeIamRoles(second)).cluster();
+ assertNull(removed.defaultIamRoleArn());
+ assertEquals(first, removed.iamRoles().getFirst().iamRoleArn());
+ assertNull(client.describeClusters(request -> request.clusterIdentifier(clusterId))
+ .clusters().getFirst().defaultIamRoleArn());
+ } finally {
+ client.deleteCluster(request -> request.clusterIdentifier(clusterId).skipFinalClusterSnapshot(true));
+ }
+ }
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIamRoleEnforcedIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIamRoleEnforcedIntegrationTest.java
index 1384cf795d..bb97bf50fa 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIamRoleEnforcedIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIamRoleEnforcedIntegrationTest.java
@@ -17,6 +17,7 @@ import java.nio.charset.StandardCharsets;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
+import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.time.Duration;
@@ -24,7 +25,9 @@ import java.util.List;
import java.util.Map;
import java.util.Objects;
+import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* Covers COPY IAM_ROLE authorization when {@code FLOCI_SERVICES_S3_ENFORCE_AUTH} is on, so the
@@ -76,7 +79,12 @@ class RedshiftInterceptorIamRoleEnforcedIntegrationTest {
}
private static Connection waitForConnection(Cluster cluster) throws SQLException {
- String url = "jdbc:postgresql://127.0.0.1:" + cluster.getEndpoint().getPort() + "/dev";
+ return waitForConnection(cluster, "extended");
+ }
+
+ private static Connection waitForConnection(Cluster cluster, String queryMode) throws SQLException {
+ String url = "jdbc:postgresql://127.0.0.1:" + cluster.getEndpoint().getPort()
+ + "/dev?preferQueryMode=" + queryMode;
return Awaitility.await()
.atMost(Duration.ofSeconds(30))
.pollInterval(Duration.ofMillis(500))
@@ -84,6 +92,67 @@ class RedshiftInterceptorIamRoleEnforcedIntegrationTest {
.until(() -> DriverManager.getConnection(url, "admin", "Secret123"), Objects::nonNull);
}
+ @Test
+ void defaultRoleCopyAndUnloadEnforcePermissionsInBothQueryModes() throws Exception {
+ clusterId = "it-default-iam-role";
+ String bucket = "redshift-default-iam-role";
+ String role = "arn:aws:iam::000000000000:role/DefaultTransferRole";
+ String denied = "arn:aws:iam::000000000000:role/DefaultDeniedRole";
+ s3.createBucket(bucket, "us-east-1");
+ s3.putObject(bucket, "input.txt", "1|alice\n".getBytes(StandardCharsets.UTF_8), "text/plain", Map.of());
+ iamService.createRole("DefaultTransferRole", "/", REDSHIFT_TRUST_POLICY, null, 0, null);
+ iamService.createRole("DefaultDeniedRole", "/", REDSHIFT_TRUST_POLICY, null, 0, null);
+ iamService.putRolePolicy("DefaultTransferRole", "S3", """
+ {"Version":"2012-10-17","Statement":[
+ {"Effect":"Allow","Action":"s3:*","Resource":"*"}
+ ]}
+ """);
+ Cluster cluster = service.createCluster(clusterId, "dc2.large", "admin", "Secret123", null, List.of(),
+ List.of(role, denied), role);
+ for (String queryMode : List.of("simple", "extended")) {
+ String table = "people_" + queryMode;
+ try (Connection connection = waitForConnection(cluster, queryMode);
+ Statement ddl = connection.createStatement()) {
+ ddl.execute("CREATE TABLE " + table + " (id int, name text)");
+ try (PreparedStatement copy = connection.prepareStatement(
+ "COPY " + table + " FROM 's3://" + bucket + "/input.txt' IAM_ROLE default")) {
+ copy.execute();
+ }
+ try (ResultSet rows = ddl.executeQuery("SELECT id, name FROM " + table)) {
+ assertTrue(rows.next());
+ assertEquals(1, rows.getInt(1));
+ assertEquals("alice", rows.getString(2));
+ }
+ try (PreparedStatement unload = connection.prepareStatement(
+ "UNLOAD ('SELECT id, name FROM " + table + "') TO 's3://" + bucket + "/"
+ + queryMode + "/' IAM_ROLE default")) {
+ unload.execute();
+ }
+ assertEquals(1, s3.listObjects(bucket, queryMode + "/", null, 100).size());
+ }
+ }
+ service.modifyClusterIamRoles(clusterId, List.of(), List.of(), denied);
+ for (String queryMode : List.of("simple", "extended")) {
+ try (Connection connection = waitForConnection(cluster, queryMode);
+ Statement statement = connection.createStatement()) {
+ SQLException error = assertThrows(SQLException.class, () -> statement.execute(
+ "COPY people_simple FROM 's3://" + bucket + "/input.txt' IAM_ROLE default"));
+ assertEquals("42501", error.getSQLState());
+ SQLException unloadError = assertThrows(SQLException.class, () -> statement.execute(
+ "UNLOAD ('SELECT 1') TO 's3://" + bucket + "/denied/' IAM_ROLE default"));
+ assertEquals("42501", unloadError.getSQLState());
+ }
+ }
+ service.modifyClusterIamRoles(clusterId, List.of(), List.of(denied));
+ try (Connection connection = waitForConnection(cluster);
+ Statement statement = connection.createStatement()) {
+ SQLException error = assertThrows(SQLException.class, () -> statement.execute(
+ "COPY people_simple FROM 's3://" + bucket + "/input.txt' IAM_ROLE default"));
+ assertEquals("42501", error.getSQLState());
+ assertTrue(error.getMessage().contains("No default IAM role"));
+ }
+ }
+
@Test
void copyWithAllowingRolePolicySucceeds() throws Exception {
clusterId = "it-copy-iam-role-allow";
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftQueryHandlerTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftQueryHandlerTest.java
index 2dce874427..43df55d056 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftQueryHandlerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftQueryHandlerTest.java
@@ -32,6 +32,46 @@ import static org.mockito.Mockito.*;
class RedshiftQueryHandlerTest {
+ @Test
+ void modifyClusterIamRolesParsesAndReturnsDefaultRole() {
+ String role = "arn:aws:iam::111111111111:role/default";
+ MultivaluedMap<String, String> params = new MultivaluedHashMap<>();
+ params.putSingle("ClusterIdentifier", "c1");
+ params.putSingle("AddIamRoles.IamRoleArn.1", role);
+ params.putSingle("DefaultIamRoleArn", role);
+ Cluster cluster = new Cluster();
+ cluster.setClusterIdentifier("c1");
+ cluster.setDefaultIamRoleArn(role);
+ when(service.modifyClusterIamRoles("c1", List.of(role), List.of(), role)).thenReturn(cluster);
+
+ Response response = handler.handle("ModifyClusterIamRoles", params);
+
+ verify(service).modifyClusterIamRoles("c1", List.of(role), List.of(), role);
+ assertTrue(response.getEntity().toString().contains("<DefaultIamRoleArn>" + role + "</DefaultIamRoleArn>"));
+ }
+
+ @Test
+ void createClusterPassesDefaultRoleToService() {
+ String role = "arn:aws:iam::111111111111:role/default";
+ MultivaluedMap<String, String> params = new MultivaluedHashMap<>();
+ params.putSingle("ClusterIdentifier", "c1");
+ params.putSingle("NodeType", "dc2.large");
+ params.putSingle("MasterUsername", "admin");
+ params.putSingle("MasterUserPassword", "pw");
+ params.putSingle("IamRoles.IamRoleArn.1", role);
+ params.putSingle("DefaultIamRoleArn", role);
+ Cluster cluster = new Cluster();
+ cluster.setClusterIdentifier("c1");
+ cluster.setDefaultIamRoleArn(role);
+ when(service.createCluster("c1", "dc2.large", "admin", "pw", null, List.of(), List.of(role), role))
+ .thenReturn(cluster);
+
+ Response response = handler.handle("CreateCluster", params);
+
+ assertTrue(response.getEntity().toString().contains("<DefaultIamRoleArn>" + role + "</DefaultIamRoleArn>"));
+ verify(service).createCluster("c1", "dc2.large", "admin", "pw", null, List.of(), List.of(role), role);
+ }
+
private RedshiftQueryHandler handler;
private RedshiftService service;
private RedshiftCredentialBroker credentialBroker;
@@ -288,7 +328,7 @@ class RedshiftQueryHandlerTest {
Cluster cluster = new Cluster();
cluster.setClusterIdentifier("test-cluster");
cluster.setClusterStatus("available");
- when(service.createCluster(any(), any(), any(), any(), any(), any())).thenReturn(cluster);
+ when(service.createCluster(any(), any(), any(), any(), any(), any(), any(), any())).thenReturn(cluster);
Response response = handler.handle("CreateCluster", params);
@@ -314,7 +354,7 @@ class RedshiftQueryHandlerTest {
cluster.setClusterStatus("available");
cluster.setMasterPasswordSecretArn("arn:aws:secretsmanager:us-east-1:acc:secret:redshift-managed");
cluster.setMasterPasswordSecretKmsKeyId("arn:aws:kms:us-east-1:acc:key/key-1");
- when(service.createClusterWithManagedMasterPassword(any(), any(), any(), any(), any(), any(), any(), any()))
+ when(service.createClusterWithManagedMasterPassword(any(), any(), any(), any(), any(), any(), any(), any(), any()))
.thenReturn(cluster);
Response response = handler.handle("CreateCluster", params,
@@ -327,7 +367,7 @@ class RedshiftQueryHandlerTest {
assertFalse(xml.contains("<MasterUserSecret>"));
verify(service).createClusterWithManagedMasterPassword(eq("managed-cluster"), eq("dc2.large"),
eq("admin"), isNull(), eq(List.of()), eq(List.of()),
- eq("arn:aws:kms:us-east-1:acc:key/key-1"), eq("us-east-1"));
+ eq("arn:aws:kms:us-east-1:acc:key/key-1"), eq("us-east-1"), isNull());
}
@Test
@@ -343,13 +383,13 @@ class RedshiftQueryHandlerTest {
Cluster cluster = new Cluster();
cluster.setClusterIdentifier("test-cluster");
cluster.setClusterStatus("available");
- when(service.createCluster(any(), any(), any(), any(), any(), any(), any())).thenReturn(cluster);
+ when(service.createCluster(any(), any(), any(), any(), any(), any(), any(), any())).thenReturn(cluster);
handler.handle("CreateCluster", params);
verify(service).createCluster(eq("test-cluster"), eq("dc2.large"), eq("admin"), eq("password123"),
isNull(), eq(List.of()), eq(List.of("arn:aws:iam::000000000000:role/first",
- "arn:aws:iam::000000000000:role/second")));
+ "arn:aws:iam::000000000000:role/second")), isNull());
}
@Test
@@ -392,7 +432,7 @@ class RedshiftQueryHandlerTest {
Cluster cluster = new Cluster();
cluster.setClusterIdentifier("test-cluster");
cluster.setClusterStatus("creating");
- when(service.createCluster(any(), any(), any(), any(), any(), any())).thenReturn(cluster);
+ when(service.createCluster(any(), any(), any(), any(), any(), any(), any(), any())).thenReturn(cluster);
Response response = handler.handle("CreateCluster", params);
String xml = (String) response.getEntity();
@@ -815,13 +855,13 @@ class RedshiftQueryHandlerTest {
Cluster cluster = new Cluster();
cluster.setClusterIdentifier("c1");
cluster.setIamRoleArns(List.of("arn:aws:iam::111111111111:role/a"));
- when(service.modifyClusterIamRoles(eq("c1"), any(), any())).thenReturn(cluster);
+ when(service.modifyClusterIamRoles(eq("c1"), any(), any(), isNull())).thenReturn(cluster);
Response response = handler.handle("ModifyClusterIamRoles", params);
assertEquals(200, response.getStatus());
verify(service).modifyClusterIamRoles("c1",
- List.of("arn:aws:iam::111111111111:role/a"), List.of("arn:aws:iam::111111111111:role/b"));
+ List.of("arn:aws:iam::111111111111:role/a"), List.of("arn:aws:iam::111111111111:role/b"), null);
String body = response.getEntity().toString();
assertTrue(body.contains("<ModifyClusterIamRolesResult>"));
assertTrue(body.contains("<IamRoleArn>arn:aws:iam::111111111111:role/a</IamRoleArn>"));
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftServiceTest.java
index af54fc1e5c..e8da3973c1 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftServiceTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftServiceTest.java
@@ -56,6 +56,87 @@ import static org.mockito.Mockito.*;
class RedshiftServiceTest {
+ @Test
+ void blankDefaultRoleCreatesWithoutDefaultAndClearsExistingDefault() {
+ String role = "arn:aws:iam::111111111111:role/copy";
+ when(cm.start(anyString(), anyString(), anyString(), anyString()))
+ .thenReturn(new RedshiftContainerHandle("container", "cluster", "localhost", 5432));
+ for (String blank : List.of("", " \t")) {
+ String identifier = blank.isEmpty() ? "empty-default" : "whitespace-default";
+ Cluster cluster = service.createCluster(identifier, "dc2.large", "admin", "pw",
+ null, List.of(), List.of(role), blank);
+ assertNull(cluster.getDefaultIamRoleArn());
+ when(clusterBackend.get(identifier)).thenReturn(Optional.of(cluster));
+
+ service.modifyClusterIamRoles(identifier, List.of(), List.of(), role);
+ service.modifyClusterIamRoles(identifier, List.of(), List.of());
+ assertEquals(role, cluster.getDefaultIamRoleArn());
+
+ service.modifyClusterIamRoles(identifier, List.of(), List.of(), blank);
+ assertNull(cluster.getDefaultIamRoleArn());
+ assertEquals(List.of(role), cluster.getIamRoleArns());
+ verify(proxyManager).updateIamRoles("111111111111:" + identifier, List.of(role), null);
+ }
+ }
+
+ @Test
+ void modifyDefaultIamRoleValidatesBeforeMutatingAndClearsWhenRemoved() throws Exception {
+ String first = "arn:aws:iam::111111111111:role/first";
+ String second = "arn:aws:iam::111111111111:role/second";
+ Cluster cluster = new Cluster();
+ cluster.setClusterIdentifier("c1");
+ cluster.setIamRoleArns(List.of(first));
+ when(clusterBackend.get("c1")).thenReturn(Optional.of(cluster));
+
+ service.modifyClusterIamRoles("c1", List.of(), List.of(), first);
+ assertEquals(first, cluster.getDefaultIamRoleArn());
+ verify(proxyManager).updateIamRoles("111111111111:c1", List.of(first), first);
+
+ AwsException missing = assertThrows(AwsException.class,
+ () -> service.modifyClusterIamRoles("c1", List.of(), List.of(first), second));
+ assertEquals("InvalidParameterValue", missing.getErrorCode());
+ assertEquals(List.of(first), cluster.getIamRoleArns());
+ assertEquals(first, cluster.getDefaultIamRoleArn());
+
+ service.modifyClusterIamRoles("c1", List.of(second), List.of(), second);
+ assertEquals(second, cluster.getDefaultIamRoleArn());
+ service.modifyClusterIamRoles("c1", List.of(), List.of(first));
+ assertEquals(second, cluster.getDefaultIamRoleArn());
+
+ ObjectMapper mapper = new ObjectMapper();
+ Cluster restored = mapper.readValue(mapper.writeValueAsBytes(cluster), Cluster.class);
+ assertEquals(second, restored.getDefaultIamRoleArn());
+ assertEquals(List.of(second), restored.getIamRoleArns());
+
+ service.modifyClusterIamRoles("c1", List.of(), List.of(second));
+ assertNull(cluster.getDefaultIamRoleArn());
+ verify(proxyManager).updateIamRoles("111111111111:c1", List.of(), null);
+ }
+
+ @Test
+ void createClusterRejectsUnassociatedOrMalformedDefaultRoleBeforeStartingContainer() {
+ String role = "arn:aws:iam::111111111111:role/copy";
+ assertThrows(AwsException.class, () -> service.createCluster("c1", "dc2.large", "admin", "pw",
+ null, List.of(), List.of(), role));
+ assertThrows(AwsException.class, () -> service.createCluster("c1", "dc2.large", "admin", "pw",
+ null, List.of(), List.of("invalid"), "invalid"));
+ verifyNoInteractions(cm);
+ verify(clusterBackend, never()).put(any(), any());
+ }
+
+ @Test
+ void createClusterStoresDefaultRoleAndConfiguresProxy() {
+ String role = "arn:aws:iam::111111111111:role/copy";
+ when(cm.start("111111111111", "c1", "admin", "pw"))
+ .thenReturn(new RedshiftContainerHandle("container", "c1", "localhost", 5432));
+ Cluster cluster = service.createCluster("c1", "dc2.large", "admin", "pw",
+ null, List.of(), List.of(role), role);
+ assertEquals(role, cluster.getDefaultIamRoleArn());
+ verify(proxyManager).startProxy(eq("111111111111:c1"), anyInt(), eq("localhost"), eq(5432),
+ anyString(), eq("admin"), eq("pw"), eq("dev"), any(), eq(List.of(role)), eq(role));
+ verify(clusterBackend, atLeastOnce()).flush();
+ }
+
private StorageFactory sf;
private AccountAwareStorageBackend<Cluster> clusterBackend;
private AccountAwareStorageBackend<Snapshot> snapshotBackend;
@@ -292,12 +373,16 @@ class RedshiftServiceTest {
clusterA.setMasterUsername("admin");
clusterA.setMasterPassword("pw-a");
clusterA.setClusterStatus("available");
+ clusterA.setDefaultIamRoleArn("arn:aws:iam::111111111111:role/default-a");
+ clusterA.setIamRoleArns(List.of(clusterA.getDefaultIamRoleArn()));
Cluster clusterB = new Cluster();
clusterB.setClusterIdentifier("cluster-b");
clusterB.setMasterUsername("admin");
clusterB.setMasterPassword("pw-b");
clusterB.setClusterStatus("available");
+ clusterB.setDefaultIamRoleArn("arn:aws:iam::222222222222:role/default-b");
+ clusterB.setIamRoleArns(List.of(clusterB.getDefaultIamRoleArn()));
when(clusterBackend.scanAllAccountEntries(any())).thenReturn(List.of(
new AccountAwareStorageBackend.AccountEntry<>("111111111111", "cluster-a", clusterA),
@@ -313,6 +398,10 @@ class RedshiftServiceTest {
// Cluster owned by a second, non-default account must also be recovered
verify(cm).adoptOrStart("111111111111", "cluster-a", "admin", "pw-a");
verify(cm).adoptOrStart("222222222222", "cluster-b", "admin", "pw-b");
+ verify(proxyManager).startProxy(eq("111111111111:cluster-a"), anyInt(), anyString(), anyInt(), anyString(),
+ eq("admin"), eq("pw-a"), eq("dev"), any(), eq(clusterA.getIamRoleArns()), eq(clusterA.getDefaultIamRoleArn()));
+ verify(proxyManager).startProxy(eq("222222222222:cluster-b"), anyInt(), anyString(), anyInt(), anyString(),
+ eq("admin"), eq("pw-b"), eq("dev"), any(), eq(clusterB.getIamRoleArns()), eq(clusterB.getDefaultIamRoleArn()));
verify(clusterBackend).putForAccount(eq("111111111111"), eq("cluster-a"), any(Cluster.class));
verify(clusterBackend).putForAccount(eq("222222222222"), eq("cluster-b"), any(Cluster.class));
verify(clusterBackend).flush();
@@ -378,7 +467,7 @@ class RedshiftServiceTest {
verify(proxyManager).startProxy(eq("111111111111:c1"), eq(7108),
eq("172.17.0.12"), eq(32830), eq("localhost"),
- eq("admin"), eq("Secret123"), eq("dev"), any(), any());
+ eq("admin"), eq("Secret123"), eq("dev"), any(), any(), isNull());
}
@Test
@@ -426,7 +515,7 @@ class RedshiftServiceTest {
verify(proxyManager).startProxy(eq("111111111111:c1"), eq(cluster.getProxyPort()),
eq("172.17.0.9"), eq(32800), eq("localhost"),
- eq("admin"), eq("Secret123"), eq("dev"), any(), any());
+ eq("admin"), eq("Secret123"), eq("dev"), any(), any(), isNull());
}
@Test
@@ -636,6 +725,8 @@ class RedshiftServiceTest {
void rebootClusterRestartsTheProxyOnTheSamePort() {
Cluster cluster = new Cluster();
cluster.setClusterIdentifier("c1");
+ cluster.setDefaultIamRoleArn("arn:aws:iam::111111111111:role/default");
+ cluster.setIamRoleArns(List.of(cluster.getDefaultIamRoleArn()));
cluster.setMasterUsername("admin");
cluster.setMasterPassword("Secret123");
cluster.setProxyPort(7107);
@@ -655,7 +746,7 @@ class RedshiftServiceTest {
verify(proxyManager).stopProxy("111111111111:c1");
verify(proxyManager).startProxy(eq("111111111111:c1"), eq(7107),
eq("172.17.0.11"), eq(32820), eq("localhost"),
- eq("admin"), eq("Secret123"), eq("dev"), any(), any());
+ eq("admin"), eq("Secret123"), eq("dev"), any(), eq(cluster.getIamRoleArns()), eq(cluster.getDefaultIamRoleArn()));
}
@Test
@@ -686,7 +777,7 @@ class RedshiftServiceTest {
// replacement container must be stopped too, once before the restart, once in
// rollback, so it is not left running behind a "failed" cluster.
verify(proxyManager).startProxy(eq("111111111111:c1"), eq(7107), any(), anyInt(),
- any(), any(), any(), any(), any(), any());
+ any(), any(), any(), any(), any(), any(), isNull());
verify(proxyManager, times(2)).stopProxy("111111111111:c1");
verify(cm, times(2)).stop("111111111111", "c1");
}
@@ -880,7 +971,7 @@ class RedshiftServiceTest {
assertEquals(List.of("arn:aws:iam::111111111111:role/keep", "arn:aws:iam::111111111111:role/new"),
updated.getIamRoleArns());
- verify(proxyManager).updateIamRoles("111111111111:c1", updated.getIamRoleArns());
+ verify(proxyManager).updateIamRoles("111111111111:c1", updated.getIamRoleArns(), null);
verify(clusterBackend).put(eq("c1"), any(Cluster.class));
verify(clusterBackend).flush();
}
@@ -931,7 +1022,7 @@ class RedshiftServiceTest {
assertEquals("InvalidParameterValue", ex.getErrorCode());
assertEquals(List.of("arn:aws:iam::111111111111:role/keep"), cluster.getIamRoleArns());
- verify(proxyManager, never()).updateIamRoles(any(), any());
+ verify(proxyManager, never()).updateIamRoles(any(), any(), any());
}
@Test
@@ -1289,7 +1380,7 @@ class RedshiftServiceTest {
assertEquals(restored.getEndpoint().getPort(), restored.getProxyPort());
verify(proxyManager).startProxy(eq("111111111111:restored"), anyInt(),
eq("172.17.0.10"), eq(32810), eq("localhost"),
- eq("admin"), eq("Secret123"), eq("dev"), any(), any());
+ eq("admin"), eq("Secret123"), eq("dev"), any(), any(), isNull());
}
@Test
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.java
index e9b5e08b26..6be1b619c5 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.java
@@ -6,12 +6,27 @@ import java.util.List;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
class CopyStatementParserTest {
+ @Test
+ void copyAndUnloadAcceptDefaultIamRole() {
+ CopyStatementParser.S3CopyFrom copy = assertInstanceOf(CopyStatementParser.S3CopyFrom.class,
+ CopyStatementParser.parse("COPY people FROM 's3://bucket/data' IAM_ROLE default CSV"));
+ assertNull(copy.iamRoleArn());
+ assertTrue(copy.defaultIamRoleRequested());
+ CopyStatementParser.S3Unload unload = assertInstanceOf(CopyStatementParser.S3Unload.class,
+ CopyStatementParser.parse("UNLOAD ('SELECT 1') TO 's3://bucket/out/' IAM_ROLE DEFAULT CSV"));
+ assertNull(unload.iamRoleArn());
+ assertTrue(unload.defaultIamRoleRequested());
+ assertNull(CopyStatementParser.parse("COPY people FROM 's3://bucket/data' IAM_ROLE defaultRole"));
+ assertNull(CopyStatementParser.parse("COPY people FROM 's3://bucket/data' IAM_ROLE default IAM_ROLE default"));
+ }
+
/** parse() returns S3Statement now; every COPY test wants the S3CopyFrom view. */
private static CopyStatementParser.S3CopyFrom copyFrom(String sql) {
CopyStatementParser.S3Statement s = CopyStatementParser.parse(sql);
@@ -99,13 +114,59 @@ class CopyStatementParserTest {
void copyWithoutIamRoleLeavesItNull() {
CopyStatementParser.S3CopyFrom c = copyFrom("COPY t FROM 's3://b/k'");
assertNull(c.iamRoleArn());
- }
-
- @Test
- void iamRoleDefaultKeywordIsStillUnsupported() {
- // Bare `default` (no quotes) does not match the quoted-ARN clause, so it falls through
- // to the existing catch-all and the whole statement is rejected (fail-open), same as today.
- assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' IAM_ROLE default"));
+ assertFalse(c.defaultIamRoleRequested());
+ }
+
+ @Test
+ void quotedDefaultRemainsLiteralRegardlessOfClusterDefault() {
+ for (String defaultRole : new String[]{null, "arn:aws:iam::111111111111:role/copy"}) {
+ for (String authorization : List.of("IAM_ROLE 'default'", "IAM_ROLE \"default\"",
+ "CREDENTIALS 'aws_iam_role=default'")) {
+ CopyStatementParser.S3Statement copy = CopyStatementParser.parse(
+ "COPY people FROM 's3://bucket/data' " + authorization, defaultRole);
+ CopyStatementParser.S3Statement unload = CopyStatementParser.parse(
+ "UNLOAD ('SELECT 1') TO 's3://bucket/out/' " + authorization, defaultRole);
+ assertNotNull(copy);
+ assertNotNull(unload);
+ assertEquals("default", copy.iamRoleArn());
+ assertEquals("default", unload.iamRoleArn());
+ assertFalse(copy.defaultIamRoleRequested());
+ assertFalse(unload.defaultIamRoleRequested());
+ }
+ }
+ }
+
+ @Test
+ void defaultIamRoleResolvesWithoutChangingCopyOptionsOrExplicitRoles() {
+ String role = "arn:aws:iam::111111111111:role/copy";
+ CopyStatementParser.S3CopyFrom copy = assertInstanceOf(CopyStatementParser.S3CopyFrom.class,
+ CopyStatementParser.parse("COPY t FROM 's3://b/k' IAM_ROLE default CSV GZIP TRUNCATECOLUMNS", role));
+ assertEquals(role, copy.iamRoleArn());
+ assertTrue(copy.defaultIamRoleRequested());
+ assertTrue(copy.csv());
+ assertTrue(copy.gzip());
+ assertTrue(copy.transforms().truncateColumns());
+ CopyStatementParser.S3CopyFrom explicit = assertInstanceOf(CopyStatementParser.S3CopyFrom.class,
+ CopyStatementParser.parse("COPY t FROM 's3://b/k' IAM_ROLE 'other-role'", role));
+ assertEquals("other-role", explicit.iamRoleArn());
+ assertFalse(explicit.defaultIamRoleRequested());
+ assertNull(CopyStatementParser.parse("SELECT 1", role));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' IAM_ROLE default ACCESS_KEY_ID 'x' "
+ + "SECRET_ACCESS_KEY 'y'", role));
+ }
+
+ @Test
+ void defaultIamRoleResolvesWithoutChangingUnloadOptions() {
+ String role = "arn:aws:iam::111111111111:role/unload";
+ CopyStatementParser.S3Unload unload = assertInstanceOf(CopyStatementParser.S3Unload.class,
+ CopyStatementParser.parse("UNLOAD ('SELECT 1') TO 's3://b/k' IAM_ROLE default CSV GZIP HEADER MANIFEST", role));
+ assertEquals(role, unload.iamRoleArn());
+ assertTrue(unload.defaultIamRoleRequested());
+ assertEquals("SELECT 1", unload.selectQuery());
+ assertTrue(unload.csv());
+ assertTrue(unload.gzip());
+ assertTrue(unload.header());
+ assertTrue(unload.manifest());
}
@Test
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManagerTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManagerTest.java
index 9ba61ad272..304f5def0d 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManagerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManagerTest.java
@@ -114,6 +114,33 @@ class RedshiftProxyManagerTest {
}
}
+ @Test
+ void defaultRoleAndAssociatedRolesAreInstalledTogetherBeforeAcceptingConnections() throws Exception {
+ RedshiftProxyManager manager = newManager();
+ String first = "arn:aws:iam::111111111111:role/first";
+ String second = "arn:aws:iam::111111111111:role/second";
+ int port = FreePorts.anyInterfacePort();
+ try {
+ manager.startProxy(KEY, port, "localhost", 1, "localhost", "admin", "pw", "dev",
+ null, List.of(first), first);
+ RedshiftAuthProxy proxy = registry(manager).get(KEY);
+ RedshiftAuthProxy.RoleConfiguration connected = proxy.roleConfiguration();
+ assertEquals(List.of(first), connected.iamRoleArns());
+ assertEquals(first, connected.defaultIamRoleArn());
+ assertThrows(UnsupportedOperationException.class, () -> connected.iamRoleArns().add(second));
+
+ manager.updateIamRoles(KEY, List.of(second), second);
+ RedshiftAuthProxy.RoleConfiguration nextConnection = proxy.roleConfiguration();
+ assertEquals(List.of(second), nextConnection.iamRoleArns());
+ assertEquals(second, nextConnection.defaultIamRoleArn());
+ assertEquals(List.of(first), connected.iamRoleArns());
+ assertEquals(first, connected.defaultIamRoleArn());
+ assertPortUnavailable(port);
+ } finally {
+ manager.stopAll();
+ }
+ }
+
@Test
void updateIamRolesForUnknownKeyIsANoOp() {
RedshiftProxyManager manager = newManager();
@@ -288,11 +315,8 @@ class RedshiftProxyManagerTest {
return (String) field.get(proxy);
}
- @SuppressWarnings("unchecked")
- private static List<String> iamRoleArns(RedshiftAuthProxy proxy) throws Exception {
- Field field = RedshiftAuthProxy.class.getDeclaredField("iamRoleArns");
- field.setAccessible(true);
- return (List<String>) field.get(proxy);
+ private static List<String> iamRoleArns(RedshiftAuthProxy proxy) {
+ return proxy.roleConfiguration().iamRoleArns();
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java
index 1007e1d5ad..298290c799 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java
@@ -53,6 +53,46 @@ import static org.mockito.Mockito.when;
class S3CopySimulatorTest {
+ @Test
+ void quotedDefaultIsValidatedAsAnArnForCopyAndUnload() {
+ IamService iamService = mock(IamService.class);
+ for (String defaultRole : new String[]{null, ROLE_ARN}) {
+ for (String authorization : List.of("IAM_ROLE 'default'", "IAM_ROLE \"default\"",
+ "CREDENTIALS 'aws_iam_role=default'")) {
+ CopyStatementParser.S3CopyFrom copy = (CopyStatementParser.S3CopyFrom) CopyStatementParser.parse(
+ "COPY people FROM 's3://bucket/data' " + authorization, defaultRole);
+ CopyStatementParser.S3Unload unload = (CopyStatementParser.S3Unload) CopyStatementParser.parse(
+ "UNLOAD ('SELECT 1') TO 's3://bucket/out/' " + authorization, defaultRole);
+ S3CopySimulator.S3TransferException copyError = assertThrows(S3CopySimulator.S3TransferException.class,
+ () -> S3CopySimulator.prepareCopy(copy, s3, iamService, "000000000000", List.of(ROLE_ARN)));
+ S3CopySimulator.S3TransferException unloadError = assertThrows(S3CopySimulator.S3TransferException.class,
+ () -> S3CopySimulator.prepareUnload(unload, s3, iamService, "000000000000", List.of(ROLE_ARN)));
+ assertEquals("42501", copyError.sqlState());
+ assertEquals("42501", unloadError.sqlState());
+ assertTrue(copyError.getMessage().contains("malformed ARN"), copyError.getMessage());
+ assertTrue(unloadError.getMessage().contains("malformed ARN"), unloadError.getMessage());
+ }
+ }
+ verifyNoInteractions(s3, iamService);
+ }
+
+ @Test
+ void missingDefaultRoleFailsCopyAndUnloadBeforeTouchingS3() {
+ CopyStatementParser.S3CopyFrom copy = (CopyStatementParser.S3CopyFrom) CopyStatementParser.parse(
+ "COPY people FROM 's3://bucket/data' IAM_ROLE default");
+ CopyStatementParser.S3Unload unload = (CopyStatementParser.S3Unload) CopyStatementParser.parse(
+ "UNLOAD ('SELECT 1') TO 's3://bucket/out/' IAM_ROLE default");
+ S3CopySimulator.S3TransferException copyError = assertThrows(S3CopySimulator.S3TransferException.class,
+ () -> S3CopySimulator.prepareCopy(copy, s3, null, "000000000000", List.of()));
+ S3CopySimulator.S3TransferException unloadError = assertThrows(S3CopySimulator.S3TransferException.class,
+ () -> S3CopySimulator.prepareUnload(unload, s3, null, "000000000000", List.of()));
+ assertEquals("42501", copyError.sqlState());
+ assertEquals("42501", unloadError.sqlState());
+ assertTrue(copyError.getMessage().contains("No default IAM role"));
+ assertTrue(unloadError.getMessage().contains("No default IAM role"));
+ verifyNoInteractions(s3);
+ }
+
private ServerSocket listener;
private Socket simClient; // simulator's "client" end
private Socket simBackend; // simulator's "backend" end