Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(redshift): support more COPY and UNLOAD options (#5204)

Vũ Thế Huy38ce99ba5401
diff --git a/docs/services/redshift.md b/docs/services/redshift.md
index 1a961ba134..5fdddb5526 100644
--- a/docs/services/redshift.md
+++ b/docs/services/redshift.md
@@ -365,10 +365,27 @@ order) through its own S3 service and streams the rows into the backing PostgreS
`FLOCI_SERVICES_S3_ENFORCE_AUTH` off, S3 policy checks are skipped. With it on, the role's
identity policy must allow the required S3 actions, and any bucket policy must not deny the
request. `IAM_ROLE default` is not supported.
+- `REGION '<region>'` is accepted and ignored: Floci has one S3.
+- `CREDENTIALS 'aws_iam_role=<arn>'` behaves like `IAM_ROLE`. `ACCESS_KEY_ID` with `SECRET_ACCESS_KEY`
+ (and an optional `SESSION_TOKEN`), or `CREDENTIALS 'aws_access_key_id=...;aws_secret_access_key=...'`,
+ are accepted but never verified or stored: S3 is read as an unsigned request, the same as with no
+ authorization clause. Combining key credentials with `IAM_ROLE`, or giving only one of the pair, is
+ not intercepted, and neither is a blank key, secret, token or role.
+- `EMPTYASNULL`, `BLANKSASNULL`, `REMOVEQUOTES` (not with `CSV` or `JSON`), `ACCEPTINVCHARS [AS 'c']`
+ (default `?`, one ASCII character) and `TRUNCATECOLUMNS` rewrite field content before loading. Field
+ level options need a single-byte `DELIMITER`, are not combined with `FORMAT AS JSON`, and normalise
+ `CRLF` line endings to `LF`. A nulled field is written as the `NULL AS` string, or `\N` in text
+ mode and an empty unquoted field in CSV mode. `TRUNCATECOLUMNS` cuts a value to the declared
+ `CHAR`/`VARCHAR` length in bytes (Redshift's meaning of the declared length, so `éééé` into
+ `VARCHAR(3)` loads `é`), never inside a multi-byte character or a backslash escape. Its column
+ lengths come from the database catalog, so it works only over the **Simple Query protocol** (`preferQueryMode=simple`);
+ over Extended Query the COPY fails with an error that names this requirement. A field-level option
+ buffers each decompressed object in memory, so an object over 64 MiB, or a heap budget shared
+ across connections that is used up, fails the COPY instead of exhausting the emulator.
- Any other clause (`FIXEDWIDTH`, `PARQUET`, `AVRO`, `ORC`, `MAXERROR`,
- `DATEFORMAT`, `TIMEFORMAT`, `REGION`, `ENCODING`, `ESCAPE`, `REMOVEQUOTES`, `BLANKSASNULL`,
- `EMPTYASNULL`, `TRUNCATECOLUMNS`, `ACCEPTINVCHARS`, `CREDENTIALS`, and so on) is not
- recognized: the statement is forwarded unchanged and PostgreSQL returns its own error.
+ `DATEFORMAT`, `TIMEFORMAT`, `ENCODING`, `ESCAPE`, `ENCRYPTED`, `MASTER_SYMMETRIC_KEY`,
+ `KMS_KEY_ID`, and so on) is not recognized: the statement is forwarded unchanged and PostgreSQL
+ returns its own error.
- A multi-statement query whose COPY is followed by another statement is not intercepted; send the
COPY on its own.
- Extended Query COPY is supported when the complete statement is present in `Parse` and has no
@@ -407,6 +424,32 @@ the result to S3 as one or more objects under `<prefix>`.
- A zero-row result still writes one object (empty, or the header row alone when
`HEADER` is set).
- `GZIP` compresses each object and appends `.gz` to its key.
+- `EXTENSION '<ext>'` is appended to each data object key, with a `.` inserted when the value does not
+ start with one: `EXTENSION 'csv'` and `EXTENSION '.csv'` both give `<prefix>0000_part_00.csv`. As in
+ AWS, the compression suffix is added only when no extension is given, so `EXTENSION 'txt.gz' GZIP`
+ gives `<prefix>0000_part_00.txt.gz`. The value is not validated; an empty value or one containing
+ `/` is not intercepted. AWS does not document whether it inserts the dot itself.
+- `REGION` and the credential clauses (`CREDENTIALS`, `ACCESS_KEY_ID`, `SECRET_ACCESS_KEY`,
+ `SESSION_TOKEN`) behave as described for COPY.
+- `ESCAPE` is accepted and relies on PostgreSQL text framing, which already escapes the backslash,
+ newline, carriage return and the delimiter. It is only valid with text framing: combined with
+ `CSV`, `ADDQUOTES` or `HEADER` (which switch the framing to CSV here) the statement is not
+ intercepted. Unlike Redshift, PostgreSQL also writes a tab as `\t`.
+- `CLEANPATH` deletes every object whose key starts with the target prefix. The objects are listed and
+ every delete is authorized up front, like the writes (with `FLOCI_SERVICES_S3_ENFORCE_AUTH` on, the
+ role needs list and delete permission, and a denial fails the UNLOAD before anything is removed or
+ written). The deletes themselves run when the first output arrives, so a query that PostgreSQL
+ rejects leaves the previous export in place. They run on a fresh listing of the prefix, authorized
+ and deleted under one bucket lock, so an object uploaded in the meantime does not survive. Objects
+ already deleted are not restored if the UNLOAD fails later. The match is a plain string prefix, so `TO 's3://b/sales_' CLEANPATH` also removes
+ `sales_archive/...`; end the prefix with `/` to limit it to one folder. It cannot be combined with
+ `ALLOWOVERWRITE`, and an empty prefix (the bucket root) is not intercepted so a typo cannot wipe a
+ bucket.
+- `ENCRYPTED AUTO` is accepted and writes objects as usual. `ENCRYPTED KMS_KEY_ID '<key>'` stores the
+ data and manifest objects with `aws:kms` server-side encryption and that key id; Floci does not
+ encrypt the data beyond what its S3 service does for those headers. Bare `ENCRYPTED` (server-side
+ encryption with the default S3 key) and the client-side `MASTER_SYMMETRIC_KEY` form are not
+ intercepted.
- `MANIFEST` writes `<prefix>manifest` listing every object with its
`content_length`.
- Without `ALLOWOVERWRITE`, a non-empty target prefix fails with SQL error XX000 and the select
@@ -419,8 +462,8 @@ the result to S3 as one or more objects under `<prefix>`.
`FLOCI_SERVICES_S3_ENFORCE_AUTH` off, S3 policy checks are skipped. With it on, the role's
identity policy must allow the required S3 actions, and any bucket policy must not deny the
request. `IAM_ROLE default` is not supported.
-- Any other option (`PARQUET`, `ENCRYPTED`, `REGION`, `CREDENTIALS`,
- `ZSTD`, `EXTENSION`, `CLEANPATH`, `PARTITION`, and so on) is not intercepted; the
+- Any other option (`PARQUET`, `JSON`, `FIXEDWIDTH`, `MASTER_SYMMETRIC_KEY`, `MANIFEST VERBOSE`,
+ `ZSTD`, `BZIP2`, `PARTITION`, and so on) is not intercepted; the
statement is forwarded and PostgreSQL reports its own error.
- Extended Query UNLOAD is supported when the complete statement is present in `Parse` and has no
bind parameters. Parameterized statements are forwarded unchanged.
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyRecordTransformer.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyRecordTransformer.java
new file mode 100644
index 0000000000..c8e559a5c3
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyRecordTransformer.java
@@ -0,0 +1,352 @@
+package io.github.hectorvent.floci.services.redshift.proxy;
+
+import java.io.ByteArrayOutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * Applies the COPY options that change field content to the bytes of one S3 object before they
+ * reach PostgreSQL's {@code COPY ... FROM STDIN}. The object is already fully in memory, so the
+ * transform works on a byte array: there are no chunk boundaries to straddle.
+ */
+final class CopyRecordTransformer {
+
+ private static final String SQLSTATE_INTERNAL = "XX000";
+ private static final byte NEWLINE = '\n';
+ private static final byte CARRIAGE_RETURN = '\r';
+ private static final byte BACKSLASH = '\\';
+ private static final byte DOUBLE_QUOTE = '"';
+ private static final byte SINGLE_QUOTE = '\'';
+
+ private record Field(int tokenEnd, byte[] value, boolean quoted) {
+ }
+
+ private final CopyStatementParser.CopyTransforms transforms;
+ private final boolean csv;
+ private final byte delimiter;
+ private final byte[] nullMarker;
+ private final List<Integer> columnMaxBytes;
+
+ CopyRecordTransformer(CopyStatementParser.S3CopyFrom spec, List<Integer> columnMaxBytes) {
+ this.transforms = spec.transforms();
+ this.csv = spec.csv();
+ String delimiterText = spec.delimiter() != null ? spec.delimiter() : (spec.csv() ? "," : "|");
+ this.delimiter = delimiterText.getBytes(StandardCharsets.UTF_8)[0];
+ String marker = spec.nullAs() != null ? spec.nullAs() : (spec.csv() ? "" : "\\N");
+ this.nullMarker = marker.getBytes(StandardCharsets.UTF_8);
+ this.columnMaxBytes = columnMaxBytes;
+ }
+
+ byte[] apply(byte[] input) {
+ byte[] data = transforms.invalidCharReplacement() != null
+ ? sanitizeUtf8(input, (byte) transforms.invalidCharReplacement().charValue())
+ : input;
+ if (!transforms.fieldLevel()) {
+ return data;
+ }
+ ByteArrayOutputStream out = new ByteArrayOutputStream(data.length + 16);
+ int position = 0;
+ while (position < data.length) {
+ position = transformRecord(data, position, out);
+ }
+ return out.toByteArray();
+ }
+
+ static byte[] sanitizeUtf8(byte[] input, byte replacement) {
+ ByteArrayOutputStream out = new ByteArrayOutputStream(input.length);
+ int i = 0;
+ while (i < input.length) {
+ int length = validSequenceLength(input, i);
+ if (length == 0) {
+ out.write(replacement);
+ i++;
+ } else {
+ out.write(input, i, length);
+ i += length;
+ }
+ }
+ return out.toByteArray();
+ }
+
+ private static int validSequenceLength(byte[] bytes, int index) {
+ int lead = bytes[index] & 0xFF;
+ if (lead < 0x80) {
+ return 1;
+ }
+ int length;
+ if (lead >= 0xC2 && lead <= 0xDF) {
+ length = 2;
+ } else if (lead >= 0xE0 && lead <= 0xEF) {
+ length = 3;
+ } else if (lead >= 0xF0 && lead <= 0xF4) {
+ length = 4;
+ } else {
+ return 0;
+ }
+ if (index + length > bytes.length) {
+ return 0;
+ }
+ for (int k = 1; k < length; k++) {
+ if ((bytes[index + k] & 0xC0) != 0x80) {
+ return 0;
+ }
+ }
+ int second = bytes[index + 1] & 0xFF;
+ boolean overlongOrOutOfRange = (lead == 0xE0 && second < 0xA0)
+ || (lead == 0xED && second > 0x9F)
+ || (lead == 0xF0 && second < 0x90)
+ || (lead == 0xF4 && second > 0x8F);
+ return overlongOrOutOfRange ? 0 : length;
+ }
+
+ private int transformRecord(byte[] data, int start, ByteArrayOutputStream out) {
+ int position = start;
+ int column = 0;
+ while (true) {
+ Field field = scanField(data, position);
+ writeField(field, column, out);
+ position = field.tokenEnd();
+ if (position >= data.length) {
+ out.write(NEWLINE);
+ return data.length;
+ }
+ if (data[position] == NEWLINE) {
+ out.write(NEWLINE);
+ return position + 1;
+ }
+ if (isCrlfAt(data, position)) {
+ out.write(NEWLINE);
+ return Math.min(position + 2, data.length);
+ }
+ if (data[position] != delimiter) {
+ throw malformed("unexpected character after closing quote");
+ }
+ out.write(delimiter);
+ position++;
+ column++;
+ }
+ }
+
+ private Field scanField(byte[] data, int start) {
+ int length = data.length;
+ if (start < length && transforms.removeQuotes()
+ && (data[start] == DOUBLE_QUOTE || data[start] == SINGLE_QUOTE)) {
+ byte quote = data[start];
+ for (int j = start + 1; j < length; j++) {
+ if (!csv && data[j] == BACKSLASH) {
+ // Text mode: a backslash escapes the next byte, so an escaped quote never closes the field.
+ j++;
+ continue;
+ }
+ boolean closes = data[j] == quote
+ && (j + 1 == length || data[j + 1] == delimiter || data[j + 1] == NEWLINE
+ || isCrlfAt(data, j + 1));
+ if (closes) {
+ return new Field(j + 1, Arrays.copyOfRange(data, start + 1, j), true);
+ }
+ }
+ throw malformed("quoted field has no closing quote");
+ }
+ if (start < length && csv && data[start] == DOUBLE_QUOTE) {
+ ByteArrayOutputStream value = new ByteArrayOutputStream();
+ int j = start + 1;
+ while (j < length) {
+ if (data[j] == DOUBLE_QUOTE) {
+ if (j + 1 < length && data[j + 1] == DOUBLE_QUOTE) {
+ value.write(DOUBLE_QUOTE);
+ j += 2;
+ continue;
+ }
+ return new Field(j + 1, value.toByteArray(), true);
+ }
+ value.write(data[j]);
+ j++;
+ }
+ throw malformed("quoted field has no closing quote");
+ }
+ int j = start;
+ while (j < length && data[j] != delimiter && data[j] != NEWLINE) {
+ if (!csv && data[j] == BACKSLASH && j + 1 < length) {
+ j++;
+ }
+ j++;
+ }
+ int valueEnd = j;
+ if (j < length && data[j] == NEWLINE && j > start && data[j - 1] == CARRIAGE_RETURN) {
+ valueEnd = j - 1;
+ }
+ return new Field(j, Arrays.copyOfRange(data, start, valueEnd), false);
+ }
+
+ /** A carriage return that ends the record: followed by a newline or by the end of the data. */
+ private static boolean isCrlfAt(byte[] data, int index) {
+ return index < data.length && data[index] == CARRIAGE_RETURN
+ && (index + 1 == data.length || data[index + 1] == NEWLINE);
+ }
+
+ private void writeField(Field field, int column, ByteArrayOutputStream out) {
+ byte[] value = field.value();
+ boolean nulled = (transforms.blanksAsNull() && isBlank(value))
+ || (transforms.emptyAsNull() && value.length == 0);
+ if (nulled) {
+ out.writeBytes(nullMarker);
+ return;
+ }
+ // A field that already is the NULL marker must not be cut: that would turn NULL into data.
+ byte[] limited = !field.quoted() && Arrays.equals(value, nullMarker) ? value : truncate(value, column);
+ if (!field.quoted()) {
+ out.writeBytes(limited);
+ } else if (csv) {
+ writeCsvQuoted(limited, out);
+ } else {
+ out.writeBytes(escapeTextField(limited));
+ }
+ }
+
+ private static boolean isBlank(byte[] value) {
+ if (value.length == 0) {
+ return false;
+ }
+ for (byte b : value) {
+ if (b != ' ' && b != '\t') {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ private byte[] truncate(byte[] value, int column) {
+ if (!transforms.truncateColumns() || columnMaxBytes == null || column >= columnMaxBytes.size()) {
+ return value;
+ }
+ Integer max = columnMaxBytes.get(column);
+ if (max == null) {
+ return value;
+ }
+ // Redshift counts CHAR(n) and VARCHAR(n) in bytes, so measure the decoded bytes and cut at a
+ // character boundary. Text mode decodes its escapes first so one is never split.
+ byte[] decoded = csv ? value : decodeTextEscapes(value);
+ int end = byteLimitBoundary(decoded, max);
+ if (end == decoded.length) {
+ return value;
+ }
+ byte[] kept = Arrays.copyOf(decoded, end);
+ return csv ? kept : encodeTextEscapes(kept);
+ }
+
+ /** Largest offset not above {@code maxBytes} that does not fall inside a multi-byte character. */
+ private static int byteLimitBoundary(byte[] bytes, int maxBytes) {
+ if (bytes.length <= maxBytes) {
+ return bytes.length;
+ }
+ int end = maxBytes;
+ while (end > 0 && (bytes[end] & 0xC0) == 0x80) {
+ end--;
+ }
+ return end;
+ }
+
+ /** Decodes COPY text escapes: backslash b f n r t v, octal NNN, hex xHH, and backslash c for any other c. */
+ private static byte[] decodeTextEscapes(byte[] value) {
+ ByteArrayOutputStream out = new ByteArrayOutputStream(value.length);
+ int i = 0;
+ while (i < value.length) {
+ byte b = value[i];
+ if (b != BACKSLASH || i + 1 >= value.length) {
+ out.write(b);
+ i++;
+ continue;
+ }
+ byte next = value[i + 1];
+ if (next >= '0' && next <= '7') {
+ int octal = 0;
+ int digits = 0;
+ i++;
+ while (i < value.length && digits < 3 && value[i] >= '0' && value[i] <= '7') {
+ octal = octal * 8 + (value[i] - '0');
+ i++;
+ digits++;
+ }
+ out.write(octal);
+ } else if (next == 'x' && i + 2 < value.length && Character.digit(value[i + 2], 16) >= 0) {
+ int hex = Character.digit(value[i + 2], 16);
+ i += 3;
+ if (i < value.length && Character.digit(value[i], 16) >= 0) {
+ hex = hex * 16 + Character.digit(value[i], 16);
+ i++;
+ }
+ out.write(hex);
+ } else {
+ out.write(switch (next) {
+ case 'b' -> '\b';
+ case 'f' -> '\f';
+ case 'n' -> '\n';
+ case 'r' -> '\r';
+ case 't' -> '\t';
+ case 'v' -> 0x0B;
+ default -> next;
+ });
+ i += 2;
+ }
+ }
+ return out.toByteArray();
+ }
+
+ private byte[] encodeTextEscapes(byte[] value) {
+ ByteArrayOutputStream out = new ByteArrayOutputStream(value.length + 4);
+ for (byte b : value) {
+ if (b == BACKSLASH || b == delimiter) {
+ out.write(BACKSLASH);
+ out.write(b);
+ } else if (b == NEWLINE) {
+ out.write(BACKSLASH);
+ out.write('n');
+ } else if (b == CARRIAGE_RETURN) {
+ out.write(BACKSLASH);
+ out.write('r');
+ } else {
+ out.write(b);
+ }
+ }
+ return out.toByteArray();
+ }
+
+ private byte[] escapeTextField(byte[] value) {
+ ByteArrayOutputStream escaped = new ByteArrayOutputStream(value.length + 4);
+ for (int i = 0; i < value.length; i++) {
+ byte b = value[i];
+ if (b == BACKSLASH && i + 1 < value.length) {
+ escaped.write(b);
+ escaped.write(value[++i]);
+ } else if (b == delimiter) {
+ escaped.write(BACKSLASH);
+ escaped.write(b);
+ } else if (b == NEWLINE) {
+ escaped.write(BACKSLASH);
+ escaped.write('n');
+ } else if (b == CARRIAGE_RETURN) {
+ escaped.write(BACKSLASH);
+ escaped.write('r');
+ } else {
+ escaped.write(b);
+ }
+ }
+ return escaped.toByteArray();
+ }
+
+ private static void writeCsvQuoted(byte[] value, ByteArrayOutputStream out) {
+ out.write(DOUBLE_QUOTE);
+ for (byte b : value) {
+ if (b == DOUBLE_QUOTE) {
+ out.write(DOUBLE_QUOTE);
+ }
+ out.write(b);
+ }
+ out.write(DOUBLE_QUOTE);
+ }
+
+ private static S3CopySimulator.S3TransferException malformed(String detail) {
+ return new S3CopySimulator.S3TransferException(SQLSTATE_INTERNAL, "COPY failed: " + detail, null);
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.java
index 22d0e82bb1..b30ac9ec3a 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.java
@@ -1,5 +1,6 @@
package io.github.hectorvent.floci.services.redshift.proxy;
+import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.List;
import java.util.regex.Matcher;
@@ -16,6 +17,25 @@ public final class CopyStatementParser {
public sealed interface S3Statement permits S3CopyFrom, S3Unload {
}
+ /** COPY options that change field content; {@link #NONE} keeps the plain byte-copy path. */
+ public record CopyTransforms(
+ boolean blanksAsNull,
+ boolean emptyAsNull,
+ boolean removeQuotes,
+ boolean truncateColumns,
+ Character invalidCharReplacement) {
+
+ public static final CopyTransforms NONE = new CopyTransforms(false, false, false, false, null);
+
+ public boolean fieldLevel() {
+ return blanksAsNull || emptyAsNull || removeQuotes || truncateColumns;
+ }
+
+ public boolean any() {
+ return fieldLevel() || invalidCharReplacement != null;
+ }
+ }
+
public record S3CopyFrom(
String targetTable,
List<String> columns,
@@ -29,7 +49,32 @@ public final class CopyStatementParser {
String iamRoleArn,
boolean jsonAuto,
boolean jsonAutoIgnoreCase,
- boolean manifest) implements S3Statement {
+ boolean manifest,
+ CopyTransforms transforms) implements S3Statement {
+
+ public S3CopyFrom {
+ if (transforms == null) {
+ transforms = CopyTransforms.NONE;
+ }
+ }
+
+ public S3CopyFrom(
+ String targetTable,
+ List<String> columns,
+ String bucket,
+ String keyOrPrefix,
+ String delimiter,
+ int headerLines,
+ boolean gzip,
+ boolean csv,
+ String nullAs,
+ String iamRoleArn,
+ boolean jsonAuto,
+ boolean jsonAutoIgnoreCase,
+ boolean manifest) {
+ this(targetTable, columns, bucket, keyOrPrefix, delimiter, headerLines, gzip, csv,
+ nullAs, iamRoleArn, jsonAuto, jsonAutoIgnoreCase, manifest, CopyTransforms.NONE);
+ }
public S3CopyFrom(
String targetTable,
@@ -78,7 +123,52 @@ public final class CopyStatementParser {
boolean allowOverwrite,
boolean parallel,
long maxFileSizeBytes,
- String iamRoleArn) implements S3Statement {
+ String iamRoleArn,
+ String extension,
+ boolean escape,
+ boolean cleanPath,
+ boolean encrypted,
+ String sseKmsKeyId) implements S3Statement {
+
+ public S3Unload(
+ String selectQuery,
+ String bucket,
+ String prefix,
+ String delimiter,
+ boolean header,
+ boolean gzip,
+ boolean csv,
+ boolean addQuotes,
+ String nullAs,
+ boolean manifest,
+ boolean allowOverwrite,
+ boolean parallel,
+ long maxFileSizeBytes,
+ String iamRoleArn,
+ String extension) {
+ this(selectQuery, bucket, prefix, delimiter, header, gzip, csv, addQuotes, nullAs, manifest,
+ allowOverwrite, parallel, maxFileSizeBytes, iamRoleArn, extension,
+ false, false, false, null);
+ }
+
+ public S3Unload(
+ String selectQuery,
+ String bucket,
+ String prefix,
+ String delimiter,
+ boolean header,
+ boolean gzip,
+ boolean csv,
+ boolean addQuotes,
+ String nullAs,
+ boolean manifest,
+ boolean allowOverwrite,
+ boolean parallel,
+ long maxFileSizeBytes,
+ String iamRoleArn) {
+ this(selectQuery, bucket, prefix, delimiter, header, gzip, csv, addQuotes, nullAs, manifest,
+ allowOverwrite, parallel, maxFileSizeBytes, iamRoleArn, null);
+ }
}
private static final Pattern COPY_PATTERN = Pattern.compile(
@@ -96,17 +186,23 @@ public final class CopyStatementParser {
private static final Pattern ADDQUOTES_PATTERN = Pattern.compile("(?i)\\bADDQUOTES\\b");
private static final Pattern MANIFEST_PATTERN = Pattern.compile("(?i)\\bMANIFEST\\b");
private static final Pattern ALLOWOVERWRITE_PATTERN = Pattern.compile("(?i)\\bALLOWOVERWRITE\\b");
+ private static final Pattern ESCAPE_PATTERN = Pattern.compile("(?i)\\bESCAPE\\b");
+ private static final Pattern CLEANPATH_PATTERN = Pattern.compile("(?i)\\bCLEANPATH\\b");
+ private static final Pattern ENCRYPTED_PATTERN = Pattern.compile("(?i)\\bENCRYPTED(?:\\s+(AUTO)\\b)?");
+ private static final Pattern KMS_KEY_ID_PATTERN = Pattern.compile(
+ "(?i)\\bKMS_KEY_ID\\s+(?:'((?:[^']|'')*)'|\"([^\"]*)\")");
+ private static final Pattern EXTENSION_PATTERN = Pattern.compile(
+ "(?i)\\bEXTENSION\\s+(?:'((?:[^']|'')*)'|\"([^\"]*)\")");
/**
* UNLOAD options this simulator cannot honour. Distinct from the COPY set:
- * MANIFEST / ALLOWOVERWRITE / PARALLEL / MAXFILESIZE are UNLOAD-supported here,
- * while EXTENSION / CLEANPATH / PARTITION are UNLOAD-only and unsupported.
+ * MANIFEST / ALLOWOVERWRITE / PARALLEL / MAXFILESIZE / EXTENSION / CLEANPATH are
+ * UNLOAD-supported here, while PARTITION is UNLOAD-only and unsupported.
*/
private static final Pattern UNLOAD_UNSUPPORTED_CLAUSE = Pattern.compile(
"(?i)\\b(FIXEDWIDTH|PARQUET|AVRO|ORC|JSON|SHAPEFILE|BZIP2|LZOP|ZSTD"
- + "|ENCRYPTED|ENCODING|REGION|CREDENTIALS|ACCESS_KEY_ID"
- + "|SECRET_ACCESS_KEY|SESSION_TOKEN|MASTER_SYMMETRIC_KEY|KMS_KEY_ID"
- + "|EXTENSION|CLEANPATH|PARTITION|MAXFILESIZE\\s+\\d+\\s*(?:TB|PB))\\b");
+ + "|ENCODING|MASTER_SYMMETRIC_KEY"
+ + "|PARTITION|MAXFILESIZE\\s+\\d+\\s*(?:TB|PB))\\b");
private static final Pattern QUALIFIED_NAME = Pattern.compile(
"(?:[A-Za-z_][A-Za-z0-9_$]*|\"[^\"]+\")(?:\\.(?:[A-Za-z_][A-Za-z0-9_$]*|\"[^\"]+\"))?");
@@ -118,6 +214,28 @@ public final class CopyStatementParser {
"(?i)\\bNULL\\s+(?:AS\\s+)?(?:'((?:[^']|'')*)'|\"([^\"]*)\")");
private static final Pattern IAM_ROLE_PATTERN = Pattern.compile(
"(?i)\\bIAM_ROLE\\s+(?:'((?:[^']|'')*)'|\"([^\"]*)\")");
+ private static final Pattern BLANKSASNULL_PATTERN = Pattern.compile("(?i)\\bBLANKSASNULL\\b");
+ private static final Pattern EMPTYASNULL_PATTERN = Pattern.compile("(?i)\\bEMPTYASNULL\\b");
+ private static final Pattern REMOVEQUOTES_PATTERN = Pattern.compile("(?i)\\bREMOVEQUOTES\\b");
+ private static final Pattern TRUNCATECOLUMNS_PATTERN = Pattern.compile("(?i)\\bTRUNCATECOLUMNS\\b");
+ private static final Pattern ACCEPTINVCHARS_PATTERN = Pattern.compile(
+ "(?i)\\bACCEPTINVCHARS(?:\\s+(?:AS\\s+)?'([ -&(-~])')?");
+ private static final Pattern CREDENTIALS_PATTERN = Pattern.compile(
+ "(?i)\\bCREDENTIALS\\s+(?:AS\\s+)?(?:'((?:[^']|'')*)'|\"([^\"]*)\")");
+ private static final Pattern ACCESS_KEY_ID_PATTERN = Pattern.compile(
+ "(?i)\\bACCESS_KEY_ID\\s+(?:AS\\s+)?(?:'((?:[^']|'')*)'|\"([^\"]*)\")");
+ private static final Pattern SECRET_ACCESS_KEY_PATTERN = Pattern.compile(
+ "(?i)\\bSECRET_ACCESS_KEY\\s+(?:AS\\s+)?(?:'((?:[^']|'')*)'|\"([^\"]*)\")");
+ private static final Pattern SESSION_TOKEN_PATTERN = Pattern.compile(
+ "(?i)\\bSESSION_TOKEN\\s+(?:AS\\s+)?(?:'((?:[^']|'')*)'|\"([^\"]*)\")");
+ private static final Pattern REGION_PATTERN = Pattern.compile(
+ "(?i)\\bREGION\\s+(?:AS\\s+)?(?:'((?:[^']|'')*)'|\"([^\"]*)\")");
+ private static final Pattern CREDENTIALS_ROLE = Pattern.compile(
+ "(?i)(?:^|;)\\s*aws_iam_role\\s*=\\s*([^;\\s]+)");
+ private static final Pattern CREDENTIALS_KEY_ID = Pattern.compile(
+ "(?i)(?:^|;)\\s*aws_access_key_id\\s*=\\s*[^;\\s]+");
+ private static final Pattern CREDENTIALS_SECRET = Pattern.compile(
+ "(?i)(?:^|;)\\s*aws_secret_access_key\\s*=\\s*[^;\\s]+");
private static final Pattern IGNOREHEADER_PATTERN = Pattern.compile(
"(?i)\\bIGNOREHEADER\\s+(?:AS\\s+)?(\\d+)\\b");
private static final Pattern HEADER_PATTERN = Pattern.compile("(?i)\\bHEADER\\b");
@@ -133,10 +251,10 @@ public final class CopyStatementParser {
*/
private static final Pattern UNSUPPORTED_CLAUSE = Pattern.compile(
"(?i)\\b(FIXEDWIDTH|PARQUET|AVRO|ORC|SHAPEFILE|BZIP2|LZOP|ZSTD|MAXERROR"
- + "|DATEFORMAT|TIMEFORMAT|ENCRYPTED|ENCODING|REGION|CREDENTIALS"
- + "|ACCESS_KEY_ID|SECRET_ACCESS_KEY|SESSION_TOKEN|MASTER_SYMMETRIC_KEY|KMS_KEY_ID"
- + "|ACCEPTINVCHARS|ACCEPTANYDATE|BLANKSASNULL|EMPTYASNULL|FILLRECORD|TRIMBLANKS"
- + "|TRUNCATECOLUMNS|IGNOREBLANKLINES|ESCAPE|REMOVEQUOTES|EXPLICIT_IDS|COMPUPDATE"
+ + "|DATEFORMAT|TIMEFORMAT|ENCRYPTED|ENCODING"
+ + "|MASTER_SYMMETRIC_KEY|KMS_KEY_ID"
+ + "|ACCEPTANYDATE|FILLRECORD|TRIMBLANKS"
+ + "|IGNOREBLANKLINES|ESCAPE|EXPLICIT_IDS|COMPUPDATE"
+ "|STATUPDATE|NOLOAD|ROUNDEC|QUOTE|SSH|READRATIO|COMPROWS|DIMENSION)\\b");
/** A {@code ;} followed by another statement: only a lone trailing {@code ;} is tolerated. */
@@ -192,8 +310,18 @@ public final class CopyStatementParser {
String nullAs = null;
String delimiter = null;
long maxFileSizeBytes = 0L;
- String iamRoleArn = null;
+ String extension = null;
+ boolean escape = false;
+ boolean cleanPath = false;
+ boolean encrypted = false;
+ boolean encryptedAuto = false;
+ String kmsKeyId = null;
+ boolean seenEscape = false;
+ boolean seenCleanPath = false;
+ boolean seenEncrypted = false;
+ boolean seenKmsKeyId = false;
+ boolean seenExtension = false;
boolean seenCsv = false;
boolean seenGzip = false;
boolean seenHeader = false;
@@ -204,7 +332,6 @@ public final class CopyStatementParser {
boolean seenNull = false;
boolean seenDelimiter = false;
boolean seenMaxFileSize = false;
- boolean seenIamRole = false;
Matcher csvM = CSV_PATTERN.matcher(options);
Matcher gzipM = GZIP_PATTERN.matcher(options);
@@ -216,8 +343,14 @@ public final class CopyStatementParser {
Matcher delimiterM = DELIMITER_PATTERN.matcher(options);
Matcher nullM = NULL_AS_PATTERN.matcher(options);
Matcher maxFileSizeM = MAXFILESIZE_PATTERN.matcher(options);
- Matcher iamRoleM = IAM_ROLE_PATTERN.matcher(options);
+ Matcher extensionM = EXTENSION_PATTERN.matcher(options);
+ Matcher escapeM = ESCAPE_PATTERN.matcher(options);
+ Matcher cleanPathM = CLEANPATH_PATTERN.matcher(options);
+ Matcher encryptedM = ENCRYPTED_PATTERN.matcher(options);
+ Matcher kmsKeyIdM = KMS_KEY_ID_PATTERN.matcher(options);
+ AuthClauses auth = new AuthClauses(options);
+ int next;
int offset = 0;
int len = options.length();
while (offset < len) {
@@ -227,7 +360,46 @@ public final class CopyStatementParser {
if (offset >= len) {
break;
}
- if (matchClause(maxFileSizeM, offset, len)) {
+ if (matchClause(extensionM, offset, len)) {
+ if (seenExtension) {
+ return null;
+ }
+ seenExtension = true;
+ extension = extractNullValue(extensionM);
+ if (extension.isEmpty() || extension.indexOf('/') >= 0) {
+ return null;
+ }
+ offset = extensionM.end();
+ } else if (matchClause(escapeM, offset, len)) {
+ if (seenEscape) {
+ return null;
+ }
+ seenEscape = true;
+ escape = true;
+ offset = escapeM.end();
+ } else if (matchClause(cleanPathM, offset, len)) {
+ if (seenCleanPath) {
+ return null;
+ }
+ seenCleanPath = true;
+ cleanPath = true;
+ offset = cleanPathM.end();
+ } else if (matchClause(encryptedM, offset, len)) {
+ if (seenEncrypted) {
+ return null;
+ }
+ seenEncrypted = true;
+ encrypted = true;
+ encryptedAuto = encryptedM.group(1) != null;
+ offset = encryptedM.end();
+ } else if (matchClause(kmsKeyIdM, offset, len)) {
+ if (seenKmsKeyId) {
+ return null;
+ }
+ seenKmsKeyId = true;
+ kmsKeyId = extractNullValue(kmsKeyIdM);
+ offset = kmsKeyIdM.end();
+ } else if (matchClause(maxFileSizeM, offset, len)) {
if (seenMaxFileSize) {
return null;
}
@@ -303,23 +475,41 @@ public final class CopyStatementParser {
seenNull = true;
nullAs = extractNullValue(nullM);
offset = nullM.end();
- } else if (matchClause(iamRoleM, offset, len)) {
- if (seenIamRole) {
+ } else if ((next = auth.consume(offset, len)) != AuthClauses.NO_MATCH) {
+ if (next == AuthClauses.REJECT) {
return null;
}
- seenIamRole = true;
- iamRoleArn = extractNullValue(iamRoleM);
- offset = iamRoleM.end();
+ offset = next;
} else {
return null;
}
}
+ if (!auth.consistent()) {
+ return null;
+ }
+ String iamRoleArn = auth.iamRoleArn();
if (delimiter == null) {
delimiter = csv ? "," : "|";
}
+ // ESCAPE relies on PostgreSQL text framing, which CSV, ADDQUOTES and HEADER all replace here.
+ if (escape && (csv || addQuotes || header)) {
+ return null;
+ }
+ // An empty prefix would make CLEANPATH delete the whole bucket.
+ if (cleanPath && (allowOverwrite || prefix.isEmpty())) {
+ return null;
+ }
+ // Only ENCRYPTED AUTO and ENCRYPTED KMS_KEY_ID are emulated. Bare ENCRYPTED and MASTER_SYMMETRIC_KEY are not.
+ if (kmsKeyId != null && (kmsKeyId.isBlank() || !encrypted || encryptedAuto)) {
+ return null;
+ }
+ if (encrypted && !encryptedAuto && kmsKeyId == null) {
+ return null;
+ }
return new S3Unload(select, bucket, prefix, delimiter, header, gzip, csv,
- addQuotes, nullAs, manifest, allowOverwrite, parallel, maxFileSizeBytes, iamRoleArn);
+ addQuotes, nullAs, manifest, allowOverwrite, parallel, maxFileSizeBytes, iamRoleArn, extension,
+ escape, cleanPath, encrypted, kmsKeyId);
}
private static S3CopyFrom parseCopy(String cleaned) {
@@ -372,8 +562,17 @@ public final class CopyStatementParser {
String nullAs = null;
String delimiter = null;
int headerLines = 0;
- String iamRoleArn = null;
+ boolean blanksAsNull = false;
+ boolean emptyAsNull = false;
+ boolean removeQuotes = false;
+ boolean truncateColumns = false;
+ Character invalidChar = null;
+ boolean seenTruncateColumns = false;
+ boolean seenBlanksAsNull = false;
+ boolean seenEmptyAsNull = false;
+ boolean seenRemoveQuotes = false;
+ boolean seenAcceptInvChars = false;
boolean seenCsv = false;
boolean seenGzip = false;
boolean seenJson = false;
@@ -381,7 +580,6 @@ public final class CopyStatementParser {
boolean seenNull = false;
boolean seenDelimiter = false;
boolean seenHeader = false;
- boolean seenIamRole = false;
Matcher csvMatcher = CSV_PATTERN.matcher(options);
Matcher gzipMatcher = GZIP_PATTERN.matcher(options);
@@ -391,8 +589,14 @@ public final class CopyStatementParser {
Matcher headerMatcher = HEADER_PATTERN.matcher(options);
Matcher delimiterMatcher = DELIMITER_PATTERN.matcher(options);
Matcher nullMatcher = NULL_AS_PATTERN.matcher(options);
- Matcher iamRoleMatcher = IAM_ROLE_PATTERN.matcher(options);
+ Matcher blanksAsNullMatcher = BLANKSASNULL_PATTERN.matcher(options);
+ Matcher emptyAsNullMatcher = EMPTYASNULL_PATTERN.matcher(options);
+ Matcher removeQuotesMatcher = REMOVEQUOTES_PATTERN.matcher(options);
+ Matcher truncateColumnsMatcher = TRUNCATECOLUMNS_PATTERN.matcher(options);
+ Matcher acceptInvCharsMatcher = ACCEPTINVCHARS_PATTERN.matcher(options);
+ AuthClauses auth = new AuthClauses(options);
+ int next;
int offset = 0;
int len = options.length();
while (offset < len) {
@@ -460,24 +664,203 @@ public final class CopyStatementParser {
seenNull = true;
nullAs = extractNullValue(nullMatcher);
offset = nullMatcher.end();
- } else if (matchClause(iamRoleMatcher, offset, len)) {
- if (seenIamRole) {
+ } else if (matchClause(blanksAsNullMatcher, offset, len)) {
+ if (seenBlanksAsNull) {
return null;
}
- seenIamRole = true;
- iamRoleArn = extractNullValue(iamRoleMatcher);
- offset = iamRoleMatcher.end();
+ seenBlanksAsNull = true;
+ blanksAsNull = true;
+ offset = blanksAsNullMatcher.end();
+ } else if (matchClause(emptyAsNullMatcher, offset, len)) {
+ if (seenEmptyAsNull) {
+ return null;
+ }
+ seenEmptyAsNull = true;
+ emptyAsNull = true;
+ offset = emptyAsNullMatcher.end();
+ } else if (matchClause(removeQuotesMatcher, offset, len)) {
+ if (seenRemoveQuotes) {
+ return null;
+ }
+ seenRemoveQuotes = true;
+ removeQuotes = true;
+ offset = removeQuotesMatcher.end();
+ } else if (matchClause(truncateColumnsMatcher, offset, len)) {
+ if (seenTruncateColumns) {
+ return null;
+ }
+ seenTruncateColumns = true;
+ truncateColumns = true;
+ offset = truncateColumnsMatcher.end();
+ } else if (matchClause(acceptInvCharsMatcher, offset, len)) {
+ if (seenAcceptInvChars) {
+ return null;
+ }
+ seenAcceptInvChars = true;
+ invalidChar = acceptInvCharsMatcher.group(1) != null
+ ? Character.valueOf(acceptInvCharsMatcher.group(1).charAt(0))
+ : Character.valueOf('?');
+ offset = acceptInvCharsMatcher.end();
+ } else if ((next = auth.consume(offset, len)) != AuthClauses.NO_MATCH) {
+ if (next == AuthClauses.REJECT) {
+ return null;
+ }
+ offset = next;
} else {
return null;
}
}
+ if (!auth.consistent()) {
+ return null;
+ }
+ String iamRoleArn = auth.iamRoleArn();
if (delimiter == null) {
delimiter = csv ? "," : "|";
}
+ CopyTransforms transforms = new CopyTransforms(
+ blanksAsNull, emptyAsNull, removeQuotes, truncateColumns, invalidChar);
+ if (transforms.any() && jsonAuto) {
+ return null;
+ }
+ if (transforms.removeQuotes() && csv) {
+ return null;
+ }
+ if (transforms.fieldLevel() && delimiter.getBytes(StandardCharsets.UTF_8).length != 1) {
+ return null;
+ }
+ if (invalidChar != null && replacementIsFramingSyntax(invalidChar, delimiter, csv)) {
+ return null;
+ }
+
return new S3CopyFrom(table, columns, bucket, keyOrPrefix, delimiter, headerLines, gzip, csv,
- nullAs, iamRoleArn, jsonAuto, jsonAutoIgnoreCase, manifest);
+ nullAs, iamRoleArn, jsonAuto, jsonAutoIgnoreCase, manifest, transforms);
+ }
+
+ /**
+ * Collects the S3 authorization and region clauses shared by COPY and UNLOAD. Key material is
+ * checked for shape only and never stored: the simulator reads S3 as the cluster role or as an
+ * unsigned request.
+ */
+ private static final class AuthClauses {
+ static final int NO_MATCH = -1;
+ static final int REJECT = -2;
+
+ private final Matcher iamRole;
+ private final Matcher credentials;
+ private final Matcher accessKeyId;
+ private final Matcher secretAccessKey;
+ private final Matcher sessionToken;
+ private final Matcher region;
+
+ private String iamRoleArn;
+ private String credentialsRoleArn;
+ private boolean seenIamRole;
+ private boolean seenCredentials;
+ private boolean seenAccessKeyId;
+ private boolean seenSecretAccessKey;
+ private boolean seenSessionToken;
+ private boolean seenRegion;
+ private boolean credentialsHaveKeys;
+ private boolean credentialsMalformed;
+
+ AuthClauses(String options) {
+ iamRole = IAM_ROLE_PATTERN.matcher(options);
+ credentials = CREDENTIALS_PATTERN.matcher(options);
+ accessKeyId = ACCESS_KEY_ID_PATTERN.matcher(options);
+ secretAccessKey = SECRET_ACCESS_KEY_PATTERN.matcher(options);
+ sessionToken = SESSION_TOKEN_PATTERN.matcher(options);
+ region = REGION_PATTERN.matcher(options);
+ }
+
+ int consume(int offset, int end) {
+ if (matchClause(iamRole, offset, end)) {
+ if (seenIamRole) {
+ return REJECT;
+ }
+ seenIamRole = true;
+ iamRoleArn = extractNullValue(iamRole);
+ return isBlank(iamRoleArn) ? REJECT : iamRole.end();
+ }
+ if (matchClause(credentials, offset, end)) {
+ if (seenCredentials) {
+ return REJECT;
+ }
+ seenCredentials = true;
+ readCredentials(extractNullValue(credentials));
+ return credentials.end();
+ }
+ if (matchClause(accessKeyId, offset, end)) {
+ if (seenAccessKeyId) {
+ return REJECT;
+ }
+ seenAccessKeyId = true;
+ return isBlank(extractNullValue(accessKeyId)) ? REJECT : accessKeyId.end();
+ }
+ if (matchClause(secretAccessKey, offset, end)) {
+ if (seenSecretAccessKey) {
+ return REJECT;
+ }
+ seenSecretAccessKey = true;
+ return isBlank(extractNullValue(secretAccessKey)) ? REJECT : secretAccessKey.end();
+ }
+ if (matchClause(sessionToken, offset, end)) {
+ if (seenSessionToken) {
+ return REJECT;
+ }
+ seenSessionToken = true;
+ return isBlank(extractNullValue(sessionToken)) ? REJECT : sessionToken.end();
+ }
+ if (matchClause(region, offset, end)) {
+ if (seenRegion) {
+ return REJECT;
+ }
+ seenRegion = true;
+ return region.end();
+ }
+ return NO_MATCH;
+ }
+
+ private void readCredentials(String value) {
+ Matcher role = CREDENTIALS_ROLE.matcher(value);
+ if (role.find()) {
+ credentialsRoleArn = role.group(1);
+ }
+ boolean hasId = CREDENTIALS_KEY_ID.matcher(value).find();
+ boolean hasSecret = CREDENTIALS_SECRET.matcher(value).find();
+ credentialsHaveKeys = hasId && hasSecret;
+ credentialsMalformed = hasId != hasSecret;
+ }
+
+ boolean consistent() {
+ if (credentialsMalformed || seenAccessKeyId != seenSecretAccessKey) {
+ return false;
+ }
+ if (seenSessionToken && !seenAccessKeyId) {
+ return false;
+ }
+ if (seenCredentials && seenAccessKeyId) {
+ return false;
+ }
+ if (seenCredentials && credentialsRoleArn == null && !credentialsHaveKeys) {
+ return false;
+ }
+ int roles = (seenIamRole ? 1 : 0) + (credentialsRoleArn != null ? 1 : 0);
+ boolean keys = seenAccessKeyId || credentialsHaveKeys;
+ return roles <= 1 && !(roles == 1 && keys);
+ }
+
+ String iamRoleArn() {
+ return iamRoleArn != null ? iamRoleArn : credentialsRoleArn;
+ }
+ }
+
+ /** The replacement is applied before records are scanned, so it must not change the framing. */
+ private static boolean replacementIsFramingSyntax(char replacement, String delimiter, boolean csv) {
+ return String.valueOf(replacement).equals(delimiter)
+ || (!csv && replacement == '\\')
+ || (csv && replacement == '"');
}
private static boolean matchClause(Matcher m, int start, int end) {
@@ -495,6 +878,10 @@ public final class CopyStatementParser {
return "\\t".equals(value) || "\t".equals(value) ? "\t" : value;
}
+ private static boolean isBlank(String value) {
+ return value == null || value.isBlank();
+ }
+
private static String extractNullValue(Matcher matcher) {
String value = matcher.group(1) != null ? matcher.group(1) : matcher.group(2);
return unescape(value);
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.java
index 5b2c00c3f8..01752a1dc2 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.java
@@ -3,6 +3,7 @@ package io.github.hectorvent.floci.services.redshift.proxy;
import io.github.hectorvent.floci.core.common.AwsException;
import io.github.hectorvent.floci.services.iam.IamService;
import io.github.hectorvent.floci.services.s3.S3Service;
+import io.github.hectorvent.floci.services.s3.model.PutObjectOptions;
import io.github.hectorvent.floci.services.s3.model.S3Object;
import org.jboss.logging.Logger;
@@ -14,7 +15,9 @@ import java.io.OutputStream;
import java.net.Socket;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
+import java.util.HashMap;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.concurrent.Semaphore;
import java.util.function.IntConsumer;
@@ -47,6 +50,11 @@ public final class S3CopySimulator {
static long UNLOAD_TARGET_FILE_BYTES = 6L * 1024 * 1024;
/** Whole-result ceiling; a larger UNLOAD is aborted rather than filling the in-memory S3 store. */
static long UNLOAD_MAX_TOTAL_BYTES = 256L * 1024 * 1024;
+ /** Decompressed size ceiling for one object when a COPY field-content option forces it into heap. */
+ static long COPY_TRANSFORM_MAX_OBJECT_BYTES = 64L * 1024 * 1024;
+ /** Shared across every connection: a transform holds the input plus about two further copies of it. */
+ static final Semaphore COPY_TRANSFORM_HEAP_MIB = new Semaphore(192);
+ private static final int COPY_TRANSFORM_HEAP_FACTOR = 3;
private static final long UNLOAD_WARN_BYTES = 32L * 1024 * 1024;
/** Shared across every connection so concurrent UNLOADs cannot multiply the per-slice heap cost without bound. */
static final Semaphore UNLOAD_HEAP_MIB = new Semaphore(192);
@@ -172,7 +180,13 @@ public final class S3CopySimulator {
}
static void streamCopyInput(CopyInput input, List<String> discoveredColumns, OutputStream backendOut) throws IOException {
- streamObjects(input.spec(), discoveredColumns, input.s3(), input.iamService(), input.roleSession(), input.keys(), backendOut);
+ streamCopyInput(input, discoveredColumns, null, backendOut);
+ }
+
+ static void streamCopyInput(CopyInput input, List<String> discoveredColumns,
+ List<Integer> columnMaxBytes, OutputStream backendOut) throws IOException {
+ streamObjects(input.spec(), discoveredColumns, columnMaxBytes, input.s3(), input.iamService(),
+ input.roleSession(), input.keys(), backendOut);
}
static void releaseCopySession(CopyInput input) {
@@ -194,6 +208,7 @@ public final class S3CopySimulator {
? RedshiftRoleAccess.resolveRoleSession(spec.iamRoleArn(), iamService, clusterAccountId, associatedRoleArns)
: null;
String probeKey = unloadDataKey(spec, 0);
+ boolean cleanPath = false;
try {
if (roleSession != null) {
RedshiftRoleAccess.authorizeRoleAction(s3, iamService, spec.iamRoleArn(), "s3:PutObject",
@@ -205,7 +220,7 @@ public final class S3CopySimulator {
RedshiftRoleAccess.objectArn(spec.iamRoleArn(), spec.bucket(), manifestKey));
s3.authorizeSignedPutObject(roleSession.accessKeyId(), roleSession.sessionToken(), spec.bucket(), manifestKey);
}
- if (!spec.allowOverwrite()) {
+ if (!spec.allowOverwrite() && !spec.cleanPath()) {
RedshiftRoleAccess.authorizeRoleAction(s3, iamService, spec.iamRoleArn(), "s3:ListBucket",
RedshiftRoleAccess.bucketArn(spec.iamRoleArn(), spec.bucket()));
s3.authorizeSignedListBucket(roleSession.accessKeyId(), roleSession.sessionToken(), spec.bucket());
@@ -220,7 +235,7 @@ public final class S3CopySimulator {
if (spec.manifest()) {
s3.authorizeAnonymousPutObject(spec.bucket(), spec.prefix() + "manifest");
}
- if (!spec.allowOverwrite()) {
+ if (!spec.allowOverwrite() && !spec.cleanPath()) {
s3.authorizeAnonymousListBucket(spec.bucket());
if (targetPrefixHasObjects(spec, s3)) {
throw new S3TransferException(SQLSTATE_INTERNAL,
@@ -229,6 +244,10 @@ public final class S3CopySimulator {
}
}
}
+ if (spec.cleanPath()) {
+ collectCleanPathKeys(spec, s3, iamService, roleSession);
+ cleanPath = true;
+ }
} catch (AwsException e) {
RedshiftRoleAccess.releaseRoleSession(roleSession, spec.iamRoleArn(), iamService);
throw new S3TransferException(unloadWriteSqlState(e), unloadWriteMessage(e, spec), e);
@@ -243,7 +262,7 @@ public final class S3CopySimulator {
"UNLOAD memory budget exhausted; retry shortly", null);
}
try {
- return new S3UnloadCollector(spec, s3, iamService, roleSession);
+ return new S3UnloadCollector(spec, s3, iamService, roleSession, cleanPath);
} catch (IOException e) {
UNLOAD_HEAP_MIB.release(UNLOAD_INITIAL_MIB);
RedshiftRoleAccess.releaseRoleSession(roleSession, spec.iamRoleArn(), iamService);
@@ -294,12 +313,22 @@ public final class S3CopySimulator {
}
try {
+ boolean noColumnList = spec.columns() == null || spec.columns().isEmpty();
+ boolean discoverForJson = spec.jsonAuto() && noColumnList;
+ boolean truncate = spec.transforms().truncateColumns();
List<String> discoveredColumns = null;
- if (spec.jsonAuto() && (spec.columns() == null || spec.columns().isEmpty())) {
- discoveredColumns = discoverTableColumns(client, backend, spec, txStatus, onStatusChange);
- if (discoveredColumns == null) {
+ List<Integer> columnMaxBytes = null;
+ if (discoverForJson || truncate) {
+ List<ColumnInfo> catalog = discoverColumnInfo(client, backend, spec, txStatus, onStatusChange);
+ if (catalog == null) {
return true;
}
+ if (discoverForJson) {
+ discoveredColumns = catalog.stream().map(ColumnInfo::name).toList();
+ }
+ if (truncate) {
+ columnMaxBytes = alignMaxBytes(spec, catalog);
+ }
}
OutputStream backendOut = backend.getOutputStream();
@@ -336,7 +365,7 @@ public final class S3CopySimulator {
// a CopyFail to the backend, whose ErrorResponse/ReadyForQuery is relayed to the client;
// or, if the backend is unreachable, one synthesized ErrorResponse/ReadyForQuery.
try {
- streamCopyInput(input, discoveredColumns, backendOut);
+ streamCopyInput(input, discoveredColumns, columnMaxBytes, backendOut);
writeCopyDone(backendOut);
drainToReadyForQuery(backendDecoder, client, onStatusChange);
} catch (RuntimeException | IOException e) {
@@ -431,8 +460,15 @@ public final class S3CopySimulator {
}
private static void streamObjects(CopyStatementParser.S3CopyFrom spec, List<String> discoveredColumns,
- S3Service s3, IamService iamService, RedshiftRoleAccess.RoleSession roleSession,
+ List<Integer> columnMaxBytes, S3Service s3, IamService iamService, RedshiftRoleAccess.RoleSession roleSession,
List<String> keys, OutputStream backendOut) throws IOException {
+ if (spec.transforms().truncateColumns() && columnMaxBytes == null) {
+ // Extended Query fixes the statement at Parse time, so there is no catalog round trip to
+ // learn column lengths; same limitation and remedy as FORMAT AS JSON 'auto'.
+ throw new S3TransferException(SQLSTATE_INTERNAL,
+ "COPY ... TRUNCATECOLUMNS needs catalog column lengths, which are only available over "
+ + "the Simple Query protocol; connect with preferQueryMode=simple", null);
+ }
byte[] buffer = new byte[CHUNK];
for (int i = 0; i < keys.size(); i++) {
if (roleSession != null) {
@@ -459,10 +495,14 @@ public final class S3CopySimulator {
if (i == 0 && spec.headerLines() > 0) {
skipLines(in, spec.headerLines());
}
+ InputStream source = in;
+ if (spec.transforms().any()) {
+ source = new ByteArrayInputStream(transformObject(spec, columnMaxBytes, in));
+ }
int read;
boolean endsWithNewline = false;
boolean hasData = false;
- while ((read = in.read(buffer)) != -1) {
+ while ((read = source.read(buffer)) != -1) {
if (read > 0) {
hasData = true;
endsWithNewline = (buffer[read - 1] == '\n');
@@ -478,10 +518,87 @@ public final class S3CopySimulator {
backendOut.flush();
}
- private static List<String> discoverTableColumns(Socket client, Socket backend,
- CopyStatementParser.S3CopyFrom spec,
- char txStatus, IntConsumer onStatusChange) throws IOException {
- String query = "SELECT a.attname FROM pg_catalog.pg_attribute a "
+ /**
+ * Reads one decompressed object into heap and transforms it. The read is bounded by a per-object
+ * ceiling and by a budget shared across connections, so a large or highly compressed object fails
+ * its own statement instead of exhausting the emulator heap.
+ */
+ private static byte[] transformObject(CopyStatementParser.S3CopyFrom spec, List<Integer> columnMaxBytes,
+ InputStream in) throws IOException {
+ int heldMib = 0;
+ try {
+ ByteArrayOutputStream buffered = new ByteArrayOutputStream();
+ byte[] chunk = new byte[CHUNK];
+ long total = 0;
+ int read;
+ while ((read = in.read(chunk)) != -1) {
+ total += read;
+ if (total > COPY_TRANSFORM_MAX_OBJECT_BYTES) {
+ throw new S3TransferException(SQLSTATE_PROGRAM_LIMIT_EXCEEDED,
+ "COPY object exceeds the " + COPY_TRANSFORM_MAX_OBJECT_BYTES
+ + "-byte limit for the field-level options "
+ + "(EMPTYASNULL, BLANKSASNULL, REMOVEQUOTES, ACCEPTINVCHARS, TRUNCATECOLUMNS)", null);
+ }
+ int wantedMib = (int) ((total * COPY_TRANSFORM_HEAP_FACTOR) / (1024 * 1024)) + 1;
+ while (heldMib < wantedMib) {
+ if (!COPY_TRANSFORM_HEAP_MIB.tryAcquire(1)) {
+ throw new S3TransferException(SQLSTATE_CONFIGURATION_LIMIT_EXCEEDED,
+ "COPY memory budget exhausted; retry shortly", null);
+ }
+ heldMib++;
+ }
+ buffered.write(chunk, 0, read);
+ }
+ return new CopyRecordTransformer(spec, columnMaxBytes).apply(buffered.toByteArray());
+ } finally {
+ COPY_TRANSFORM_HEAP_MIB.release(heldMib);
+ }
+ }
+
+ record ColumnInfo(String name, Integer maxBytes) {
+ }
+
+ /** Positional max character length per COPY column; null entries mean no limit. */
+ static List<Integer> alignMaxBytes(CopyStatementParser.S3CopyFrom spec, List<ColumnInfo> catalog) {
+ if (spec.columns() == null || spec.columns().isEmpty()) {
+ return catalog.stream().map(ColumnInfo::maxBytes).toList();
+ }
+ Map<String, Integer> byName = new HashMap<>();
+ for (ColumnInfo info : catalog) {
+ byName.put(info.name(), info.maxBytes());
+ }
+ List<Integer> aligned = new ArrayList<>();
+ for (String column : spec.columns()) {
+ String key = column.startsWith("\"") ? unquoteIdentifier(column) : column.toLowerCase(Locale.ROOT);
+ aligned.add(byName.get(key));
+ }
+ return aligned;
+ }
+
+ private static List<String> parseDataRow(byte[] body) {
+ int count = ((body[0] & 0xFF) << 8) | (body[1] & 0xFF);
+ List<String> values = new ArrayList<>(count);
+ int offset = 2;
+ for (int i = 0; i < count; i++) {
+ int length = ((body[offset] & 0xFF) << 24) | ((body[offset + 1] & 0xFF) << 16)
+ | ((body[offset + 2] & 0xFF) << 8) | (body[offset + 3] & 0xFF);
+ offset += 4;
+ if (length < 0) {
+ values.add(null);
+ continue;
+ }
+ values.add(new String(body, offset, length, StandardCharsets.UTF_8));
+ offset += length;
+ }
+ return values;
+ }
+
+ private static List<ColumnInfo> discoverColumnInfo(Socket client, Socket backend,
+ CopyStatementParser.S3CopyFrom spec,
+ char txStatus, IntConsumer onStatusChange) throws IOException {
+ // 1042 = bpchar, 1043 = varchar; atttypmod carries the declared length plus a 4-byte header.
+ String query = "SELECT a.attname, CASE WHEN a.atttypid IN (1042, 1043) AND a.atttypmod > 4 "
+ + "THEN a.atttypmod - 4 END FROM pg_catalog.pg_attribute a "
+ "WHERE a.attrelid = to_regclass('" + quoteLiteral(spec.targetTable()) + "') "
+ "AND a.attnum > 0 AND NOT a.attisdropped "
+ "ORDER BY a.attnum";
@@ -491,21 +608,17 @@ public final class S3CopySimulator {
backendOut.flush();
PostgresWireDecoder backendDecoder = new PostgresWireDecoder(backend.getInputStream());
- List<String> cols = new ArrayList<>();
+ List<ColumnInfo> cols = new ArrayList<>();
PostgresWireDecoder.FrontendMessage msg;
while ((msg = backendDecoder.nextMessage()) != null) {
char type = msg.type();
if (type == 'D') {
- byte[] body = msg.body();
- if (body.length >= 6) {
- int colCount = ((body[0] & 0xFF) << 8) | (body[1] & 0xFF);
- if (colCount >= 1) {
- int colLen = ((body[2] & 0xFF) << 24) | ((body[3] & 0xFF) << 16)
- | ((body[4] & 0xFF) << 8) | (body[5] & 0xFF);
- if (colLen > 0 && 6 + colLen <= body.length) {
- cols.add(new String(body, 6, colLen, StandardCharsets.UTF_8));
- }
- }
+ List<String> values = parseDataRow(msg.body());
+ if (!values.isEmpty() && values.get(0) != null) {
+ Integer maxBytes = values.size() >= 2 && values.get(1) != null
+ ? Integer.valueOf(values.get(1))
+ : null;
+ cols.add(new ColumnInfo(values.get(0), maxBytes));
}
} else if (type == 'E') {
forward(client, msg);
@@ -886,6 +999,80 @@ public final class S3CopySimulator {
return true;
}
+ /**
+ * Lists every object under the target prefix and authorizes the delete of each, so a denial fails
+ * the UNLOAD before anything is removed or written. It runs at preparation, to fail early, and
+ * again when the first output arrives, to delete what is there then. Nothing is deleted here, so
+ * a query the backend rejects, or a failed memory reservation, leaves the previous export in place.
+ */
+ private static List<String> collectCleanPathKeys(CopyStatementParser.S3Unload spec, S3Service s3,
+ IamService iamService,
+ RedshiftRoleAccess.RoleSession roleSession) {
+ authorizeCleanPathList(spec, s3, iamService, roleSession);
+ List<String> keys = new ArrayList<>();
+ String continuationToken = null;
+ do {
+ S3Service.ListObjectsResult result = s3.listObjectsWithPrefixes(
+ spec.bucket(), spec.prefix(), null, LIST_PAGE_SIZE, continuationToken, null);
+ if (result != null && result.objects() != null) {
+ for (S3Object object : result.objects()) {
+ keys.add(object.getKey());
+ }
+ }
+ continuationToken = (result != null && result.isTruncated()) ? result.nextContinuationToken() : null;
+ } while (continuationToken != null);
+
+ for (String key : keys) {
+ authorizeCleanPathDelete(spec, s3, iamService, roleSession, key);
+ }
+ return keys;
+ }
+
+ private static void authorizeCleanPathList(CopyStatementParser.S3Unload spec, S3Service s3,
+ IamService iamService,
+ RedshiftRoleAccess.RoleSession roleSession) {
+ if (roleSession != null) {
+ RedshiftRoleAccess.authorizeRoleAction(s3, iamService, spec.iamRoleArn(), "s3:ListBucket",
+ RedshiftRoleAccess.bucketArn(spec.iamRoleArn(), spec.bucket()));
+ s3.authorizeSignedListBucket(roleSession.accessKeyId(), roleSession.sessionToken(), spec.bucket());
+ } else {
+ s3.authorizeAnonymousListBucket(spec.bucket());
+ }
+ }
+
+ private static void authorizeCleanPathDelete(CopyStatementParser.S3Unload spec, S3Service s3,
+ IamService iamService,
+ RedshiftRoleAccess.RoleSession roleSession, String key) {
+ if (roleSession != null) {
+ RedshiftRoleAccess.authorizeRoleAction(s3, iamService, spec.iamRoleArn(), "s3:DeleteObject",
+ RedshiftRoleAccess.objectArn(spec.iamRoleArn(), spec.bucket(), key));
+ s3.authorizeSignedDeleteObject(roleSession.accessKeyId(), roleSession.sessionToken(),
+ spec.bucket(), key);
+ } else {
+ s3.authorizeAnonymousDeleteObject(spec.bucket(), key);
+ }
+ }
+
+ /** Server-side encryption headers requested by ENCRYPTED KMS_KEY_ID, or null when none apply. */
+ private static PutObjectOptions unloadObjectOptions(CopyStatementParser.S3Unload spec) {
+ if (spec.sseKmsKeyId() == null) {
+ return null;
+ }
+ return new PutObjectOptions()
+ .withServerSideEncryption("aws:kms")
+ .withSseKmsKeyId(spec.sseKmsKeyId());
+ }
+
+ private static void putUnloadObject(S3Service s3, CopyStatementParser.S3Unload spec, String key,
+ byte[] payload, String contentType) {
+ PutObjectOptions options = unloadObjectOptions(spec);
+ if (options == null) {
+ s3.putObject(spec.bucket(), key, payload, contentType, Map.of());
+ } else {
+ s3.putObject(spec.bucket(), key, payload, contentType, Map.of(), options);
+ }
+ }
+
private static boolean targetPrefixHasObjects(CopyStatementParser.S3Unload spec, S3Service s3) {
S3Service.ListObjectsResult r = s3.listObjectsWithPrefixes(
spec.bucket(), spec.prefix(), null, 1, null, null);
@@ -920,6 +1107,8 @@ public final class S3CopySimulator {
private final String contentType;
private final List<String> writtenKeys = new ArrayList<>();
private final List<Integer> writtenLengths = new ArrayList<>();
+ private final boolean cleanPathPending;
+ private boolean cleanPathDone;
private ByteArrayOutputStream sink = new ByteArrayOutputStream();
private OutputStream acc;
@@ -938,8 +1127,10 @@ public final class S3CopySimulator {
private int heldMib = UNLOAD_INITIAL_MIB;
private S3UnloadCollector(CopyStatementParser.S3Unload spec, S3Service s3,
- IamService iamService, RedshiftRoleAccess.RoleSession roleSession) throws IOException {
+ IamService iamService, RedshiftRoleAccess.RoleSession roleSession,
+ boolean cleanPath) throws IOException {
this.spec = spec;
+ this.cleanPathPending = cleanPath;
this.s3 = s3;
this.iamService = iamService;
this.roleSession = roleSession;
@@ -953,6 +1144,7 @@ public final class S3CopySimulator {
@Override
public void accept(byte[] body) throws IOException {
ensureOpen();
+ runCleanPath();
int dataStart = 0;
if (capturingHeader) {
int newline = -1;
@@ -1001,6 +1193,7 @@ public final class S3CopySimulator {
@Override
public void complete() throws IOException {
ensureOpen();
+ runCleanPath();
byte[] payload = finishSlice(acc, sink);
if (slicePayload > 0 || writtenKeys.isEmpty()) {
writePayload(payload, sliceIndex);
@@ -1015,10 +1208,10 @@ public final class S3CopySimulator {
} else {
s3.authorizeAnonymousPutObject(spec.bucket(), key);
}
- s3.putObject(spec.bucket(), key,
+ putUnloadObject(s3, spec, key,
manifestJson(spec.bucket(), writtenKeys, writtenLengths)
.getBytes(StandardCharsets.UTF_8),
- "application/json", Map.of());
+ "application/json");
} catch (RuntimeException e) {
fail(SQLSTATE_INTERNAL, "UNLOAD manifest write failed", e);
}
@@ -1052,6 +1245,25 @@ public final class S3CopySimulator {
closed = true;
}
+ /**
+ * CLEANPATH, second half: once, before the first write, removes what is under the prefix then.
+ * S3 lists, authorizes every delete and deletes under one bucket lock, so an object uploaded
+ * since preparation, or while this runs, does not survive next to the new export.
+ */
+ private void runCleanPath() {
+ if (cleanPathDone || !cleanPathPending) {
+ return;
+ }
+ cleanPathDone = true;
+ try {
+ authorizeCleanPathList(spec, s3, iamService, roleSession);
+ s3.deleteObjectsWithPrefix(spec.bucket(), spec.prefix(),
+ key -> authorizeCleanPathDelete(spec, s3, iamService, roleSession, key));
+ } catch (RuntimeException e) {
+ fail(SQLSTATE_INTERNAL, "UNLOAD CLEANPATH failed", e);
+ }
+ }
+
private void acquireForCurrentSlice() {
int wantedMib = (int) ((rawSlice * 3) / (1024 * 1024)) + 1;
while (heldMib < wantedMib) {
@@ -1079,7 +1291,7 @@ public final class S3CopySimulator {
} else {
s3.authorizeAnonymousPutObject(spec.bucket(), key);
}
- s3.putObject(spec.bucket(), key, payload, contentType, Map.of());
+ putUnloadObject(s3, spec, key, payload, contentType);
} catch (AwsException e) {
fail(unloadWriteSqlState(e), unloadWriteMessage(e, spec), e);
} catch (RuntimeException e) {
@@ -1134,6 +1346,11 @@ public final class S3CopySimulator {
String base = spec.parallel()
? spec.prefix() + String.format("%04d_part_00", index)
: spec.prefix() + String.format("%03d", index);
+ if (spec.extension() != null) {
+ // AWS adds the compression suffix only when no extension is given (its example is
+ // EXTENSION 'txt.gz' GZIP), so the caller's extension is the whole suffix.
+ return spec.extension().startsWith(".") ? base + spec.extension() : base + "." + spec.extension();
+ }
return spec.gzip() ? base + ".gz" : base;
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java b/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java
index bdae4fc76d..f64c627cd6 100644
--- a/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java
+++ b/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java
@@ -113,6 +113,7 @@ public class S3Service implements Resettable, ResourceProvider {
private static final Set<String> SUPPORTED_SERVER_SIDE_ENCRYPTION_VALUES = Set.of("AES256", "aws:kms", "aws:kms:dsse", "aws:fsx");
private static final String SSE_C_ALGORITHM = "AES256";
private static final int SSE_C_KEY_BYTES = 32;
+ private static final int DELETE_PREFIX_PAGE_SIZE = 1000;
@FunctionalInterface
interface LambdaInvoker {
@@ -1915,6 +1916,37 @@ public class S3Service implements Resettable, ResourceProvider {
}
}
+ /**
+ * Deletes every object whose key starts with {@code prefix}. The bucket monitor is held across the
+ * listing, the authorization of each key and the deletes, so a concurrent PutObject cannot land
+ * between them and survive. {@code authorizeDelete} runs for every key before the first delete, so
+ * a denial removes nothing.
+ */
+ public int deleteObjectsWithPrefix(String bucketName, String prefix, Consumer<String> authorizeDelete) {
+ Bucket bucket = bucketStore.get(bucketName)
+ .orElseThrow(() -> new AwsException("NoSuchBucket",
+ "The specified bucket does not exist.", 404));
+ synchronized (bucket) {
+ List<String> keys = new ArrayList<>();
+ String continuationToken = null;
+ do {
+ ListObjectsResult page = listObjectsWithPrefixes(
+ bucketName, prefix, null, DELETE_PREFIX_PAGE_SIZE, continuationToken, null);
+ for (S3Object object : page.objects()) {
+ keys.add(object.getKey());
+ }
+ continuationToken = page.isTruncated() ? page.nextContinuationToken() : null;
+ } while (continuationToken != null);
+ for (String key : keys) {
+ authorizeDelete.accept(key);
+ }
+ for (String key : keys) {
+ deleteObject(bucketName, key);
+ }
+ return keys.size();
+ }
+ }
+
// A current delete marker answers 404 NoSuchKey, the same as a missing key. That is deliberate:
// the S3 conditional-deletes guide says "If the latest version of the object is a delete marker,
// the object doesn't exist and the DeleteObject API will fail and return a 412 Precondition
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java
index 084bac753b..9ffeaadc7d 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java
@@ -186,6 +186,115 @@ class RedshiftInterceptorIntegrationTest {
}
}
+ @Test
+ void copyAcceptsRegionKeyCredentialsAndNullHandlingOptions() throws Exception {
+ String bucket = "redshift-copy-options";
+ s3.createBucket(bucket, "us-east-1");
+ s3.putObject(bucket, "people/p1.txt",
+ "1|alice\n2|\n3| \n".getBytes(StandardCharsets.UTF_8), "text/plain", Map.of());
+
+ try (Connection connection = waitForConnection(sharedCluster, "admin", "Secret123");
+ Statement statement = connection.createStatement()) {
+ statement.execute("CREATE TABLE copy_opts_people (id int, name text)");
+ statement.execute("COPY copy_opts_people FROM 's3://" + bucket + "/people/p1.txt' "
+ + "ACCESS_KEY_ID 'test' SECRET_ACCESS_KEY 'test' REGION 'us-east-1' "
+ + "EMPTYASNULL BLANKSASNULL");
+ try (ResultSet rows = statement.executeQuery(
+ "SELECT count(*) FROM copy_opts_people WHERE name IS NULL")) {
+ assertTrue(rows.next());
+ assertEquals(2, rows.getInt(1));
+ }
+ }
+ }
+
+ @Test
+ void copyTruncateColumnsCutsOverlongValuesOverSimpleQuery() throws Exception {
+ String bucket = "redshift-copy-truncate";
+ s3.createBucket(bucket, "us-east-1");
+ s3.putObject(bucket, "t.txt",
+ "1|abcdef\n".getBytes(StandardCharsets.UTF_8), "text/plain", Map.of());
+
+ try (Connection connection = waitForConnection(
+ simpleQueryJdbcUrl(sharedCluster), "admin", "Secret123");
+ Statement statement = connection.createStatement()) {
+ statement.execute("CREATE TABLE copy_trunc (id int, name varchar(3))");
+ statement.execute("COPY copy_trunc FROM 's3://" + bucket + "/t.txt' TRUNCATECOLUMNS");
+ try (ResultSet rows = statement.executeQuery("SELECT name FROM copy_trunc")) {
+ assertTrue(rows.next());
+ assertEquals("abc", rows.getString(1));
+ }
+ }
+ }
+
+ @Test
+ void copyTruncateColumnsOverExtendedQueryExplainsTheSimpleQueryRequirement() throws Exception {
+ String bucket = "redshift-copy-truncate-ext";
+ s3.createBucket(bucket, "us-east-1");
+ s3.putObject(bucket, "t.txt",
+ "1|abcdef\n".getBytes(StandardCharsets.UTF_8), "text/plain", Map.of());
+
+ try (Connection connection = waitForConnection(sharedCluster, "admin", "Secret123");
+ Statement statement = connection.createStatement()) {
+ statement.execute("CREATE TABLE copy_trunc_ext (id int, name varchar(3))");
+ SQLException failure = assertThrows(SQLException.class, () -> statement.execute(
+ "COPY copy_trunc_ext FROM 's3://" + bucket + "/t.txt' TRUNCATECOLUMNS"));
+ assertTrue(failure.getMessage().contains("preferQueryMode=simple"), failure.getMessage());
+ }
+ }
+
+ @Test
+ void unloadAppliesExtensionAndAcceptsRegion() throws Exception {
+ String bucket = "redshift-unload-extension";
+ s3.createBucket(bucket, "us-east-1");
+
+ try (Connection connection = waitForConnection(sharedCluster, "admin", "Secret123");
+ Statement statement = connection.createStatement()) {
+ statement.execute("UNLOAD ('select 1') TO 's3://" + bucket + "/ext/' "
+ + "REGION 'us-east-1' EXTENSION '.csv' ALLOWOVERWRITE");
+ }
+
+ List<S3Object> objects = s3.listObjects(bucket, "ext/", null, 100);
+ assertEquals(1, objects.size());
+ assertEquals("ext/0000_part_00.csv", objects.get(0).getKey());
+ }
+
+ @Test
+ void unloadCleanPathRemovesOnlyObjectsUnderThePrefix() throws Exception {
+ String bucket = "redshift-unload-cleanpath";
+ s3.createBucket(bucket, "us-east-1");
+ s3.putObject(bucket, "clean/old.txt", "old\n".getBytes(StandardCharsets.UTF_8), "text/plain", Map.of());
+ s3.putObject(bucket, "clean2/keep.txt", "keep\n".getBytes(StandardCharsets.UTF_8), "text/plain", Map.of());
+
+ try (Connection connection = waitForConnection(sharedCluster, "admin", "Secret123");
+ Statement statement = connection.createStatement()) {
+ statement.execute("UNLOAD ('select 1') TO 's3://" + bucket + "/clean/' CLEANPATH");
+ }
+
+ List<S3Object> cleaned = s3.listObjects(bucket, "clean/", null, 100);
+ assertEquals(1, cleaned.size());
+ assertEquals("clean/0000_part_00", cleaned.get(0).getKey());
+ assertTrue(s3.objectExists(bucket, "clean2/keep.txt"));
+ }
+
+ @Test
+ void unloadEscapeEscapesTheDelimiterInData() throws Exception {
+ String bucket = "redshift-unload-escape";
+ s3.createBucket(bucket, "us-east-1");
+
+ try (Connection connection = waitForConnection(sharedCluster, "admin", "Secret123");
+ Statement statement = connection.createStatement()) {
+ statement.execute("CREATE TABLE unload_escape (v text)");
+ statement.execute("INSERT INTO unload_escape VALUES ('a|b')");
+ statement.execute("UNLOAD ('select v from unload_escape') TO 's3://" + bucket
+ + "/esc/' ESCAPE ALLOWOVERWRITE");
+ }
+
+ List<S3Object> objects = s3.listObjects(bucket, "esc/", null, 100);
+ assertEquals(1, objects.size());
+ assertEquals("a\\|b\n", new String(
+ s3.getObject(bucket, objects.get(0).getKey()).getData(), StandardCharsets.UTF_8));
+ }
+
@Test
void namedPreparedCopyAndUnloadCanBeExecutedTwice() throws Exception {
Cluster cluster = sharedCluster;
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyRecordTransformerTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyRecordTransformerTest.java
new file mode 100644
index 0000000000..01cce72510
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyRecordTransformerTest.java
@@ -0,0 +1,178 @@
+package io.github.hectorvent.floci.services.redshift.proxy;
+
+import org.junit.jupiter.api.Test;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+class CopyRecordTransformerTest {
+
+ private static CopyStatementParser.CopyTransforms transforms(
+ boolean blanks, boolean empty, boolean removeQuotes, boolean truncate, Character invalid) {
+ return new CopyStatementParser.CopyTransforms(blanks, empty, removeQuotes, truncate, invalid);
+ }
+
+ private static String apply(String delimiter, boolean csv, String nullAs,
+ CopyStatementParser.CopyTransforms transforms, String input) {
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "t", List.of(), "b", "k", delimiter, 0, false, csv, nullAs, null,
+ false, false, false, transforms);
+ byte[] out = new CopyRecordTransformer(spec, null)
+ .apply(input.getBytes(StandardCharsets.UTF_8));
+ return new String(out, StandardCharsets.UTF_8);
+ }
+
+ @Test
+ void emptyAsNullReplacesEmptyTextFields() {
+ assertEquals("1|\\N|x\n2|b|\\N\n",
+ apply("|", false, null, transforms(false, true, false, false, null), "1||x\n2|b|\n"));
+ }
+
+ @Test
+ void blanksAsNullOnlyTouchesWhitespaceFields() {
+ assertEquals("1|\\N|x\n2||y\n",
+ apply("|", false, null, transforms(true, false, false, false, null), "1| |x\n2||y\n"));
+ }
+
+ @Test
+ void customNullStringIsUsedAsMarker() {
+ assertEquals("1|NULL|x\n",
+ apply("|", false, "NULL", transforms(false, true, false, false, null), "1||x\n"));
+ }
+
+ @Test
+ void removeQuotesKeepsDelimiterInsideQuotesAndEscapesItForPostgres() {
+ assertEquals("1|a\\|b|c\n",
+ apply("|", false, null, transforms(false, false, true, false, null), "1|\"a|b\"|c\n"));
+ assertEquals("x\n",
+ apply("|", false, null, transforms(false, false, true, false, null), "'x'\n"));
+ }
+
+ @Test
+ void removeQuotesWithoutClosingQuoteFails() {
+ assertThrows(S3CopySimulator.S3TransferException.class,
+ () -> apply("|", false, null, transforms(false, false, true, false, null), "1|\"abc\n"));
+ }
+
+ @Test
+ void csvQuotedEmptyFieldBecomesNullAndQuotedCommaSurvives() {
+ assertEquals("1,,x\n",
+ apply(",", true, null, transforms(false, true, false, false, null), "1,\"\",x\n"));
+ assertEquals("1,\"a,b\",\n",
+ apply(",", true, null, transforms(true, false, false, false, null), "1,\"a,b\", \n"));
+ }
+
+ @Test
+ void crlfLineEndingsAreNormalisedAndLastFieldIsNotPolluted() {
+ assertEquals("a|b\n1|\\N\n",
+ apply("|", false, null, transforms(false, true, false, false, null), "a|b\r\n1|\r\n"));
+ }
+
+ @Test
+ void lastRecordWithoutNewlineOrWithTrailingDelimiterIsTerminated() {
+ assertEquals("1|\\N\n",
+ apply("|", false, null, transforms(false, true, false, false, null), "1|"));
+ assertEquals("1|b\n",
+ apply("|", false, null, transforms(false, true, false, false, null), "1|b"));
+ }
+
+ @Test
+ void backslashEscapedDelimiterStaysInsideItsField() {
+ assertEquals("a\\|b|c\n",
+ apply("|", false, null, transforms(false, true, false, false, null), "a\\|b|c\n"));
+ }
+
+ @Test
+ void sanitizeUtf8ReplacesEachInvalidByteAndKeepsValidCharacters() {
+ byte[] input = {'a', (byte) 0xFF, 'b', (byte) 0xC3, (byte) 0x28, (byte) 0xC3, (byte) 0xA9};
+ assertEquals("a?b?(é",
+ new String(CopyRecordTransformer.sanitizeUtf8(input, (byte) '?'), StandardCharsets.UTF_8));
+ }
+
+ private static String applyWithLimits(String delimiter, boolean csv, List<Integer> limits, String input) {
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "t", List.of(), "b", "k", delimiter, 0, false, csv, null, null,
+ false, false, false, transforms(false, false, false, true, null));
+ byte[] out = new CopyRecordTransformer(spec, limits).apply(input.getBytes(StandardCharsets.UTF_8));
+ return new String(out, StandardCharsets.UTF_8);
+ }
+
+ @Test
+ void truncateColumnsCutsOnlyLimitedColumns() {
+ assertEquals("1|abc|xyzxyz\n",
+ applyWithLimits("|", false, Arrays.asList(null, 3, null), "1|abcdef|xyzxyz\n"));
+ }
+
+ @Test
+ void truncateColumnsNeverSplitsAMultiByteCharacter() {
+ // The limit counts bytes, so the 2-byte e-acute does not fit twice into 3 bytes.
+ assertEquals("a\n", applyWithLimits("|", false, List.of(2), "aébc\n"));
+ assertEquals("é\n", applyWithLimits("|", false, List.of(3), "éééé\n"));
+ assertEquals("éé\n", applyWithLimits("|", false, List.of(4), "ééé\n"));
+ }
+
+ @Test
+ void truncateColumnsLeavesAValueThatFitsItsByteLimit() {
+ assertEquals("éé\n", applyWithLimits("|", false, List.of(4), "éé\n"));
+ }
+
+ @Test
+ void truncateColumnsKeepsCsvQuoting() {
+ assertEquals("\"abc\",x\n", applyWithLimits(",", true, Arrays.asList(3, null), "\"abcdef\",x\n"));
+ }
+
+ @Test
+ void truncateColumnsLeavesTheNullMarkerIntact() {
+ assertEquals("\\N|x|y\n",
+ applyWithLimits("|", false, Arrays.asList(1, 10, 10), "\\N|x|y\n"));
+ }
+
+ @Test
+ void truncateColumnsKeepsAnEscapedBackslashWhole() {
+ // Decoded value is ab, backslash, c. The 3-byte cut keeps the backslash, re-escaped.
+ assertEquals("ab\\\\|hello\n",
+ applyWithLimits("|", false, Arrays.asList(3, null), "ab\\\\c|hello\n"));
+ }
+
+ @Test
+ void truncateColumnsCountsAnEscapedDelimiterAsOneByte() {
+ assertEquals("a\\|b|x\n", applyWithLimits("|", false, Arrays.asList(3, null), "a\\|b|x\n"));
+ }
+
+ @Test
+ void truncateColumnsNeverSplitsAnOctalEscape() {
+ // Backslash 101, 102 and 103 decode to A, B and C.
+ assertEquals("AB\n", applyWithLimits("|", false, List.of(2), "\\101\\102\\103\n"));
+ }
+
+ @Test
+ void removeQuotesDoesNotCloseOnAnEscapedQuoteBeforeTheDelimiter() {
+ assertEquals("a\\\"\\|b|x\n",
+ apply("|", false, null, transforms(false, false, true, false, null), "\"a\\\"|b\"|x\n"));
+ }
+
+ @Test
+ void removeQuotesRecognisesTheClosingQuoteBeforeCrlf() {
+ assertEquals("a\nb\n",
+ apply("|", false, null, transforms(false, false, true, false, null), "'a'\r\n'b'\n"));
+ }
+
+ @Test
+ void csvQuotedLastFieldBeforeCrlfIsAccepted() {
+ assertEquals("1,\"x\"\n",
+ apply(",", true, null, transforms(false, true, false, false, null), "1,\"x\"\r\n"));
+ }
+
+ @Test
+ void acceptInvalidCharsAloneDoesNotNeedFieldParsing() {
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "t", List.of(), "b", "k", "|", 0, false, false, null, null, false, false, false,
+ transforms(false, false, false, false, '#'));
+ byte[] out = new CopyRecordTransformer(spec, null).apply(new byte[]{'a', (byte) 0xFF, '\n'});
+ assertEquals("a#\n", new String(out, StandardCharsets.UTF_8));
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.java
index 75270541b0..e9b5e08b26 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.java
@@ -307,8 +307,8 @@ class CopyStatementParserTest {
void unloadRejectsUnsupportedOptions() {
assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/p/' PARQUET"));
assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/p/' ENCRYPTED"));
- assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/p/' REGION 'us-west-2'"));
- assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/p/' EXTENSION 'csv'"));
+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/p/' MASTER_SYMMETRIC_KEY 'k'"));
+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/p/' KMS_KEY_ID 'k'"));
assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/p/' ZSTD"));
assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/p/' PARTITION BY (dt)"));
}
@@ -327,6 +327,185 @@ class CopyStatementParserTest {
assertEquals("csv", u.nullAs());
}
+ @Test
+ void copyAcceptsAndIgnoresRegion() {
+ CopyStatementParser.S3CopyFrom c = copyFrom(
+ "COPY t FROM 's3://b/k' REGION 'eu-west-1'");
+ assertNotNull(c);
+ assertNull(c.iamRoleArn());
+ }
+
+ @Test
+ void copyMapsCredentialsIamRoleToIamRoleArn() {
+ CopyStatementParser.S3CopyFrom c = copyFrom(
+ "COPY t FROM 's3://b/k' CREDENTIALS 'aws_iam_role=arn:aws:iam::000000000000:role/r'");
+ assertEquals("arn:aws:iam::000000000000:role/r", c.iamRoleArn());
+ }
+
+ @Test
+ void copyAcceptsKeyCredentialsWithoutRole() {
+ CopyStatementParser.S3CopyFrom c = copyFrom(
+ "COPY t FROM 's3://b/k' ACCESS_KEY_ID 'test' SECRET_ACCESS_KEY 'test' SESSION_TOKEN 'test'");
+ assertNotNull(c);
+ assertNull(c.iamRoleArn());
+ }
+
+ @Test
+ void copyAcceptsKeyCredentialsInsideCredentialsString() {
+ CopyStatementParser.S3CopyFrom c = copyFrom(
+ "COPY t FROM 's3://b/k' CREDENTIALS 'aws_access_key_id=test;aws_secret_access_key=test'");
+ assertNotNull(c);
+ assertNull(c.iamRoleArn());
+ }
+
+ @Test
+ void copyRejectsConflictingOrIncompleteAuthorization() {
+ assertNull(CopyStatementParser.parse(
+ "COPY t FROM 's3://b/k' IAM_ROLE 'arn:aws:iam::000000000000:role/r' "
+ + "CREDENTIALS 'aws_iam_role=arn:aws:iam::000000000000:role/r2'"));
+ assertNull(CopyStatementParser.parse(
+ "COPY t FROM 's3://b/k' IAM_ROLE 'arn:aws:iam::000000000000:role/r' "
+ + "ACCESS_KEY_ID 'test' SECRET_ACCESS_KEY 'test'"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' ACCESS_KEY_ID 'test'"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' ACCESS_KEY_ID '' SECRET_ACCESS_KEY ''"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' ACCESS_KEY_ID 'test' SECRET_ACCESS_KEY ' '"));
+ assertNull(CopyStatementParser.parse(
+ "COPY t FROM 's3://b/k' ACCESS_KEY_ID 'test' SECRET_ACCESS_KEY 'test' SESSION_TOKEN ''"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' IAM_ROLE ''"));
+ assertNull(CopyStatementParser.parse(
+ "UNLOAD ('select 1') TO 's3://b/out/' ACCESS_KEY_ID '' SECRET_ACCESS_KEY ''"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' SESSION_TOKEN 'test'"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' REGION 'a' REGION 'b'"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' CREDENTIALS 'aws_access_key_id=test'"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' CREDENTIALS 'something_else=1'"));
+ }
+
+ @Test
+ void copyStillRejectsEncryptionClauses() {
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' ENCRYPTED"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' MASTER_SYMMETRIC_KEY 'k'"));
+ }
+
+ @Test
+ void unloadAcceptsRegionAndCredentials() {
+ CopyStatementParser.S3Unload u = unload(
+ "UNLOAD ('select 1') TO 's3://b/out/' REGION 'us-east-1' "
+ + "CREDENTIALS 'aws_iam_role=arn:aws:iam::000000000000:role/r'");
+ assertEquals("arn:aws:iam::000000000000:role/r", u.iamRoleArn());
+ assertNotNull(unload("UNLOAD ('select 1') TO 's3://b/out/' ACCESS_KEY_ID 'test' SECRET_ACCESS_KEY 'test'"));
+ }
+
+ @Test
+ void unloadParsesExtensionVerbatim() {
+ CopyStatementParser.S3Unload u = unload(
+ "UNLOAD ('select 1') TO 's3://b/out/' EXTENSION '.csv'");
+ assertEquals(".csv", u.extension());
+ assertNull(unload("UNLOAD ('select 1') TO 's3://b/out/'").extension());
+ }
+
+ @Test
+ void unloadRejectsUnsafeOrRepeatedExtension() {
+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' EXTENSION ''"));
+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' EXTENSION 'a/b'"));
+ assertNull(CopyStatementParser.parse(
+ "UNLOAD ('select 1') TO 's3://b/out/' EXTENSION '.a' EXTENSION '.b'"));
+ }
+
+ @Test
+ void copyParsesFieldContentOptions() {
+ CopyStatementParser.S3CopyFrom c = copyFrom(
+ "COPY t FROM 's3://b/k' EMPTYASNULL BLANKSASNULL REMOVEQUOTES ACCEPTINVCHARS AS '#'");
+ assertTrue(c.transforms().emptyAsNull());
+ assertTrue(c.transforms().blanksAsNull());
+ assertTrue(c.transforms().removeQuotes());
+ assertEquals('#', c.transforms().invalidCharReplacement());
+ }
+
+ @Test
+ void copyParsesTruncateColumns() {
+ assertTrue(copyFrom("COPY t FROM 's3://b/k' TRUNCATECOLUMNS").transforms().truncateColumns());
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' TRUNCATECOLUMNS TRUNCATECOLUMNS"));
+ assertNull(CopyStatementParser.parse(
+ "COPY t FROM 's3://b/k' FORMAT AS JSON 'auto' TRUNCATECOLUMNS"));
+ }
+
+ @Test
+ void acceptInvCharsDefaultsToQuestionMark() {
+ CopyStatementParser.S3CopyFrom c = copyFrom("COPY t FROM 's3://b/k' ACCEPTINVCHARS");
+ assertEquals('?', c.transforms().invalidCharReplacement());
+ assertEquals(CopyStatementParser.CopyTransforms.NONE, copyFrom("COPY t FROM 's3://b/k'").transforms());
+ }
+
+ @Test
+ void acceptInvCharsReplacementCannotBeFramingSyntax() {
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' ACCEPTINVCHARS AS '|'"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' ACCEPTINVCHARS AS '\\'"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' CSV ACCEPTINVCHARS AS '\"'"));
+ assertNotNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' ACCEPTINVCHARS AS '#'"));
+ }
+
+ @Test
+ void copyRejectsInvalidTransformCombinations() {
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' CSV REMOVEQUOTES"));
+ assertNull(CopyStatementParser.parse(
+ "COPY t FROM 's3://b/k' FORMAT AS JSON 'auto' EMPTYASNULL"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' DELIMITER 'é' EMPTYASNULL"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' EMPTYASNULL EMPTYASNULL"));
+ }
+
+ @Test
+ void unloadParsesEscapeOnlyWhenFramingStaysText() {
+ assertTrue(unload("UNLOAD ('select 1') TO 's3://b/out/' ESCAPE").escape());
+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' CSV ESCAPE"));
+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' ADDQUOTES ESCAPE"));
+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' HEADER ESCAPE"));
+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' ESCAPE ESCAPE"));
+ }
+
+ @Test
+ void unloadParsesCleanPathAndFailsOpenWhenItCouldBeUnsafe() {
+ assertTrue(unload("UNLOAD ('select 1') TO 's3://b/out/' CLEANPATH").cleanPath());
+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' CLEANPATH ALLOWOVERWRITE"));
+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/' CLEANPATH"));
+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' CLEANPATH CLEANPATH"));
+ }
+
+ @Test
+ void unloadParsesServerSideEncryptionOptions() {
+ CopyStatementParser.S3Unload kms = unload(
+ "UNLOAD ('select 1') TO 's3://b/out/' ENCRYPTED KMS_KEY_ID 'key-1'");
+ assertTrue(kms.encrypted());
+ assertEquals("key-1", kms.sseKmsKeyId());
+
+ CopyStatementParser.S3Unload reversed = unload(
+ "UNLOAD ('select 1') TO 's3://b/out/' KMS_KEY_ID 'key-2' ENCRYPTED");
+ assertEquals("key-2", reversed.sseKmsKeyId());
+
+ CopyStatementParser.S3Unload auto = unload("UNLOAD ('select 1') TO 's3://b/out/' ENCRYPTED AUTO");
+ assertTrue(auto.encrypted());
+ assertNull(auto.sseKmsKeyId());
+ assertFalse(unload("UNLOAD ('select 1') TO 's3://b/out/'").encrypted());
+ }
+
+ @Test
+ void unloadEncryptionFailsOpenWhenItWouldNeedClientSideKeys() {
+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' ENCRYPTED"));
+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' KMS_KEY_ID 'k'"));
+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' ENCRYPTED KMS_KEY_ID ' '"));
+ assertNull(CopyStatementParser.parse(
+ "UNLOAD ('select 1') TO 's3://b/out/' ENCRYPTED AUTO KMS_KEY_ID 'k'"));
+ assertNull(CopyStatementParser.parse(
+ "UNLOAD ('select 1') TO 's3://b/out/' ENCRYPTED MASTER_SYMMETRIC_KEY 'k'"));
+ }
+
+ @Test
+ void unloadRejectsConflictingAuthorization() {
+ assertNull(CopyStatementParser.parse(
+ "UNLOAD ('select 1') TO 's3://b/out/' IAM_ROLE 'arn:aws:iam::000000000000:role/r' "
+ + "ACCESS_KEY_ID 'test' SECRET_ACCESS_KEY 'test'"));
+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' REGION 'a' REGION 'b'"));
+ }
+
@Test
void parseCopy_jsonAuto_explicitColumns() {
CopyStatementParser.S3CopyFrom c = copyFrom("COPY users (id, name) FROM 's3://mybucket/users.json' "
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java
index aad137b3c7..eddd2a7da1 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java
@@ -4,10 +4,13 @@ import io.github.hectorvent.floci.core.common.AwsException;
import io.github.hectorvent.floci.services.iam.IamService;
import io.github.hectorvent.floci.services.iam.model.IamRole;
import io.github.hectorvent.floci.services.s3.S3Service;
+import io.github.hectorvent.floci.services.s3.model.PutObjectOptions;
import io.github.hectorvent.floci.services.s3.model.S3Object;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.mockito.ArgumentCaptor;
+import org.mockito.InOrder;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
@@ -18,11 +21,13 @@ import java.net.Socket;
import java.net.SocketTimeoutException;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Consumer;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;
@@ -36,7 +41,9 @@ import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.ArgumentMatchers.isNull;
+import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.inOrder;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
@@ -326,6 +333,152 @@ class S3CopySimulatorTest {
S3CopySimulator.unloadBackendSql(spec));
}
+ private CopyStatementParser.S3Unload unloadSpecWithExtension(boolean gzip, boolean manifest, String extension) {
+ return new CopyStatementParser.S3Unload("select a,b from t", "wh", "out/",
+ "|", false, gzip, false, false, null, manifest, true, true, 0, null, extension);
+ }
+
+ private CopyStatementParser.S3Unload unloadSpecWithOptions(
+ boolean manifest, boolean cleanPath, boolean encrypted, String kmsKeyId) {
+ return new CopyStatementParser.S3Unload("select a,b from t", "wh", "out/", "|", false, false, false,
+ false, null, manifest, false, true, 0, null, null, false, cleanPath, encrypted, kmsKeyId);
+ }
+
+ @Test
+ void cleanPathDeletesOnlyObjectsUnderThePrefixBeforeWriting() throws Exception {
+ S3Object old1 = new S3Object("wh", "out/old1", new byte[0], "text/plain");
+ S3Object old2 = new S3Object("wh", "out/old2", new byte[0], "text/plain");
+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("out/"), isNull(), anyInt(), any(), any()))
+ .thenReturn(new S3Service.ListObjectsResult(List.of(old1, old2), List.of(), false, null));
+ Map<String, byte[]> written = new ConcurrentHashMap<>();
+ when(s3.putObject(eq("wh"), any(), any(), any(), any())).thenAnswer(inv -> {
+ written.put(inv.getArgument(1), inv.getArgument(2));
+ return null;
+ });
+
+ Thread backend = backendThread(() -> playUnloadBackend("1|alice\n"));
+ S3CopySimulator.runUnload(simClient, simBackend, unloadSpecWithOptions(false, true, false, null),
+ s3, null, 'I');
+ joinBackend(backend);
+
+ verify(s3).deleteObjectsWithPrefix(eq("wh"), eq("out/"), any());
+ assertTrue(written.containsKey("out/0000_part_00"), written.keySet().toString());
+ }
+
+ @Test
+ void cleanPathDeletesNothingUntilOutputArrivesAndDeletesBeforeTheFirstWrite() throws Exception {
+ S3Object old1 = new S3Object("wh", "out/old1", new byte[0], "text/plain");
+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("out/"), isNull(), anyInt(), any(), any()))
+ .thenReturn(new S3Service.ListObjectsResult(List.of(old1), List.of(), false, null));
+
+ try (S3CopySimulator.UnloadCollector collector =
+ S3CopySimulator.prepareUnload(unloadSpecWithOptions(false, true, false, null), s3, null)) {
+ verify(s3, never()).deleteObjectsWithPrefix(any(), any(), any());
+ collector.complete();
+ }
+
+ InOrder order = inOrder(s3);
+ order.verify(s3).deleteObjectsWithPrefix(eq("wh"), eq("out/"), any());
+ order.verify(s3).putObject(eq("wh"), eq("out/0000_part_00"), any(), any(), any());
+ }
+
+ @Test
+ void cleanPathAuthorizesEachDeleteThroughTheCallbackItHandsToS3() throws Exception {
+ doAnswer(inv -> {
+ Consumer<String> authorize = inv.getArgument(2);
+ authorize.accept("out/late");
+ return 1;
+ }).when(s3).deleteObjectsWithPrefix(eq("wh"), eq("out/"), any());
+
+ try (S3CopySimulator.UnloadCollector collector =
+ S3CopySimulator.prepareUnload(unloadSpecWithOptions(false, true, false, null), s3, null)) {
+ collector.complete();
+ }
+
+ verify(s3).authorizeAnonymousDeleteObject("wh", "out/late");
+ }
+
+ @Test
+ void cleanPathKeepsObjectsWhenTheQueryNeverProducesOutput() throws Exception {
+ S3Object old1 = new S3Object("wh", "out/old1", new byte[0], "text/plain");
+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("out/"), isNull(), anyInt(), any(), any()))
+ .thenReturn(new S3Service.ListObjectsResult(List.of(old1), List.of(), false, null));
+
+ // Closed without complete(): the backend rejected the query, e.g. an unknown column.
+ try (S3CopySimulator.UnloadCollector collector =
+ S3CopySimulator.prepareUnload(unloadSpecWithOptions(false, true, false, null), s3, null)) {
+ assertNotNull(collector);
+ }
+
+ verify(s3, never()).deleteObjectsWithPrefix(any(), any(), any());
+ }
+
+ @Test
+ void cleanPathDeniedDeleteFailsBeforeAnythingIsWritten() {
+ S3Object old1 = new S3Object("wh", "out/old1", new byte[0], "text/plain");
+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("out/"), isNull(), anyInt(), any(), any()))
+ .thenReturn(new S3Service.ListObjectsResult(List.of(old1), List.of(), false, null));
+ doThrow(new AwsException("AccessDenied", "denied", 403))
+ .when(s3).authorizeAnonymousDeleteObject("wh", "out/old1");
+
+ assertThrows(S3CopySimulator.S3TransferException.class,
+ () -> S3CopySimulator.prepareUnload(unloadSpecWithOptions(false, true, false, null), s3, null));
+
+ verify(s3, never()).deleteObjectsWithPrefix(any(), any(), any());
+ verify(s3, never()).putObject(any(), any(), any(), any(), any());
+ }
+
+ @Test
+ void kmsKeyIdIsStoredOnDataAndManifestObjects() throws Exception {
+ ArgumentCaptor<PutObjectOptions> options = ArgumentCaptor.forClass(PutObjectOptions.class);
+ when(s3.putObject(eq("wh"), any(), any(), any(), any(), options.capture())).thenReturn(null);
+
+ Thread backend = backendThread(() -> playUnloadBackend("1|alice\n"));
+ S3CopySimulator.runUnload(simClient, simBackend, unloadSpecWithOptions(true, false, true, "key-1"),
+ s3, null, 'I');
+ joinBackend(backend);
+
+ assertEquals(2, options.getAllValues().size());
+ for (PutObjectOptions captured : options.getAllValues()) {
+ assertEquals("aws:kms", captured.getServerSideEncryption());
+ assertEquals("key-1", captured.getSseKmsKeyId());
+ }
+ }
+
+ @Test
+ void unloadAppendsExtensionToDataKeys() throws Exception {
+ Map<String, byte[]> written = new ConcurrentHashMap<>();
+ when(s3.putObject(eq("wh"), any(), any(), any(), any())).thenAnswer(inv -> {
+ written.put(inv.getArgument(1), inv.getArgument(2));
+ return null;
+ });
+
+ Thread backend = backendThread(() -> playUnloadBackend("1|alice\n"));
+ S3CopySimulator.runUnload(simClient, simBackend, unloadSpecWithExtension(false, false, ".csv"),
+ s3, null, 'I');
+ joinBackend(backend);
+
+ assertTrue(written.containsKey("out/0000_part_00.csv"), written.keySet().toString());
+ }
+
+ @Test
+ void unloadWithExtensionDoesNotAddAGzipSuffixAndListsTheNameInManifest() throws Exception {
+ Map<String, byte[]> written = new ConcurrentHashMap<>();
+ when(s3.putObject(eq("wh"), any(), any(), any(), any())).thenAnswer(inv -> {
+ written.put(inv.getArgument(1), inv.getArgument(2));
+ return null;
+ });
+
+ Thread backend = backendThread(() -> playUnloadBackend("1|alice\n"));
+ S3CopySimulator.runUnload(simClient, simBackend, unloadSpecWithExtension(true, true, "txt.gz"),
+ s3, null, 'I');
+ joinBackend(backend);
+
+ assertTrue(written.containsKey("out/0000_part_00.txt.gz"), written.keySet().toString());
+ String manifest = new String(written.get("out/manifest"), StandardCharsets.UTF_8);
+ assertTrue(manifest.contains("s3://wh/out/0000_part_00.txt.gz"), manifest);
+ }
+
@Test
void unloadWritesSingleObjectAndForwardsCommandComplete() throws Exception {
Map<String, byte[]> written = new ConcurrentHashMap<>();
@@ -1364,6 +1517,107 @@ class S3CopySimulatorTest {
assertEquals('d', rawBytes[0]);
}
+ private static String copyDataPayload(byte[] frames) {
+ ByteArrayOutputStream payload = new ByteArrayOutputStream();
+ int i = 0;
+ while (i < frames.length) {
+ int length = ((frames[i + 1] & 0xFF) << 24) | ((frames[i + 2] & 0xFF) << 16)
+ | ((frames[i + 3] & 0xFF) << 8) | (frames[i + 4] & 0xFF);
+ payload.write(frames, i + 5, length - 4);
+ i += 1 + length;
+ }
+ return payload.toString(StandardCharsets.UTF_8);
+ }
+
+ @Test
+ void streamCopyInput_appliesFieldTransforms() throws Exception {
+ CopyStatementParser.CopyTransforms transforms =
+ new CopyStatementParser.CopyTransforms(true, true, false, false, null);
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "people", List.of(), "wh", "p.txt", "|", 0, false, false, null, null,
+ false, false, false, transforms);
+ S3Object dataObj = new S3Object("wh", "p.txt",
+ "1| \n2||x\n".getBytes(StandardCharsets.UTF_8), "text/plain");
+ when(s3.getObject("wh", "p.txt")).thenReturn(dataObj);
+
+ S3CopySimulator.CopyInput input = new S3CopySimulator.CopyInput(spec, List.of("p.txt"), s3, null, null);
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ S3CopySimulator.streamCopyInput(input, out);
+
+ assertEquals("1|\\N\n2|\\N|x\n", copyDataPayload(out.toByteArray()));
+ }
+
+ @Test
+ void streamCopyInput_transformsAfterGzipDecodeAndHeaderSkip() throws Exception {
+ CopyStatementParser.CopyTransforms transforms =
+ new CopyStatementParser.CopyTransforms(false, true, false, false, null);
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "people", List.of(), "wh", "p.gz", "|", 1, true, false, null, null,
+ false, false, false, transforms);
+ ByteArrayOutputStream gz = new ByteArrayOutputStream();
+ try (GZIPOutputStream gzip = new GZIPOutputStream(gz)) {
+ gzip.write("id|name\n1|\n".getBytes(StandardCharsets.UTF_8));
+ }
+ when(s3.getObject("wh", "p.gz")).thenReturn(new S3Object("wh", "p.gz", gz.toByteArray(), "application/gzip"));
+
+ S3CopySimulator.CopyInput input = new S3CopySimulator.CopyInput(spec, List.of("p.gz"), s3, null, null);
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ S3CopySimulator.streamCopyInput(input, out);
+
+ assertEquals("1|\\N\n", copyDataPayload(out.toByteArray()));
+ }
+
+ @Test
+ void alignMaxBytes_followsExplicitColumnListAndFoldsUnquotedNames() {
+ List<S3CopySimulator.ColumnInfo> catalog = List.of(
+ new S3CopySimulator.ColumnInfo("id", null),
+ new S3CopySimulator.ColumnInfo("name", 3),
+ new S3CopySimulator.ColumnInfo("Note", 5));
+ CopyStatementParser.S3CopyFrom explicit = new CopyStatementParser.S3CopyFrom(
+ "t", List.of("NAME", "\"Note\""), "wh", "k", "|", 0, false, false, null, null);
+ assertEquals(Arrays.asList(3, 5), S3CopySimulator.alignMaxBytes(explicit, catalog));
+
+ CopyStatementParser.S3CopyFrom all = new CopyStatementParser.S3CopyFrom(
+ "t", List.of(), "wh", "k", "|", 0, false, false, null, null);
+ assertEquals(Arrays.asList(null, 3, 5), S3CopySimulator.alignMaxBytes(all, catalog));
+ }
+
+ @Test
+ void streamCopyInput_withTruncateColumnsAndNoCatalogFailsAndNamesSimpleQueryMode() {
+ CopyStatementParser.CopyTransforms transforms =
+ new CopyStatementParser.CopyTransforms(false, false, false, true, null);
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "t", List.of(), "wh", "k", "|", 0, false, false, null, null, false, false, false, transforms);
+ S3CopySimulator.CopyInput input = new S3CopySimulator.CopyInput(spec, List.of("k"), s3, null, null);
+
+ S3CopySimulator.S3TransferException failure = assertThrows(S3CopySimulator.S3TransferException.class,
+ () -> S3CopySimulator.streamCopyInput(input, new ByteArrayOutputStream()));
+ assertTrue(failure.getMessage().contains("preferQueryMode=simple"), failure.getMessage());
+ }
+
+ @Test
+ void streamCopyInput_withFieldTransformOverTheObjectLimitFailsTheStatementAndReleasesTheBudget() {
+ CopyStatementParser.CopyTransforms transforms =
+ new CopyStatementParser.CopyTransforms(true, false, false, false, null);
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "t", List.of(), "wh", "k", "|", 0, false, false, null, null, false, false, false, transforms);
+ when(s3.getObject("wh", "k")).thenReturn(
+ new S3Object("wh", "k", "1||x\n1||x\n".getBytes(StandardCharsets.UTF_8), "text/plain"));
+ S3CopySimulator.CopyInput input = new S3CopySimulator.CopyInput(spec, List.of("k"), s3, null, null);
+
+ long saved = S3CopySimulator.COPY_TRANSFORM_MAX_OBJECT_BYTES;
+ int availableBefore = S3CopySimulator.COPY_TRANSFORM_HEAP_MIB.availablePermits();
+ S3CopySimulator.COPY_TRANSFORM_MAX_OBJECT_BYTES = 8;
+ try {
+ S3CopySimulator.S3TransferException failure = assertThrows(S3CopySimulator.S3TransferException.class,
+ () -> S3CopySimulator.streamCopyInput(input, new ByteArrayOutputStream()));
+ assertTrue(failure.getMessage().contains("8-byte limit"), failure.getMessage());
+ assertEquals(availableBefore, S3CopySimulator.COPY_TRANSFORM_HEAP_MIB.availablePermits());
+ } finally {
+ S3CopySimulator.COPY_TRANSFORM_MAX_OBJECT_BYTES = saved;
+ }
+ }
+
@Test
void copyBackendSql_quotesDiscoveredColumnIdentifiers() {
CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
diff --git a/src/test/java/io/github/hectorvent/floci/services/s3/S3ServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/s3/S3ServiceTest.java
index 67490f6b5b..f60f35aba5 100644
--- a/src/test/java/io/github/hectorvent/floci/services/s3/S3ServiceTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/s3/S3ServiceTest.java
@@ -35,6 +35,10 @@ import java.util.HexFormat;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.FutureTask;
+import java.util.concurrent.TimeUnit;
import java.util.stream.Stream;
import static org.junit.jupiter.api.Assertions.*;
@@ -1634,4 +1638,72 @@ class S3ServiceTest {
void authorizeSignedDeleteObjectIsANoOpWhenEnforceAuthIsOff() {
assertDoesNotThrow(() -> s3Service.authorizeSignedDeleteObject("ASIAFAKEKEY00000001", "sessiontoken", "signed-del-bucket", "some/key"));
}
+
+ @Test
+ void deleteObjectsWithPrefixRemovesOnlyTheMatchingKeys() {
+ s3Service.createBucket("prefix-del", "us-east-1");
+ s3Service.putObject("prefix-del", "out/a", new byte[]{1}, "text/plain", Map.of());
+ s3Service.putObject("prefix-del", "out/b", new byte[]{2}, "text/plain", Map.of());
+ s3Service.putObject("prefix-del", "keep/c", new byte[]{3}, "text/plain", Map.of());
+
+ int deleted = s3Service.deleteObjectsWithPrefix("prefix-del", "out/", key -> { });
+
+ assertEquals(2, deleted);
+ assertFalse(s3Service.objectExists("prefix-del", "out/a"));
+ assertFalse(s3Service.objectExists("prefix-del", "out/b"));
+ assertTrue(s3Service.objectExists("prefix-del", "keep/c"));
+ }
+
+ @Test
+ void deleteObjectsWithPrefixRemovesNothingWhenAnyKeyIsDenied() {
+ s3Service.createBucket("prefix-deny", "us-east-1");
+ s3Service.putObject("prefix-deny", "out/a", new byte[]{1}, "text/plain", Map.of());
+ s3Service.putObject("prefix-deny", "out/b", new byte[]{2}, "text/plain", Map.of());
+
+ assertThrows(AwsException.class, () -> s3Service.deleteObjectsWithPrefix("prefix-deny", "out/", key -> {
+ if (key.equals("out/b")) {
+ throw new AwsException("AccessDenied", "denied", 403);
+ }
+ }));
+
+ assertTrue(s3Service.objectExists("prefix-deny", "out/a"));
+ assertTrue(s3Service.objectExists("prefix-deny", "out/b"));
+ }
+
+ @Test
+ void deleteObjectsWithPrefixHoldsOffAConcurrentPutUntilItIsDone() throws Exception {
+ s3Service.createBucket("prefix-lock", "us-east-1");
+ s3Service.putObject("prefix-lock", "out/old", new byte[]{1}, "text/plain", Map.of());
+ CountDownLatch authorizing = new CountDownLatch(1);
+ CountDownLatch release = new CountDownLatch(1);
+
+ CompletableFuture<Integer> delete = CompletableFuture.supplyAsync(() ->
+ s3Service.deleteObjectsWithPrefix("prefix-lock", "out/", key -> {
+ authorizing.countDown();
+ try {
+ assertTrue(release.await(10, TimeUnit.SECONDS));
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new IllegalStateException(e);
+ }
+ }));
+ assertTrue(authorizing.await(10, TimeUnit.SECONDS));
+
+ FutureTask<S3Object> put = new FutureTask<>(() ->
+ s3Service.putObject("prefix-lock", "out/late", new byte[]{2}, "text/plain", Map.of()));
+ Thread putThread = new Thread(put);
+ putThread.start();
+ long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10);
+ while (putThread.getState() != Thread.State.BLOCKED && System.nanoTime() < deadline) {
+ Thread.onSpinWait();
+ }
+ assertEquals(Thread.State.BLOCKED, putThread.getState());
+ assertFalse(put.isDone());
+
+ release.countDown();
+ assertEquals(1, delete.get(10, TimeUnit.SECONDS));
+ put.get(10, TimeUnit.SECONDS);
+ assertFalse(s3Service.objectExists("prefix-lock", "out/old"));
+ assertTrue(s3Service.objectExists("prefix-lock", "out/late"));
+ }
}