Light, fluffy, and always free - The AWS Local Emulator alternative
feat(redshift): support more COPY and UNLOAD options (#5204)
Vũ Thế Huy38ce99ba5401
diff --git a/docs/services/redshift.md b/docs/services/redshift.mdindex 1a961ba134..5fdddb5526 100644--- a/docs/services/redshift.md+++ b/docs/services/redshift.md@@ -365,10 +365,27 @@ order) through its own S3 service and streams the rows into the backing PostgreS`FLOCI_SERVICES_S3_ENFORCE_AUTH` off, S3 policy checks are skipped. With it on, the role'sidentity policy must allow the required S3 actions, and any bucket policy must not deny therequest. `IAM_ROLE default` is not supported.+- `REGION '<region>'` is accepted and ignored: Floci has one S3.+- `CREDENTIALS 'aws_iam_role=<arn>'` behaves like `IAM_ROLE`. `ACCESS_KEY_ID` with `SECRET_ACCESS_KEY`+ (and an optional `SESSION_TOKEN`), or `CREDENTIALS 'aws_access_key_id=...;aws_secret_access_key=...'`,+ are accepted but never verified or stored: S3 is read as an unsigned request, the same as with no+ authorization clause. Combining key credentials with `IAM_ROLE`, or giving only one of the pair, is+ not intercepted, and neither is a blank key, secret, token or role.+- `EMPTYASNULL`, `BLANKSASNULL`, `REMOVEQUOTES` (not with `CSV` or `JSON`), `ACCEPTINVCHARS [AS 'c']`+ (default `?`, one ASCII character) and `TRUNCATECOLUMNS` rewrite field content before loading. Field+ level options need a single-byte `DELIMITER`, are not combined with `FORMAT AS JSON`, and normalise+ `CRLF` line endings to `LF`. A nulled field is written as the `NULL AS` string, or `\N` in text+ mode and an empty unquoted field in CSV mode. `TRUNCATECOLUMNS` cuts a value to the declared+ `CHAR`/`VARCHAR` length in bytes (Redshift's meaning of the declared length, so `éééé` into+ `VARCHAR(3)` loads `é`), never inside a multi-byte character or a backslash escape. Its column+ lengths come from the database catalog, so it works only over the **Simple Query protocol** (`preferQueryMode=simple`);+ over Extended Query the COPY fails with an error that names this requirement. A field-level option+ buffers each decompressed object in memory, so an object over 64 MiB, or a heap budget shared+ across connections that is used up, fails the COPY instead of exhausting the emulator.- Any other clause (`FIXEDWIDTH`, `PARQUET`, `AVRO`, `ORC`, `MAXERROR`,- `DATEFORMAT`, `TIMEFORMAT`, `REGION`, `ENCODING`, `ESCAPE`, `REMOVEQUOTES`, `BLANKSASNULL`,- `EMPTYASNULL`, `TRUNCATECOLUMNS`, `ACCEPTINVCHARS`, `CREDENTIALS`, and so on) is not- recognized: the statement is forwarded unchanged and PostgreSQL returns its own error.+ `DATEFORMAT`, `TIMEFORMAT`, `ENCODING`, `ESCAPE`, `ENCRYPTED`, `MASTER_SYMMETRIC_KEY`,+ `KMS_KEY_ID`, and so on) is not recognized: the statement is forwarded unchanged and PostgreSQL+ returns its own error.- A multi-statement query whose COPY is followed by another statement is not intercepted; send theCOPY on its own.- Extended Query COPY is supported when the complete statement is present in `Parse` and has no@@ -407,6 +424,32 @@ the result to S3 as one or more objects under `<prefix>`.- A zero-row result still writes one object (empty, or the header row alone when`HEADER` is set).- `GZIP` compresses each object and appends `.gz` to its key.+- `EXTENSION '<ext>'` is appended to each data object key, with a `.` inserted when the value does not+ start with one: `EXTENSION 'csv'` and `EXTENSION '.csv'` both give `<prefix>0000_part_00.csv`. As in+ AWS, the compression suffix is added only when no extension is given, so `EXTENSION 'txt.gz' GZIP`+ gives `<prefix>0000_part_00.txt.gz`. The value is not validated; an empty value or one containing+ `/` is not intercepted. AWS does not document whether it inserts the dot itself.+- `REGION` and the credential clauses (`CREDENTIALS`, `ACCESS_KEY_ID`, `SECRET_ACCESS_KEY`,+ `SESSION_TOKEN`) behave as described for COPY.+- `ESCAPE` is accepted and relies on PostgreSQL text framing, which already escapes the backslash,+ newline, carriage return and the delimiter. It is only valid with text framing: combined with+ `CSV`, `ADDQUOTES` or `HEADER` (which switch the framing to CSV here) the statement is not+ intercepted. Unlike Redshift, PostgreSQL also writes a tab as `\t`.+- `CLEANPATH` deletes every object whose key starts with the target prefix. The objects are listed and+ every delete is authorized up front, like the writes (with `FLOCI_SERVICES_S3_ENFORCE_AUTH` on, the+ role needs list and delete permission, and a denial fails the UNLOAD before anything is removed or+ written). The deletes themselves run when the first output arrives, so a query that PostgreSQL+ rejects leaves the previous export in place. They run on a fresh listing of the prefix, authorized+ and deleted under one bucket lock, so an object uploaded in the meantime does not survive. Objects+ already deleted are not restored if the UNLOAD fails later. The match is a plain string prefix, so `TO 's3://b/sales_' CLEANPATH` also removes+ `sales_archive/...`; end the prefix with `/` to limit it to one folder. It cannot be combined with+ `ALLOWOVERWRITE`, and an empty prefix (the bucket root) is not intercepted so a typo cannot wipe a+ bucket.+- `ENCRYPTED AUTO` is accepted and writes objects as usual. `ENCRYPTED KMS_KEY_ID '<key>'` stores the+ data and manifest objects with `aws:kms` server-side encryption and that key id; Floci does not+ encrypt the data beyond what its S3 service does for those headers. Bare `ENCRYPTED` (server-side+ encryption with the default S3 key) and the client-side `MASTER_SYMMETRIC_KEY` form are not+ intercepted.- `MANIFEST` writes `<prefix>manifest` listing every object with its`content_length`.- Without `ALLOWOVERWRITE`, a non-empty target prefix fails with SQL error XX000 and the select@@ -419,8 +462,8 @@ the result to S3 as one or more objects under `<prefix>`.`FLOCI_SERVICES_S3_ENFORCE_AUTH` off, S3 policy checks are skipped. With it on, the role'sidentity policy must allow the required S3 actions, and any bucket policy must not deny therequest. `IAM_ROLE default` is not supported.-- Any other option (`PARQUET`, `ENCRYPTED`, `REGION`, `CREDENTIALS`,- `ZSTD`, `EXTENSION`, `CLEANPATH`, `PARTITION`, and so on) is not intercepted; the+- Any other option (`PARQUET`, `JSON`, `FIXEDWIDTH`, `MASTER_SYMMETRIC_KEY`, `MANIFEST VERBOSE`,+ `ZSTD`, `BZIP2`, `PARTITION`, and so on) is not intercepted; thestatement is forwarded and PostgreSQL reports its own error.- Extended Query UNLOAD is supported when the complete statement is present in `Parse` and has nobind parameters. Parameterized statements are forwarded unchanged.diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyRecordTransformer.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyRecordTransformer.javanew file mode 100644index 0000000000..c8e559a5c3--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyRecordTransformer.java@@ -0,0 +1,352 @@+package io.github.hectorvent.floci.services.redshift.proxy;++import java.io.ByteArrayOutputStream;+import java.nio.charset.StandardCharsets;+import java.util.Arrays;+import java.util.List;++/**+ * Applies the COPY options that change field content to the bytes of one S3 object before they+ * reach PostgreSQL's {@code COPY ... FROM STDIN}. The object is already fully in memory, so the+ * transform works on a byte array: there are no chunk boundaries to straddle.+ */+final class CopyRecordTransformer {++ private static final String SQLSTATE_INTERNAL = "XX000";+ private static final byte NEWLINE = '\n';+ private static final byte CARRIAGE_RETURN = '\r';+ private static final byte BACKSLASH = '\\';+ private static final byte DOUBLE_QUOTE = '"';+ private static final byte SINGLE_QUOTE = '\'';++ private record Field(int tokenEnd, byte[] value, boolean quoted) {+ }++ private final CopyStatementParser.CopyTransforms transforms;+ private final boolean csv;+ private final byte delimiter;+ private final byte[] nullMarker;+ private final List<Integer> columnMaxBytes;++ CopyRecordTransformer(CopyStatementParser.S3CopyFrom spec, List<Integer> columnMaxBytes) {+ this.transforms = spec.transforms();+ this.csv = spec.csv();+ String delimiterText = spec.delimiter() != null ? spec.delimiter() : (spec.csv() ? "," : "|");+ this.delimiter = delimiterText.getBytes(StandardCharsets.UTF_8)[0];+ String marker = spec.nullAs() != null ? spec.nullAs() : (spec.csv() ? "" : "\\N");+ this.nullMarker = marker.getBytes(StandardCharsets.UTF_8);+ this.columnMaxBytes = columnMaxBytes;+ }++ byte[] apply(byte[] input) {+ byte[] data = transforms.invalidCharReplacement() != null+ ? sanitizeUtf8(input, (byte) transforms.invalidCharReplacement().charValue())+ : input;+ if (!transforms.fieldLevel()) {+ return data;+ }+ ByteArrayOutputStream out = new ByteArrayOutputStream(data.length + 16);+ int position = 0;+ while (position < data.length) {+ position = transformRecord(data, position, out);+ }+ return out.toByteArray();+ }++ static byte[] sanitizeUtf8(byte[] input, byte replacement) {+ ByteArrayOutputStream out = new ByteArrayOutputStream(input.length);+ int i = 0;+ while (i < input.length) {+ int length = validSequenceLength(input, i);+ if (length == 0) {+ out.write(replacement);+ i++;+ } else {+ out.write(input, i, length);+ i += length;+ }+ }+ return out.toByteArray();+ }++ private static int validSequenceLength(byte[] bytes, int index) {+ int lead = bytes[index] & 0xFF;+ if (lead < 0x80) {+ return 1;+ }+ int length;+ if (lead >= 0xC2 && lead <= 0xDF) {+ length = 2;+ } else if (lead >= 0xE0 && lead <= 0xEF) {+ length = 3;+ } else if (lead >= 0xF0 && lead <= 0xF4) {+ length = 4;+ } else {+ return 0;+ }+ if (index + length > bytes.length) {+ return 0;+ }+ for (int k = 1; k < length; k++) {+ if ((bytes[index + k] & 0xC0) != 0x80) {+ return 0;+ }+ }+ int second = bytes[index + 1] & 0xFF;+ boolean overlongOrOutOfRange = (lead == 0xE0 && second < 0xA0)+ || (lead == 0xED && second > 0x9F)+ || (lead == 0xF0 && second < 0x90)+ || (lead == 0xF4 && second > 0x8F);+ return overlongOrOutOfRange ? 0 : length;+ }++ private int transformRecord(byte[] data, int start, ByteArrayOutputStream out) {+ int position = start;+ int column = 0;+ while (true) {+ Field field = scanField(data, position);+ writeField(field, column, out);+ position = field.tokenEnd();+ if (position >= data.length) {+ out.write(NEWLINE);+ return data.length;+ }+ if (data[position] == NEWLINE) {+ out.write(NEWLINE);+ return position + 1;+ }+ if (isCrlfAt(data, position)) {+ out.write(NEWLINE);+ return Math.min(position + 2, data.length);+ }+ if (data[position] != delimiter) {+ throw malformed("unexpected character after closing quote");+ }+ out.write(delimiter);+ position++;+ column++;+ }+ }++ private Field scanField(byte[] data, int start) {+ int length = data.length;+ if (start < length && transforms.removeQuotes()+ && (data[start] == DOUBLE_QUOTE || data[start] == SINGLE_QUOTE)) {+ byte quote = data[start];+ for (int j = start + 1; j < length; j++) {+ if (!csv && data[j] == BACKSLASH) {+ // Text mode: a backslash escapes the next byte, so an escaped quote never closes the field.+ j++;+ continue;+ }+ boolean closes = data[j] == quote+ && (j + 1 == length || data[j + 1] == delimiter || data[j + 1] == NEWLINE+ || isCrlfAt(data, j + 1));+ if (closes) {+ return new Field(j + 1, Arrays.copyOfRange(data, start + 1, j), true);+ }+ }+ throw malformed("quoted field has no closing quote");+ }+ if (start < length && csv && data[start] == DOUBLE_QUOTE) {+ ByteArrayOutputStream value = new ByteArrayOutputStream();+ int j = start + 1;+ while (j < length) {+ if (data[j] == DOUBLE_QUOTE) {+ if (j + 1 < length && data[j + 1] == DOUBLE_QUOTE) {+ value.write(DOUBLE_QUOTE);+ j += 2;+ continue;+ }+ return new Field(j + 1, value.toByteArray(), true);+ }+ value.write(data[j]);+ j++;+ }+ throw malformed("quoted field has no closing quote");+ }+ int j = start;+ while (j < length && data[j] != delimiter && data[j] != NEWLINE) {+ if (!csv && data[j] == BACKSLASH && j + 1 < length) {+ j++;+ }+ j++;+ }+ int valueEnd = j;+ if (j < length && data[j] == NEWLINE && j > start && data[j - 1] == CARRIAGE_RETURN) {+ valueEnd = j - 1;+ }+ return new Field(j, Arrays.copyOfRange(data, start, valueEnd), false);+ }++ /** A carriage return that ends the record: followed by a newline or by the end of the data. */+ private static boolean isCrlfAt(byte[] data, int index) {+ return index < data.length && data[index] == CARRIAGE_RETURN+ && (index + 1 == data.length || data[index + 1] == NEWLINE);+ }++ private void writeField(Field field, int column, ByteArrayOutputStream out) {+ byte[] value = field.value();+ boolean nulled = (transforms.blanksAsNull() && isBlank(value))+ || (transforms.emptyAsNull() && value.length == 0);+ if (nulled) {+ out.writeBytes(nullMarker);+ return;+ }+ // A field that already is the NULL marker must not be cut: that would turn NULL into data.+ byte[] limited = !field.quoted() && Arrays.equals(value, nullMarker) ? value : truncate(value, column);+ if (!field.quoted()) {+ out.writeBytes(limited);+ } else if (csv) {+ writeCsvQuoted(limited, out);+ } else {+ out.writeBytes(escapeTextField(limited));+ }+ }++ private static boolean isBlank(byte[] value) {+ if (value.length == 0) {+ return false;+ }+ for (byte b : value) {+ if (b != ' ' && b != '\t') {+ return false;+ }+ }+ return true;+ }++ private byte[] truncate(byte[] value, int column) {+ if (!transforms.truncateColumns() || columnMaxBytes == null || column >= columnMaxBytes.size()) {+ return value;+ }+ Integer max = columnMaxBytes.get(column);+ if (max == null) {+ return value;+ }+ // Redshift counts CHAR(n) and VARCHAR(n) in bytes, so measure the decoded bytes and cut at a+ // character boundary. Text mode decodes its escapes first so one is never split.+ byte[] decoded = csv ? value : decodeTextEscapes(value);+ int end = byteLimitBoundary(decoded, max);+ if (end == decoded.length) {+ return value;+ }+ byte[] kept = Arrays.copyOf(decoded, end);+ return csv ? kept : encodeTextEscapes(kept);+ }++ /** Largest offset not above {@code maxBytes} that does not fall inside a multi-byte character. */+ private static int byteLimitBoundary(byte[] bytes, int maxBytes) {+ if (bytes.length <= maxBytes) {+ return bytes.length;+ }+ int end = maxBytes;+ while (end > 0 && (bytes[end] & 0xC0) == 0x80) {+ end--;+ }+ return end;+ }++ /** Decodes COPY text escapes: backslash b f n r t v, octal NNN, hex xHH, and backslash c for any other c. */+ private static byte[] decodeTextEscapes(byte[] value) {+ ByteArrayOutputStream out = new ByteArrayOutputStream(value.length);+ int i = 0;+ while (i < value.length) {+ byte b = value[i];+ if (b != BACKSLASH || i + 1 >= value.length) {+ out.write(b);+ i++;+ continue;+ }+ byte next = value[i + 1];+ if (next >= '0' && next <= '7') {+ int octal = 0;+ int digits = 0;+ i++;+ while (i < value.length && digits < 3 && value[i] >= '0' && value[i] <= '7') {+ octal = octal * 8 + (value[i] - '0');+ i++;+ digits++;+ }+ out.write(octal);+ } else if (next == 'x' && i + 2 < value.length && Character.digit(value[i + 2], 16) >= 0) {+ int hex = Character.digit(value[i + 2], 16);+ i += 3;+ if (i < value.length && Character.digit(value[i], 16) >= 0) {+ hex = hex * 16 + Character.digit(value[i], 16);+ i++;+ }+ out.write(hex);+ } else {+ out.write(switch (next) {+ case 'b' -> '\b';+ case 'f' -> '\f';+ case 'n' -> '\n';+ case 'r' -> '\r';+ case 't' -> '\t';+ case 'v' -> 0x0B;+ default -> next;+ });+ i += 2;+ }+ }+ return out.toByteArray();+ }++ private byte[] encodeTextEscapes(byte[] value) {+ ByteArrayOutputStream out = new ByteArrayOutputStream(value.length + 4);+ for (byte b : value) {+ if (b == BACKSLASH || b == delimiter) {+ out.write(BACKSLASH);+ out.write(b);+ } else if (b == NEWLINE) {+ out.write(BACKSLASH);+ out.write('n');+ } else if (b == CARRIAGE_RETURN) {+ out.write(BACKSLASH);+ out.write('r');+ } else {+ out.write(b);+ }+ }+ return out.toByteArray();+ }++ private byte[] escapeTextField(byte[] value) {+ ByteArrayOutputStream escaped = new ByteArrayOutputStream(value.length + 4);+ for (int i = 0; i < value.length; i++) {+ byte b = value[i];+ if (b == BACKSLASH && i + 1 < value.length) {+ escaped.write(b);+ escaped.write(value[++i]);+ } else if (b == delimiter) {+ escaped.write(BACKSLASH);+ escaped.write(b);+ } else if (b == NEWLINE) {+ escaped.write(BACKSLASH);+ escaped.write('n');+ } else if (b == CARRIAGE_RETURN) {+ escaped.write(BACKSLASH);+ escaped.write('r');+ } else {+ escaped.write(b);+ }+ }+ return escaped.toByteArray();+ }++ private static void writeCsvQuoted(byte[] value, ByteArrayOutputStream out) {+ out.write(DOUBLE_QUOTE);+ for (byte b : value) {+ if (b == DOUBLE_QUOTE) {+ out.write(DOUBLE_QUOTE);+ }+ out.write(b);+ }+ out.write(DOUBLE_QUOTE);+ }++ private static S3CopySimulator.S3TransferException malformed(String detail) {+ return new S3CopySimulator.S3TransferException(SQLSTATE_INTERNAL, "COPY failed: " + detail, null);+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.javaindex 22d0e82bb1..b30ac9ec3a 100644--- a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.java+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.java@@ -1,5 +1,6 @@package io.github.hectorvent.floci.services.redshift.proxy;+import java.nio.charset.StandardCharsets;import java.util.Arrays;import java.util.List;import java.util.regex.Matcher;@@ -16,6 +17,25 @@ public final class CopyStatementParser {public sealed interface S3Statement permits S3CopyFrom, S3Unload {}+ /** COPY options that change field content; {@link #NONE} keeps the plain byte-copy path. */+ public record CopyTransforms(+ boolean blanksAsNull,+ boolean emptyAsNull,+ boolean removeQuotes,+ boolean truncateColumns,+ Character invalidCharReplacement) {++ public static final CopyTransforms NONE = new CopyTransforms(false, false, false, false, null);++ public boolean fieldLevel() {+ return blanksAsNull || emptyAsNull || removeQuotes || truncateColumns;+ }++ public boolean any() {+ return fieldLevel() || invalidCharReplacement != null;+ }+ }+public record S3CopyFrom(String targetTable,List<String> columns,@@ -29,7 +49,32 @@ public final class CopyStatementParser {String iamRoleArn,boolean jsonAuto,boolean jsonAutoIgnoreCase,- boolean manifest) implements S3Statement {+ boolean manifest,+ CopyTransforms transforms) implements S3Statement {++ public S3CopyFrom {+ if (transforms == null) {+ transforms = CopyTransforms.NONE;+ }+ }++ public S3CopyFrom(+ String targetTable,+ List<String> columns,+ String bucket,+ String keyOrPrefix,+ String delimiter,+ int headerLines,+ boolean gzip,+ boolean csv,+ String nullAs,+ String iamRoleArn,+ boolean jsonAuto,+ boolean jsonAutoIgnoreCase,+ boolean manifest) {+ this(targetTable, columns, bucket, keyOrPrefix, delimiter, headerLines, gzip, csv,+ nullAs, iamRoleArn, jsonAuto, jsonAutoIgnoreCase, manifest, CopyTransforms.NONE);+ }public S3CopyFrom(String targetTable,@@ -78,7 +123,52 @@ public final class CopyStatementParser {boolean allowOverwrite,boolean parallel,long maxFileSizeBytes,- String iamRoleArn) implements S3Statement {+ String iamRoleArn,+ String extension,+ boolean escape,+ boolean cleanPath,+ boolean encrypted,+ String sseKmsKeyId) implements S3Statement {++ public S3Unload(+ String selectQuery,+ String bucket,+ String prefix,+ String delimiter,+ boolean header,+ boolean gzip,+ boolean csv,+ boolean addQuotes,+ String nullAs,+ boolean manifest,+ boolean allowOverwrite,+ boolean parallel,+ long maxFileSizeBytes,+ String iamRoleArn,+ String extension) {+ this(selectQuery, bucket, prefix, delimiter, header, gzip, csv, addQuotes, nullAs, manifest,+ allowOverwrite, parallel, maxFileSizeBytes, iamRoleArn, extension,+ false, false, false, null);+ }++ public S3Unload(+ String selectQuery,+ String bucket,+ String prefix,+ String delimiter,+ boolean header,+ boolean gzip,+ boolean csv,+ boolean addQuotes,+ String nullAs,+ boolean manifest,+ boolean allowOverwrite,+ boolean parallel,+ long maxFileSizeBytes,+ String iamRoleArn) {+ this(selectQuery, bucket, prefix, delimiter, header, gzip, csv, addQuotes, nullAs, manifest,+ allowOverwrite, parallel, maxFileSizeBytes, iamRoleArn, null);+ }}private static final Pattern COPY_PATTERN = Pattern.compile(@@ -96,17 +186,23 @@ public final class CopyStatementParser {private static final Pattern ADDQUOTES_PATTERN = Pattern.compile("(?i)\\bADDQUOTES\\b");private static final Pattern MANIFEST_PATTERN = Pattern.compile("(?i)\\bMANIFEST\\b");private static final Pattern ALLOWOVERWRITE_PATTERN = Pattern.compile("(?i)\\bALLOWOVERWRITE\\b");+ private static final Pattern ESCAPE_PATTERN = Pattern.compile("(?i)\\bESCAPE\\b");+ private static final Pattern CLEANPATH_PATTERN = Pattern.compile("(?i)\\bCLEANPATH\\b");+ private static final Pattern ENCRYPTED_PATTERN = Pattern.compile("(?i)\\bENCRYPTED(?:\\s+(AUTO)\\b)?");+ private static final Pattern KMS_KEY_ID_PATTERN = Pattern.compile(+ "(?i)\\bKMS_KEY_ID\\s+(?:'((?:[^']|'')*)'|\"([^\"]*)\")");+ private static final Pattern EXTENSION_PATTERN = Pattern.compile(+ "(?i)\\bEXTENSION\\s+(?:'((?:[^']|'')*)'|\"([^\"]*)\")");/*** UNLOAD options this simulator cannot honour. Distinct from the COPY set:- * MANIFEST / ALLOWOVERWRITE / PARALLEL / MAXFILESIZE are UNLOAD-supported here,- * while EXTENSION / CLEANPATH / PARTITION are UNLOAD-only and unsupported.+ * MANIFEST / ALLOWOVERWRITE / PARALLEL / MAXFILESIZE / EXTENSION / CLEANPATH are+ * UNLOAD-supported here, while PARTITION is UNLOAD-only and unsupported.*/private static final Pattern UNLOAD_UNSUPPORTED_CLAUSE = Pattern.compile("(?i)\\b(FIXEDWIDTH|PARQUET|AVRO|ORC|JSON|SHAPEFILE|BZIP2|LZOP|ZSTD"- + "|ENCRYPTED|ENCODING|REGION|CREDENTIALS|ACCESS_KEY_ID"- + "|SECRET_ACCESS_KEY|SESSION_TOKEN|MASTER_SYMMETRIC_KEY|KMS_KEY_ID"- + "|EXTENSION|CLEANPATH|PARTITION|MAXFILESIZE\\s+\\d+\\s*(?:TB|PB))\\b");+ + "|ENCODING|MASTER_SYMMETRIC_KEY"+ + "|PARTITION|MAXFILESIZE\\s+\\d+\\s*(?:TB|PB))\\b");private static final Pattern QUALIFIED_NAME = Pattern.compile("(?:[A-Za-z_][A-Za-z0-9_$]*|\"[^\"]+\")(?:\\.(?:[A-Za-z_][A-Za-z0-9_$]*|\"[^\"]+\"))?");@@ -118,6 +214,28 @@ public final class CopyStatementParser {"(?i)\\bNULL\\s+(?:AS\\s+)?(?:'((?:[^']|'')*)'|\"([^\"]*)\")");private static final Pattern IAM_ROLE_PATTERN = Pattern.compile("(?i)\\bIAM_ROLE\\s+(?:'((?:[^']|'')*)'|\"([^\"]*)\")");+ private static final Pattern BLANKSASNULL_PATTERN = Pattern.compile("(?i)\\bBLANKSASNULL\\b");+ private static final Pattern EMPTYASNULL_PATTERN = Pattern.compile("(?i)\\bEMPTYASNULL\\b");+ private static final Pattern REMOVEQUOTES_PATTERN = Pattern.compile("(?i)\\bREMOVEQUOTES\\b");+ private static final Pattern TRUNCATECOLUMNS_PATTERN = Pattern.compile("(?i)\\bTRUNCATECOLUMNS\\b");+ private static final Pattern ACCEPTINVCHARS_PATTERN = Pattern.compile(+ "(?i)\\bACCEPTINVCHARS(?:\\s+(?:AS\\s+)?'([ -&(-~])')?");+ private static final Pattern CREDENTIALS_PATTERN = Pattern.compile(+ "(?i)\\bCREDENTIALS\\s+(?:AS\\s+)?(?:'((?:[^']|'')*)'|\"([^\"]*)\")");+ private static final Pattern ACCESS_KEY_ID_PATTERN = Pattern.compile(+ "(?i)\\bACCESS_KEY_ID\\s+(?:AS\\s+)?(?:'((?:[^']|'')*)'|\"([^\"]*)\")");+ private static final Pattern SECRET_ACCESS_KEY_PATTERN = Pattern.compile(+ "(?i)\\bSECRET_ACCESS_KEY\\s+(?:AS\\s+)?(?:'((?:[^']|'')*)'|\"([^\"]*)\")");+ private static final Pattern SESSION_TOKEN_PATTERN = Pattern.compile(+ "(?i)\\bSESSION_TOKEN\\s+(?:AS\\s+)?(?:'((?:[^']|'')*)'|\"([^\"]*)\")");+ private static final Pattern REGION_PATTERN = Pattern.compile(+ "(?i)\\bREGION\\s+(?:AS\\s+)?(?:'((?:[^']|'')*)'|\"([^\"]*)\")");+ private static final Pattern CREDENTIALS_ROLE = Pattern.compile(+ "(?i)(?:^|;)\\s*aws_iam_role\\s*=\\s*([^;\\s]+)");+ private static final Pattern CREDENTIALS_KEY_ID = Pattern.compile(+ "(?i)(?:^|;)\\s*aws_access_key_id\\s*=\\s*[^;\\s]+");+ private static final Pattern CREDENTIALS_SECRET = Pattern.compile(+ "(?i)(?:^|;)\\s*aws_secret_access_key\\s*=\\s*[^;\\s]+");private static final Pattern IGNOREHEADER_PATTERN = Pattern.compile("(?i)\\bIGNOREHEADER\\s+(?:AS\\s+)?(\\d+)\\b");private static final Pattern HEADER_PATTERN = Pattern.compile("(?i)\\bHEADER\\b");@@ -133,10 +251,10 @@ public final class CopyStatementParser {*/private static final Pattern UNSUPPORTED_CLAUSE = Pattern.compile("(?i)\\b(FIXEDWIDTH|PARQUET|AVRO|ORC|SHAPEFILE|BZIP2|LZOP|ZSTD|MAXERROR"- + "|DATEFORMAT|TIMEFORMAT|ENCRYPTED|ENCODING|REGION|CREDENTIALS"- + "|ACCESS_KEY_ID|SECRET_ACCESS_KEY|SESSION_TOKEN|MASTER_SYMMETRIC_KEY|KMS_KEY_ID"- + "|ACCEPTINVCHARS|ACCEPTANYDATE|BLANKSASNULL|EMPTYASNULL|FILLRECORD|TRIMBLANKS"- + "|TRUNCATECOLUMNS|IGNOREBLANKLINES|ESCAPE|REMOVEQUOTES|EXPLICIT_IDS|COMPUPDATE"+ + "|DATEFORMAT|TIMEFORMAT|ENCRYPTED|ENCODING"+ + "|MASTER_SYMMETRIC_KEY|KMS_KEY_ID"+ + "|ACCEPTANYDATE|FILLRECORD|TRIMBLANKS"+ + "|IGNOREBLANKLINES|ESCAPE|EXPLICIT_IDS|COMPUPDATE"+ "|STATUPDATE|NOLOAD|ROUNDEC|QUOTE|SSH|READRATIO|COMPROWS|DIMENSION)\\b");/** A {@code ;} followed by another statement: only a lone trailing {@code ;} is tolerated. */@@ -192,8 +310,18 @@ public final class CopyStatementParser {String nullAs = null;String delimiter = null;long maxFileSizeBytes = 0L;- String iamRoleArn = null;+ String extension = null;+ boolean escape = false;+ boolean cleanPath = false;+ boolean encrypted = false;+ boolean encryptedAuto = false;+ String kmsKeyId = null;+ boolean seenEscape = false;+ boolean seenCleanPath = false;+ boolean seenEncrypted = false;+ boolean seenKmsKeyId = false;+ boolean seenExtension = false;boolean seenCsv = false;boolean seenGzip = false;boolean seenHeader = false;@@ -204,7 +332,6 @@ public final class CopyStatementParser {boolean seenNull = false;boolean seenDelimiter = false;boolean seenMaxFileSize = false;- boolean seenIamRole = false;Matcher csvM = CSV_PATTERN.matcher(options);Matcher gzipM = GZIP_PATTERN.matcher(options);@@ -216,8 +343,14 @@ public final class CopyStatementParser {Matcher delimiterM = DELIMITER_PATTERN.matcher(options);Matcher nullM = NULL_AS_PATTERN.matcher(options);Matcher maxFileSizeM = MAXFILESIZE_PATTERN.matcher(options);- Matcher iamRoleM = IAM_ROLE_PATTERN.matcher(options);+ Matcher extensionM = EXTENSION_PATTERN.matcher(options);+ Matcher escapeM = ESCAPE_PATTERN.matcher(options);+ Matcher cleanPathM = CLEANPATH_PATTERN.matcher(options);+ Matcher encryptedM = ENCRYPTED_PATTERN.matcher(options);+ Matcher kmsKeyIdM = KMS_KEY_ID_PATTERN.matcher(options);+ AuthClauses auth = new AuthClauses(options);+ int next;int offset = 0;int len = options.length();while (offset < len) {@@ -227,7 +360,46 @@ public final class CopyStatementParser {if (offset >= len) {break;}- if (matchClause(maxFileSizeM, offset, len)) {+ if (matchClause(extensionM, offset, len)) {+ if (seenExtension) {+ return null;+ }+ seenExtension = true;+ extension = extractNullValue(extensionM);+ if (extension.isEmpty() || extension.indexOf('/') >= 0) {+ return null;+ }+ offset = extensionM.end();+ } else if (matchClause(escapeM, offset, len)) {+ if (seenEscape) {+ return null;+ }+ seenEscape = true;+ escape = true;+ offset = escapeM.end();+ } else if (matchClause(cleanPathM, offset, len)) {+ if (seenCleanPath) {+ return null;+ }+ seenCleanPath = true;+ cleanPath = true;+ offset = cleanPathM.end();+ } else if (matchClause(encryptedM, offset, len)) {+ if (seenEncrypted) {+ return null;+ }+ seenEncrypted = true;+ encrypted = true;+ encryptedAuto = encryptedM.group(1) != null;+ offset = encryptedM.end();+ } else if (matchClause(kmsKeyIdM, offset, len)) {+ if (seenKmsKeyId) {+ return null;+ }+ seenKmsKeyId = true;+ kmsKeyId = extractNullValue(kmsKeyIdM);+ offset = kmsKeyIdM.end();+ } else if (matchClause(maxFileSizeM, offset, len)) {if (seenMaxFileSize) {return null;}@@ -303,23 +475,41 @@ public final class CopyStatementParser {seenNull = true;nullAs = extractNullValue(nullM);offset = nullM.end();- } else if (matchClause(iamRoleM, offset, len)) {- if (seenIamRole) {+ } else if ((next = auth.consume(offset, len)) != AuthClauses.NO_MATCH) {+ if (next == AuthClauses.REJECT) {return null;}- seenIamRole = true;- iamRoleArn = extractNullValue(iamRoleM);- offset = iamRoleM.end();+ offset = next;} else {return null;}}+ if (!auth.consistent()) {+ return null;+ }+ String iamRoleArn = auth.iamRoleArn();if (delimiter == null) {delimiter = csv ? "," : "|";}+ // ESCAPE relies on PostgreSQL text framing, which CSV, ADDQUOTES and HEADER all replace here.+ if (escape && (csv || addQuotes || header)) {+ return null;+ }+ // An empty prefix would make CLEANPATH delete the whole bucket.+ if (cleanPath && (allowOverwrite || prefix.isEmpty())) {+ return null;+ }+ // Only ENCRYPTED AUTO and ENCRYPTED KMS_KEY_ID are emulated. Bare ENCRYPTED and MASTER_SYMMETRIC_KEY are not.+ if (kmsKeyId != null && (kmsKeyId.isBlank() || !encrypted || encryptedAuto)) {+ return null;+ }+ if (encrypted && !encryptedAuto && kmsKeyId == null) {+ return null;+ }return new S3Unload(select, bucket, prefix, delimiter, header, gzip, csv,- addQuotes, nullAs, manifest, allowOverwrite, parallel, maxFileSizeBytes, iamRoleArn);+ addQuotes, nullAs, manifest, allowOverwrite, parallel, maxFileSizeBytes, iamRoleArn, extension,+ escape, cleanPath, encrypted, kmsKeyId);}private static S3CopyFrom parseCopy(String cleaned) {@@ -372,8 +562,17 @@ public final class CopyStatementParser {String nullAs = null;String delimiter = null;int headerLines = 0;- String iamRoleArn = null;+ boolean blanksAsNull = false;+ boolean emptyAsNull = false;+ boolean removeQuotes = false;+ boolean truncateColumns = false;+ Character invalidChar = null;+ boolean seenTruncateColumns = false;+ boolean seenBlanksAsNull = false;+ boolean seenEmptyAsNull = false;+ boolean seenRemoveQuotes = false;+ boolean seenAcceptInvChars = false;boolean seenCsv = false;boolean seenGzip = false;boolean seenJson = false;@@ -381,7 +580,6 @@ public final class CopyStatementParser {boolean seenNull = false;boolean seenDelimiter = false;boolean seenHeader = false;- boolean seenIamRole = false;Matcher csvMatcher = CSV_PATTERN.matcher(options);Matcher gzipMatcher = GZIP_PATTERN.matcher(options);@@ -391,8 +589,14 @@ public final class CopyStatementParser {Matcher headerMatcher = HEADER_PATTERN.matcher(options);Matcher delimiterMatcher = DELIMITER_PATTERN.matcher(options);Matcher nullMatcher = NULL_AS_PATTERN.matcher(options);- Matcher iamRoleMatcher = IAM_ROLE_PATTERN.matcher(options);+ Matcher blanksAsNullMatcher = BLANKSASNULL_PATTERN.matcher(options);+ Matcher emptyAsNullMatcher = EMPTYASNULL_PATTERN.matcher(options);+ Matcher removeQuotesMatcher = REMOVEQUOTES_PATTERN.matcher(options);+ Matcher truncateColumnsMatcher = TRUNCATECOLUMNS_PATTERN.matcher(options);+ Matcher acceptInvCharsMatcher = ACCEPTINVCHARS_PATTERN.matcher(options);+ AuthClauses auth = new AuthClauses(options);+ int next;int offset = 0;int len = options.length();while (offset < len) {@@ -460,24 +664,203 @@ public final class CopyStatementParser {seenNull = true;nullAs = extractNullValue(nullMatcher);offset = nullMatcher.end();- } else if (matchClause(iamRoleMatcher, offset, len)) {- if (seenIamRole) {+ } else if (matchClause(blanksAsNullMatcher, offset, len)) {+ if (seenBlanksAsNull) {return null;}- seenIamRole = true;- iamRoleArn = extractNullValue(iamRoleMatcher);- offset = iamRoleMatcher.end();+ seenBlanksAsNull = true;+ blanksAsNull = true;+ offset = blanksAsNullMatcher.end();+ } else if (matchClause(emptyAsNullMatcher, offset, len)) {+ if (seenEmptyAsNull) {+ return null;+ }+ seenEmptyAsNull = true;+ emptyAsNull = true;+ offset = emptyAsNullMatcher.end();+ } else if (matchClause(removeQuotesMatcher, offset, len)) {+ if (seenRemoveQuotes) {+ return null;+ }+ seenRemoveQuotes = true;+ removeQuotes = true;+ offset = removeQuotesMatcher.end();+ } else if (matchClause(truncateColumnsMatcher, offset, len)) {+ if (seenTruncateColumns) {+ return null;+ }+ seenTruncateColumns = true;+ truncateColumns = true;+ offset = truncateColumnsMatcher.end();+ } else if (matchClause(acceptInvCharsMatcher, offset, len)) {+ if (seenAcceptInvChars) {+ return null;+ }+ seenAcceptInvChars = true;+ invalidChar = acceptInvCharsMatcher.group(1) != null+ ? Character.valueOf(acceptInvCharsMatcher.group(1).charAt(0))+ : Character.valueOf('?');+ offset = acceptInvCharsMatcher.end();+ } else if ((next = auth.consume(offset, len)) != AuthClauses.NO_MATCH) {+ if (next == AuthClauses.REJECT) {+ return null;+ }+ offset = next;} else {return null;}}+ if (!auth.consistent()) {+ return null;+ }+ String iamRoleArn = auth.iamRoleArn();if (delimiter == null) {delimiter = csv ? "," : "|";}+ CopyTransforms transforms = new CopyTransforms(+ blanksAsNull, emptyAsNull, removeQuotes, truncateColumns, invalidChar);+ if (transforms.any() && jsonAuto) {+ return null;+ }+ if (transforms.removeQuotes() && csv) {+ return null;+ }+ if (transforms.fieldLevel() && delimiter.getBytes(StandardCharsets.UTF_8).length != 1) {+ return null;+ }+ if (invalidChar != null && replacementIsFramingSyntax(invalidChar, delimiter, csv)) {+ return null;+ }+return new S3CopyFrom(table, columns, bucket, keyOrPrefix, delimiter, headerLines, gzip, csv,- nullAs, iamRoleArn, jsonAuto, jsonAutoIgnoreCase, manifest);+ nullAs, iamRoleArn, jsonAuto, jsonAutoIgnoreCase, manifest, transforms);+ }++ /**+ * Collects the S3 authorization and region clauses shared by COPY and UNLOAD. Key material is+ * checked for shape only and never stored: the simulator reads S3 as the cluster role or as an+ * unsigned request.+ */+ private static final class AuthClauses {+ static final int NO_MATCH = -1;+ static final int REJECT = -2;++ private final Matcher iamRole;+ private final Matcher credentials;+ private final Matcher accessKeyId;+ private final Matcher secretAccessKey;+ private final Matcher sessionToken;+ private final Matcher region;++ private String iamRoleArn;+ private String credentialsRoleArn;+ private boolean seenIamRole;+ private boolean seenCredentials;+ private boolean seenAccessKeyId;+ private boolean seenSecretAccessKey;+ private boolean seenSessionToken;+ private boolean seenRegion;+ private boolean credentialsHaveKeys;+ private boolean credentialsMalformed;++ AuthClauses(String options) {+ iamRole = IAM_ROLE_PATTERN.matcher(options);+ credentials = CREDENTIALS_PATTERN.matcher(options);+ accessKeyId = ACCESS_KEY_ID_PATTERN.matcher(options);+ secretAccessKey = SECRET_ACCESS_KEY_PATTERN.matcher(options);+ sessionToken = SESSION_TOKEN_PATTERN.matcher(options);+ region = REGION_PATTERN.matcher(options);+ }++ int consume(int offset, int end) {+ if (matchClause(iamRole, offset, end)) {+ if (seenIamRole) {+ return REJECT;+ }+ seenIamRole = true;+ iamRoleArn = extractNullValue(iamRole);+ return isBlank(iamRoleArn) ? REJECT : iamRole.end();+ }+ if (matchClause(credentials, offset, end)) {+ if (seenCredentials) {+ return REJECT;+ }+ seenCredentials = true;+ readCredentials(extractNullValue(credentials));+ return credentials.end();+ }+ if (matchClause(accessKeyId, offset, end)) {+ if (seenAccessKeyId) {+ return REJECT;+ }+ seenAccessKeyId = true;+ return isBlank(extractNullValue(accessKeyId)) ? REJECT : accessKeyId.end();+ }+ if (matchClause(secretAccessKey, offset, end)) {+ if (seenSecretAccessKey) {+ return REJECT;+ }+ seenSecretAccessKey = true;+ return isBlank(extractNullValue(secretAccessKey)) ? REJECT : secretAccessKey.end();+ }+ if (matchClause(sessionToken, offset, end)) {+ if (seenSessionToken) {+ return REJECT;+ }+ seenSessionToken = true;+ return isBlank(extractNullValue(sessionToken)) ? REJECT : sessionToken.end();+ }+ if (matchClause(region, offset, end)) {+ if (seenRegion) {+ return REJECT;+ }+ seenRegion = true;+ return region.end();+ }+ return NO_MATCH;+ }++ private void readCredentials(String value) {+ Matcher role = CREDENTIALS_ROLE.matcher(value);+ if (role.find()) {+ credentialsRoleArn = role.group(1);+ }+ boolean hasId = CREDENTIALS_KEY_ID.matcher(value).find();+ boolean hasSecret = CREDENTIALS_SECRET.matcher(value).find();+ credentialsHaveKeys = hasId && hasSecret;+ credentialsMalformed = hasId != hasSecret;+ }++ boolean consistent() {+ if (credentialsMalformed || seenAccessKeyId != seenSecretAccessKey) {+ return false;+ }+ if (seenSessionToken && !seenAccessKeyId) {+ return false;+ }+ if (seenCredentials && seenAccessKeyId) {+ return false;+ }+ if (seenCredentials && credentialsRoleArn == null && !credentialsHaveKeys) {+ return false;+ }+ int roles = (seenIamRole ? 1 : 0) + (credentialsRoleArn != null ? 1 : 0);+ boolean keys = seenAccessKeyId || credentialsHaveKeys;+ return roles <= 1 && !(roles == 1 && keys);+ }++ String iamRoleArn() {+ return iamRoleArn != null ? iamRoleArn : credentialsRoleArn;+ }+ }++ /** The replacement is applied before records are scanned, so it must not change the framing. */+ private static boolean replacementIsFramingSyntax(char replacement, String delimiter, boolean csv) {+ return String.valueOf(replacement).equals(delimiter)+ || (!csv && replacement == '\\')+ || (csv && replacement == '"');}private static boolean matchClause(Matcher m, int start, int end) {@@ -495,6 +878,10 @@ public final class CopyStatementParser {return "\\t".equals(value) || "\t".equals(value) ? "\t" : value;}+ private static boolean isBlank(String value) {+ return value == null || value.isBlank();+ }+private static String extractNullValue(Matcher matcher) {String value = matcher.group(1) != null ? matcher.group(1) : matcher.group(2);return unescape(value);diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.javaindex 5b2c00c3f8..01752a1dc2 100644--- a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.java+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.java@@ -3,6 +3,7 @@ package io.github.hectorvent.floci.services.redshift.proxy;import io.github.hectorvent.floci.core.common.AwsException;import io.github.hectorvent.floci.services.iam.IamService;import io.github.hectorvent.floci.services.s3.S3Service;+import io.github.hectorvent.floci.services.s3.model.PutObjectOptions;import io.github.hectorvent.floci.services.s3.model.S3Object;import org.jboss.logging.Logger;@@ -14,7 +15,9 @@ import java.io.OutputStream;import java.net.Socket;import java.nio.charset.StandardCharsets;import java.util.ArrayList;+import java.util.HashMap;import java.util.List;+import java.util.Locale;import java.util.Map;import java.util.concurrent.Semaphore;import java.util.function.IntConsumer;@@ -47,6 +50,11 @@ public final class S3CopySimulator {static long UNLOAD_TARGET_FILE_BYTES = 6L * 1024 * 1024;/** Whole-result ceiling; a larger UNLOAD is aborted rather than filling the in-memory S3 store. */static long UNLOAD_MAX_TOTAL_BYTES = 256L * 1024 * 1024;+ /** Decompressed size ceiling for one object when a COPY field-content option forces it into heap. */+ static long COPY_TRANSFORM_MAX_OBJECT_BYTES = 64L * 1024 * 1024;+ /** Shared across every connection: a transform holds the input plus about two further copies of it. */+ static final Semaphore COPY_TRANSFORM_HEAP_MIB = new Semaphore(192);+ private static final int COPY_TRANSFORM_HEAP_FACTOR = 3;private static final long UNLOAD_WARN_BYTES = 32L * 1024 * 1024;/** Shared across every connection so concurrent UNLOADs cannot multiply the per-slice heap cost without bound. */static final Semaphore UNLOAD_HEAP_MIB = new Semaphore(192);@@ -172,7 +180,13 @@ public final class S3CopySimulator {}static void streamCopyInput(CopyInput input, List<String> discoveredColumns, OutputStream backendOut) throws IOException {- streamObjects(input.spec(), discoveredColumns, input.s3(), input.iamService(), input.roleSession(), input.keys(), backendOut);+ streamCopyInput(input, discoveredColumns, null, backendOut);+ }++ static void streamCopyInput(CopyInput input, List<String> discoveredColumns,+ List<Integer> columnMaxBytes, OutputStream backendOut) throws IOException {+ streamObjects(input.spec(), discoveredColumns, columnMaxBytes, input.s3(), input.iamService(),+ input.roleSession(), input.keys(), backendOut);}static void releaseCopySession(CopyInput input) {@@ -194,6 +208,7 @@ public final class S3CopySimulator {? RedshiftRoleAccess.resolveRoleSession(spec.iamRoleArn(), iamService, clusterAccountId, associatedRoleArns): null;String probeKey = unloadDataKey(spec, 0);+ boolean cleanPath = false;try {if (roleSession != null) {RedshiftRoleAccess.authorizeRoleAction(s3, iamService, spec.iamRoleArn(), "s3:PutObject",@@ -205,7 +220,7 @@ public final class S3CopySimulator {RedshiftRoleAccess.objectArn(spec.iamRoleArn(), spec.bucket(), manifestKey));s3.authorizeSignedPutObject(roleSession.accessKeyId(), roleSession.sessionToken(), spec.bucket(), manifestKey);}- if (!spec.allowOverwrite()) {+ if (!spec.allowOverwrite() && !spec.cleanPath()) {RedshiftRoleAccess.authorizeRoleAction(s3, iamService, spec.iamRoleArn(), "s3:ListBucket",RedshiftRoleAccess.bucketArn(spec.iamRoleArn(), spec.bucket()));s3.authorizeSignedListBucket(roleSession.accessKeyId(), roleSession.sessionToken(), spec.bucket());@@ -220,7 +235,7 @@ public final class S3CopySimulator {if (spec.manifest()) {s3.authorizeAnonymousPutObject(spec.bucket(), spec.prefix() + "manifest");}- if (!spec.allowOverwrite()) {+ if (!spec.allowOverwrite() && !spec.cleanPath()) {s3.authorizeAnonymousListBucket(spec.bucket());if (targetPrefixHasObjects(spec, s3)) {throw new S3TransferException(SQLSTATE_INTERNAL,@@ -229,6 +244,10 @@ public final class S3CopySimulator {}}}+ if (spec.cleanPath()) {+ collectCleanPathKeys(spec, s3, iamService, roleSession);+ cleanPath = true;+ }} catch (AwsException e) {RedshiftRoleAccess.releaseRoleSession(roleSession, spec.iamRoleArn(), iamService);throw new S3TransferException(unloadWriteSqlState(e), unloadWriteMessage(e, spec), e);@@ -243,7 +262,7 @@ public final class S3CopySimulator {"UNLOAD memory budget exhausted; retry shortly", null);}try {- return new S3UnloadCollector(spec, s3, iamService, roleSession);+ return new S3UnloadCollector(spec, s3, iamService, roleSession, cleanPath);} catch (IOException e) {UNLOAD_HEAP_MIB.release(UNLOAD_INITIAL_MIB);RedshiftRoleAccess.releaseRoleSession(roleSession, spec.iamRoleArn(), iamService);@@ -294,12 +313,22 @@ public final class S3CopySimulator {}try {+ boolean noColumnList = spec.columns() == null || spec.columns().isEmpty();+ boolean discoverForJson = spec.jsonAuto() && noColumnList;+ boolean truncate = spec.transforms().truncateColumns();List<String> discoveredColumns = null;- if (spec.jsonAuto() && (spec.columns() == null || spec.columns().isEmpty())) {- discoveredColumns = discoverTableColumns(client, backend, spec, txStatus, onStatusChange);- if (discoveredColumns == null) {+ List<Integer> columnMaxBytes = null;+ if (discoverForJson || truncate) {+ List<ColumnInfo> catalog = discoverColumnInfo(client, backend, spec, txStatus, onStatusChange);+ if (catalog == null) {return true;}+ if (discoverForJson) {+ discoveredColumns = catalog.stream().map(ColumnInfo::name).toList();+ }+ if (truncate) {+ columnMaxBytes = alignMaxBytes(spec, catalog);+ }}OutputStream backendOut = backend.getOutputStream();@@ -336,7 +365,7 @@ public final class S3CopySimulator {// a CopyFail to the backend, whose ErrorResponse/ReadyForQuery is relayed to the client;// or, if the backend is unreachable, one synthesized ErrorResponse/ReadyForQuery.try {- streamCopyInput(input, discoveredColumns, backendOut);+ streamCopyInput(input, discoveredColumns, columnMaxBytes, backendOut);writeCopyDone(backendOut);drainToReadyForQuery(backendDecoder, client, onStatusChange);} catch (RuntimeException | IOException e) {@@ -431,8 +460,15 @@ public final class S3CopySimulator {}private static void streamObjects(CopyStatementParser.S3CopyFrom spec, List<String> discoveredColumns,- S3Service s3, IamService iamService, RedshiftRoleAccess.RoleSession roleSession,+ List<Integer> columnMaxBytes, S3Service s3, IamService iamService, RedshiftRoleAccess.RoleSession roleSession,List<String> keys, OutputStream backendOut) throws IOException {+ if (spec.transforms().truncateColumns() && columnMaxBytes == null) {+ // Extended Query fixes the statement at Parse time, so there is no catalog round trip to+ // learn column lengths; same limitation and remedy as FORMAT AS JSON 'auto'.+ throw new S3TransferException(SQLSTATE_INTERNAL,+ "COPY ... TRUNCATECOLUMNS needs catalog column lengths, which are only available over "+ + "the Simple Query protocol; connect with preferQueryMode=simple", null);+ }byte[] buffer = new byte[CHUNK];for (int i = 0; i < keys.size(); i++) {if (roleSession != null) {@@ -459,10 +495,14 @@ public final class S3CopySimulator {if (i == 0 && spec.headerLines() > 0) {skipLines(in, spec.headerLines());}+ InputStream source = in;+ if (spec.transforms().any()) {+ source = new ByteArrayInputStream(transformObject(spec, columnMaxBytes, in));+ }int read;boolean endsWithNewline = false;boolean hasData = false;- while ((read = in.read(buffer)) != -1) {+ while ((read = source.read(buffer)) != -1) {if (read > 0) {hasData = true;endsWithNewline = (buffer[read - 1] == '\n');@@ -478,10 +518,87 @@ public final class S3CopySimulator {backendOut.flush();}- private static List<String> discoverTableColumns(Socket client, Socket backend,- CopyStatementParser.S3CopyFrom spec,- char txStatus, IntConsumer onStatusChange) throws IOException {- String query = "SELECT a.attname FROM pg_catalog.pg_attribute a "+ /**+ * Reads one decompressed object into heap and transforms it. The read is bounded by a per-object+ * ceiling and by a budget shared across connections, so a large or highly compressed object fails+ * its own statement instead of exhausting the emulator heap.+ */+ private static byte[] transformObject(CopyStatementParser.S3CopyFrom spec, List<Integer> columnMaxBytes,+ InputStream in) throws IOException {+ int heldMib = 0;+ try {+ ByteArrayOutputStream buffered = new ByteArrayOutputStream();+ byte[] chunk = new byte[CHUNK];+ long total = 0;+ int read;+ while ((read = in.read(chunk)) != -1) {+ total += read;+ if (total > COPY_TRANSFORM_MAX_OBJECT_BYTES) {+ throw new S3TransferException(SQLSTATE_PROGRAM_LIMIT_EXCEEDED,+ "COPY object exceeds the " + COPY_TRANSFORM_MAX_OBJECT_BYTES+ + "-byte limit for the field-level options "+ + "(EMPTYASNULL, BLANKSASNULL, REMOVEQUOTES, ACCEPTINVCHARS, TRUNCATECOLUMNS)", null);+ }+ int wantedMib = (int) ((total * COPY_TRANSFORM_HEAP_FACTOR) / (1024 * 1024)) + 1;+ while (heldMib < wantedMib) {+ if (!COPY_TRANSFORM_HEAP_MIB.tryAcquire(1)) {+ throw new S3TransferException(SQLSTATE_CONFIGURATION_LIMIT_EXCEEDED,+ "COPY memory budget exhausted; retry shortly", null);+ }+ heldMib++;+ }+ buffered.write(chunk, 0, read);+ }+ return new CopyRecordTransformer(spec, columnMaxBytes).apply(buffered.toByteArray());+ } finally {+ COPY_TRANSFORM_HEAP_MIB.release(heldMib);+ }+ }++ record ColumnInfo(String name, Integer maxBytes) {+ }++ /** Positional max character length per COPY column; null entries mean no limit. */+ static List<Integer> alignMaxBytes(CopyStatementParser.S3CopyFrom spec, List<ColumnInfo> catalog) {+ if (spec.columns() == null || spec.columns().isEmpty()) {+ return catalog.stream().map(ColumnInfo::maxBytes).toList();+ }+ Map<String, Integer> byName = new HashMap<>();+ for (ColumnInfo info : catalog) {+ byName.put(info.name(), info.maxBytes());+ }+ List<Integer> aligned = new ArrayList<>();+ for (String column : spec.columns()) {+ String key = column.startsWith("\"") ? unquoteIdentifier(column) : column.toLowerCase(Locale.ROOT);+ aligned.add(byName.get(key));+ }+ return aligned;+ }++ private static List<String> parseDataRow(byte[] body) {+ int count = ((body[0] & 0xFF) << 8) | (body[1] & 0xFF);+ List<String> values = new ArrayList<>(count);+ int offset = 2;+ for (int i = 0; i < count; i++) {+ int length = ((body[offset] & 0xFF) << 24) | ((body[offset + 1] & 0xFF) << 16)+ | ((body[offset + 2] & 0xFF) << 8) | (body[offset + 3] & 0xFF);+ offset += 4;+ if (length < 0) {+ values.add(null);+ continue;+ }+ values.add(new String(body, offset, length, StandardCharsets.UTF_8));+ offset += length;+ }+ return values;+ }++ private static List<ColumnInfo> discoverColumnInfo(Socket client, Socket backend,+ CopyStatementParser.S3CopyFrom spec,+ char txStatus, IntConsumer onStatusChange) throws IOException {+ // 1042 = bpchar, 1043 = varchar; atttypmod carries the declared length plus a 4-byte header.+ String query = "SELECT a.attname, CASE WHEN a.atttypid IN (1042, 1043) AND a.atttypmod > 4 "+ + "THEN a.atttypmod - 4 END FROM pg_catalog.pg_attribute a "+ "WHERE a.attrelid = to_regclass('" + quoteLiteral(spec.targetTable()) + "') "+ "AND a.attnum > 0 AND NOT a.attisdropped "+ "ORDER BY a.attnum";@@ -491,21 +608,17 @@ public final class S3CopySimulator {backendOut.flush();PostgresWireDecoder backendDecoder = new PostgresWireDecoder(backend.getInputStream());- List<String> cols = new ArrayList<>();+ List<ColumnInfo> cols = new ArrayList<>();PostgresWireDecoder.FrontendMessage msg;while ((msg = backendDecoder.nextMessage()) != null) {char type = msg.type();if (type == 'D') {- byte[] body = msg.body();- if (body.length >= 6) {- int colCount = ((body[0] & 0xFF) << 8) | (body[1] & 0xFF);- if (colCount >= 1) {- int colLen = ((body[2] & 0xFF) << 24) | ((body[3] & 0xFF) << 16)- | ((body[4] & 0xFF) << 8) | (body[5] & 0xFF);- if (colLen > 0 && 6 + colLen <= body.length) {- cols.add(new String(body, 6, colLen, StandardCharsets.UTF_8));- }- }+ List<String> values = parseDataRow(msg.body());+ if (!values.isEmpty() && values.get(0) != null) {+ Integer maxBytes = values.size() >= 2 && values.get(1) != null+ ? Integer.valueOf(values.get(1))+ : null;+ cols.add(new ColumnInfo(values.get(0), maxBytes));}} else if (type == 'E') {forward(client, msg);@@ -886,6 +999,80 @@ public final class S3CopySimulator {return true;}+ /**+ * Lists every object under the target prefix and authorizes the delete of each, so a denial fails+ * the UNLOAD before anything is removed or written. It runs at preparation, to fail early, and+ * again when the first output arrives, to delete what is there then. Nothing is deleted here, so+ * a query the backend rejects, or a failed memory reservation, leaves the previous export in place.+ */+ private static List<String> collectCleanPathKeys(CopyStatementParser.S3Unload spec, S3Service s3,+ IamService iamService,+ RedshiftRoleAccess.RoleSession roleSession) {+ authorizeCleanPathList(spec, s3, iamService, roleSession);+ List<String> keys = new ArrayList<>();+ String continuationToken = null;+ do {+ S3Service.ListObjectsResult result = s3.listObjectsWithPrefixes(+ spec.bucket(), spec.prefix(), null, LIST_PAGE_SIZE, continuationToken, null);+ if (result != null && result.objects() != null) {+ for (S3Object object : result.objects()) {+ keys.add(object.getKey());+ }+ }+ continuationToken = (result != null && result.isTruncated()) ? result.nextContinuationToken() : null;+ } while (continuationToken != null);++ for (String key : keys) {+ authorizeCleanPathDelete(spec, s3, iamService, roleSession, key);+ }+ return keys;+ }++ private static void authorizeCleanPathList(CopyStatementParser.S3Unload spec, S3Service s3,+ IamService iamService,+ RedshiftRoleAccess.RoleSession roleSession) {+ if (roleSession != null) {+ RedshiftRoleAccess.authorizeRoleAction(s3, iamService, spec.iamRoleArn(), "s3:ListBucket",+ RedshiftRoleAccess.bucketArn(spec.iamRoleArn(), spec.bucket()));+ s3.authorizeSignedListBucket(roleSession.accessKeyId(), roleSession.sessionToken(), spec.bucket());+ } else {+ s3.authorizeAnonymousListBucket(spec.bucket());+ }+ }++ private static void authorizeCleanPathDelete(CopyStatementParser.S3Unload spec, S3Service s3,+ IamService iamService,+ RedshiftRoleAccess.RoleSession roleSession, String key) {+ if (roleSession != null) {+ RedshiftRoleAccess.authorizeRoleAction(s3, iamService, spec.iamRoleArn(), "s3:DeleteObject",+ RedshiftRoleAccess.objectArn(spec.iamRoleArn(), spec.bucket(), key));+ s3.authorizeSignedDeleteObject(roleSession.accessKeyId(), roleSession.sessionToken(),+ spec.bucket(), key);+ } else {+ s3.authorizeAnonymousDeleteObject(spec.bucket(), key);+ }+ }++ /** Server-side encryption headers requested by ENCRYPTED KMS_KEY_ID, or null when none apply. */+ private static PutObjectOptions unloadObjectOptions(CopyStatementParser.S3Unload spec) {+ if (spec.sseKmsKeyId() == null) {+ return null;+ }+ return new PutObjectOptions()+ .withServerSideEncryption("aws:kms")+ .withSseKmsKeyId(spec.sseKmsKeyId());+ }++ private static void putUnloadObject(S3Service s3, CopyStatementParser.S3Unload spec, String key,+ byte[] payload, String contentType) {+ PutObjectOptions options = unloadObjectOptions(spec);+ if (options == null) {+ s3.putObject(spec.bucket(), key, payload, contentType, Map.of());+ } else {+ s3.putObject(spec.bucket(), key, payload, contentType, Map.of(), options);+ }+ }+private static boolean targetPrefixHasObjects(CopyStatementParser.S3Unload spec, S3Service s3) {S3Service.ListObjectsResult r = s3.listObjectsWithPrefixes(spec.bucket(), spec.prefix(), null, 1, null, null);@@ -920,6 +1107,8 @@ public final class S3CopySimulator {private final String contentType;private final List<String> writtenKeys = new ArrayList<>();private final List<Integer> writtenLengths = new ArrayList<>();+ private final boolean cleanPathPending;+ private boolean cleanPathDone;private ByteArrayOutputStream sink = new ByteArrayOutputStream();private OutputStream acc;@@ -938,8 +1127,10 @@ public final class S3CopySimulator {private int heldMib = UNLOAD_INITIAL_MIB;private S3UnloadCollector(CopyStatementParser.S3Unload spec, S3Service s3,- IamService iamService, RedshiftRoleAccess.RoleSession roleSession) throws IOException {+ IamService iamService, RedshiftRoleAccess.RoleSession roleSession,+ boolean cleanPath) throws IOException {this.spec = spec;+ this.cleanPathPending = cleanPath;this.s3 = s3;this.iamService = iamService;this.roleSession = roleSession;@@ -953,6 +1144,7 @@ public final class S3CopySimulator {@Overridepublic void accept(byte[] body) throws IOException {ensureOpen();+ runCleanPath();int dataStart = 0;if (capturingHeader) {int newline = -1;@@ -1001,6 +1193,7 @@ public final class S3CopySimulator {@Overridepublic void complete() throws IOException {ensureOpen();+ runCleanPath();byte[] payload = finishSlice(acc, sink);if (slicePayload > 0 || writtenKeys.isEmpty()) {writePayload(payload, sliceIndex);@@ -1015,10 +1208,10 @@ public final class S3CopySimulator {} else {s3.authorizeAnonymousPutObject(spec.bucket(), key);}- s3.putObject(spec.bucket(), key,+ putUnloadObject(s3, spec, key,manifestJson(spec.bucket(), writtenKeys, writtenLengths).getBytes(StandardCharsets.UTF_8),- "application/json", Map.of());+ "application/json");} catch (RuntimeException e) {fail(SQLSTATE_INTERNAL, "UNLOAD manifest write failed", e);}@@ -1052,6 +1245,25 @@ public final class S3CopySimulator {closed = true;}+ /**+ * CLEANPATH, second half: once, before the first write, removes what is under the prefix then.+ * S3 lists, authorizes every delete and deletes under one bucket lock, so an object uploaded+ * since preparation, or while this runs, does not survive next to the new export.+ */+ private void runCleanPath() {+ if (cleanPathDone || !cleanPathPending) {+ return;+ }+ cleanPathDone = true;+ try {+ authorizeCleanPathList(spec, s3, iamService, roleSession);+ s3.deleteObjectsWithPrefix(spec.bucket(), spec.prefix(),+ key -> authorizeCleanPathDelete(spec, s3, iamService, roleSession, key));+ } catch (RuntimeException e) {+ fail(SQLSTATE_INTERNAL, "UNLOAD CLEANPATH failed", e);+ }+ }+private void acquireForCurrentSlice() {int wantedMib = (int) ((rawSlice * 3) / (1024 * 1024)) + 1;while (heldMib < wantedMib) {@@ -1079,7 +1291,7 @@ public final class S3CopySimulator {} else {s3.authorizeAnonymousPutObject(spec.bucket(), key);}- s3.putObject(spec.bucket(), key, payload, contentType, Map.of());+ putUnloadObject(s3, spec, key, payload, contentType);} catch (AwsException e) {fail(unloadWriteSqlState(e), unloadWriteMessage(e, spec), e);} catch (RuntimeException e) {@@ -1134,6 +1346,11 @@ public final class S3CopySimulator {String base = spec.parallel()? spec.prefix() + String.format("%04d_part_00", index): spec.prefix() + String.format("%03d", index);+ if (spec.extension() != null) {+ // AWS adds the compression suffix only when no extension is given (its example is+ // EXTENSION 'txt.gz' GZIP), so the caller's extension is the whole suffix.+ return spec.extension().startsWith(".") ? base + spec.extension() : base + "." + spec.extension();+ }return spec.gzip() ? base + ".gz" : base;}diff --git a/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java b/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.javaindex bdae4fc76d..f64c627cd6 100644--- a/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java+++ b/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java@@ -113,6 +113,7 @@ public class S3Service implements Resettable, ResourceProvider {private static final Set<String> SUPPORTED_SERVER_SIDE_ENCRYPTION_VALUES = Set.of("AES256", "aws:kms", "aws:kms:dsse", "aws:fsx");private static final String SSE_C_ALGORITHM = "AES256";private static final int SSE_C_KEY_BYTES = 32;+ private static final int DELETE_PREFIX_PAGE_SIZE = 1000;@FunctionalInterfaceinterface LambdaInvoker {@@ -1915,6 +1916,37 @@ public class S3Service implements Resettable, ResourceProvider {}}+ /**+ * Deletes every object whose key starts with {@code prefix}. The bucket monitor is held across the+ * listing, the authorization of each key and the deletes, so a concurrent PutObject cannot land+ * between them and survive. {@code authorizeDelete} runs for every key before the first delete, so+ * a denial removes nothing.+ */+ public int deleteObjectsWithPrefix(String bucketName, String prefix, Consumer<String> authorizeDelete) {+ Bucket bucket = bucketStore.get(bucketName)+ .orElseThrow(() -> new AwsException("NoSuchBucket",+ "The specified bucket does not exist.", 404));+ synchronized (bucket) {+ List<String> keys = new ArrayList<>();+ String continuationToken = null;+ do {+ ListObjectsResult page = listObjectsWithPrefixes(+ bucketName, prefix, null, DELETE_PREFIX_PAGE_SIZE, continuationToken, null);+ for (S3Object object : page.objects()) {+ keys.add(object.getKey());+ }+ continuationToken = page.isTruncated() ? page.nextContinuationToken() : null;+ } while (continuationToken != null);+ for (String key : keys) {+ authorizeDelete.accept(key);+ }+ for (String key : keys) {+ deleteObject(bucketName, key);+ }+ return keys.size();+ }+ }+// A current delete marker answers 404 NoSuchKey, the same as a missing key. That is deliberate:// the S3 conditional-deletes guide says "If the latest version of the object is a delete marker,// the object doesn't exist and the DeleteObject API will fail and return a 412 Preconditiondiff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.javaindex 084bac753b..9ffeaadc7d 100644--- a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java@@ -186,6 +186,115 @@ class RedshiftInterceptorIntegrationTest {}}+ @Test+ void copyAcceptsRegionKeyCredentialsAndNullHandlingOptions() throws Exception {+ String bucket = "redshift-copy-options";+ s3.createBucket(bucket, "us-east-1");+ s3.putObject(bucket, "people/p1.txt",+ "1|alice\n2|\n3| \n".getBytes(StandardCharsets.UTF_8), "text/plain", Map.of());++ try (Connection connection = waitForConnection(sharedCluster, "admin", "Secret123");+ Statement statement = connection.createStatement()) {+ statement.execute("CREATE TABLE copy_opts_people (id int, name text)");+ statement.execute("COPY copy_opts_people FROM 's3://" + bucket + "/people/p1.txt' "+ + "ACCESS_KEY_ID 'test' SECRET_ACCESS_KEY 'test' REGION 'us-east-1' "+ + "EMPTYASNULL BLANKSASNULL");+ try (ResultSet rows = statement.executeQuery(+ "SELECT count(*) FROM copy_opts_people WHERE name IS NULL")) {+ assertTrue(rows.next());+ assertEquals(2, rows.getInt(1));+ }+ }+ }++ @Test+ void copyTruncateColumnsCutsOverlongValuesOverSimpleQuery() throws Exception {+ String bucket = "redshift-copy-truncate";+ s3.createBucket(bucket, "us-east-1");+ s3.putObject(bucket, "t.txt",+ "1|abcdef\n".getBytes(StandardCharsets.UTF_8), "text/plain", Map.of());++ try (Connection connection = waitForConnection(+ simpleQueryJdbcUrl(sharedCluster), "admin", "Secret123");+ Statement statement = connection.createStatement()) {+ statement.execute("CREATE TABLE copy_trunc (id int, name varchar(3))");+ statement.execute("COPY copy_trunc FROM 's3://" + bucket + "/t.txt' TRUNCATECOLUMNS");+ try (ResultSet rows = statement.executeQuery("SELECT name FROM copy_trunc")) {+ assertTrue(rows.next());+ assertEquals("abc", rows.getString(1));+ }+ }+ }++ @Test+ void copyTruncateColumnsOverExtendedQueryExplainsTheSimpleQueryRequirement() throws Exception {+ String bucket = "redshift-copy-truncate-ext";+ s3.createBucket(bucket, "us-east-1");+ s3.putObject(bucket, "t.txt",+ "1|abcdef\n".getBytes(StandardCharsets.UTF_8), "text/plain", Map.of());++ try (Connection connection = waitForConnection(sharedCluster, "admin", "Secret123");+ Statement statement = connection.createStatement()) {+ statement.execute("CREATE TABLE copy_trunc_ext (id int, name varchar(3))");+ SQLException failure = assertThrows(SQLException.class, () -> statement.execute(+ "COPY copy_trunc_ext FROM 's3://" + bucket + "/t.txt' TRUNCATECOLUMNS"));+ assertTrue(failure.getMessage().contains("preferQueryMode=simple"), failure.getMessage());+ }+ }++ @Test+ void unloadAppliesExtensionAndAcceptsRegion() throws Exception {+ String bucket = "redshift-unload-extension";+ s3.createBucket(bucket, "us-east-1");++ try (Connection connection = waitForConnection(sharedCluster, "admin", "Secret123");+ Statement statement = connection.createStatement()) {+ statement.execute("UNLOAD ('select 1') TO 's3://" + bucket + "/ext/' "+ + "REGION 'us-east-1' EXTENSION '.csv' ALLOWOVERWRITE");+ }++ List<S3Object> objects = s3.listObjects(bucket, "ext/", null, 100);+ assertEquals(1, objects.size());+ assertEquals("ext/0000_part_00.csv", objects.get(0).getKey());+ }++ @Test+ void unloadCleanPathRemovesOnlyObjectsUnderThePrefix() throws Exception {+ String bucket = "redshift-unload-cleanpath";+ s3.createBucket(bucket, "us-east-1");+ s3.putObject(bucket, "clean/old.txt", "old\n".getBytes(StandardCharsets.UTF_8), "text/plain", Map.of());+ s3.putObject(bucket, "clean2/keep.txt", "keep\n".getBytes(StandardCharsets.UTF_8), "text/plain", Map.of());++ try (Connection connection = waitForConnection(sharedCluster, "admin", "Secret123");+ Statement statement = connection.createStatement()) {+ statement.execute("UNLOAD ('select 1') TO 's3://" + bucket + "/clean/' CLEANPATH");+ }++ List<S3Object> cleaned = s3.listObjects(bucket, "clean/", null, 100);+ assertEquals(1, cleaned.size());+ assertEquals("clean/0000_part_00", cleaned.get(0).getKey());+ assertTrue(s3.objectExists(bucket, "clean2/keep.txt"));+ }++ @Test+ void unloadEscapeEscapesTheDelimiterInData() throws Exception {+ String bucket = "redshift-unload-escape";+ s3.createBucket(bucket, "us-east-1");++ try (Connection connection = waitForConnection(sharedCluster, "admin", "Secret123");+ Statement statement = connection.createStatement()) {+ statement.execute("CREATE TABLE unload_escape (v text)");+ statement.execute("INSERT INTO unload_escape VALUES ('a|b')");+ statement.execute("UNLOAD ('select v from unload_escape') TO 's3://" + bucket+ + "/esc/' ESCAPE ALLOWOVERWRITE");+ }++ List<S3Object> objects = s3.listObjects(bucket, "esc/", null, 100);+ assertEquals(1, objects.size());+ assertEquals("a\\|b\n", new String(+ s3.getObject(bucket, objects.get(0).getKey()).getData(), StandardCharsets.UTF_8));+ }+@Testvoid namedPreparedCopyAndUnloadCanBeExecutedTwice() throws Exception {Cluster cluster = sharedCluster;diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyRecordTransformerTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyRecordTransformerTest.javanew file mode 100644index 0000000000..01cce72510--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyRecordTransformerTest.java@@ -0,0 +1,178 @@+package io.github.hectorvent.floci.services.redshift.proxy;++import org.junit.jupiter.api.Test;++import java.nio.charset.StandardCharsets;+import java.util.Arrays;+import java.util.List;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertThrows;++class CopyRecordTransformerTest {++ private static CopyStatementParser.CopyTransforms transforms(+ boolean blanks, boolean empty, boolean removeQuotes, boolean truncate, Character invalid) {+ return new CopyStatementParser.CopyTransforms(blanks, empty, removeQuotes, truncate, invalid);+ }++ private static String apply(String delimiter, boolean csv, String nullAs,+ CopyStatementParser.CopyTransforms transforms, String input) {+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "t", List.of(), "b", "k", delimiter, 0, false, csv, nullAs, null,+ false, false, false, transforms);+ byte[] out = new CopyRecordTransformer(spec, null)+ .apply(input.getBytes(StandardCharsets.UTF_8));+ return new String(out, StandardCharsets.UTF_8);+ }++ @Test+ void emptyAsNullReplacesEmptyTextFields() {+ assertEquals("1|\\N|x\n2|b|\\N\n",+ apply("|", false, null, transforms(false, true, false, false, null), "1||x\n2|b|\n"));+ }++ @Test+ void blanksAsNullOnlyTouchesWhitespaceFields() {+ assertEquals("1|\\N|x\n2||y\n",+ apply("|", false, null, transforms(true, false, false, false, null), "1| |x\n2||y\n"));+ }++ @Test+ void customNullStringIsUsedAsMarker() {+ assertEquals("1|NULL|x\n",+ apply("|", false, "NULL", transforms(false, true, false, false, null), "1||x\n"));+ }++ @Test+ void removeQuotesKeepsDelimiterInsideQuotesAndEscapesItForPostgres() {+ assertEquals("1|a\\|b|c\n",+ apply("|", false, null, transforms(false, false, true, false, null), "1|\"a|b\"|c\n"));+ assertEquals("x\n",+ apply("|", false, null, transforms(false, false, true, false, null), "'x'\n"));+ }++ @Test+ void removeQuotesWithoutClosingQuoteFails() {+ assertThrows(S3CopySimulator.S3TransferException.class,+ () -> apply("|", false, null, transforms(false, false, true, false, null), "1|\"abc\n"));+ }++ @Test+ void csvQuotedEmptyFieldBecomesNullAndQuotedCommaSurvives() {+ assertEquals("1,,x\n",+ apply(",", true, null, transforms(false, true, false, false, null), "1,\"\",x\n"));+ assertEquals("1,\"a,b\",\n",+ apply(",", true, null, transforms(true, false, false, false, null), "1,\"a,b\", \n"));+ }++ @Test+ void crlfLineEndingsAreNormalisedAndLastFieldIsNotPolluted() {+ assertEquals("a|b\n1|\\N\n",+ apply("|", false, null, transforms(false, true, false, false, null), "a|b\r\n1|\r\n"));+ }++ @Test+ void lastRecordWithoutNewlineOrWithTrailingDelimiterIsTerminated() {+ assertEquals("1|\\N\n",+ apply("|", false, null, transforms(false, true, false, false, null), "1|"));+ assertEquals("1|b\n",+ apply("|", false, null, transforms(false, true, false, false, null), "1|b"));+ }++ @Test+ void backslashEscapedDelimiterStaysInsideItsField() {+ assertEquals("a\\|b|c\n",+ apply("|", false, null, transforms(false, true, false, false, null), "a\\|b|c\n"));+ }++ @Test+ void sanitizeUtf8ReplacesEachInvalidByteAndKeepsValidCharacters() {+ byte[] input = {'a', (byte) 0xFF, 'b', (byte) 0xC3, (byte) 0x28, (byte) 0xC3, (byte) 0xA9};+ assertEquals("a?b?(é",+ new String(CopyRecordTransformer.sanitizeUtf8(input, (byte) '?'), StandardCharsets.UTF_8));+ }++ private static String applyWithLimits(String delimiter, boolean csv, List<Integer> limits, String input) {+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "t", List.of(), "b", "k", delimiter, 0, false, csv, null, null,+ false, false, false, transforms(false, false, false, true, null));+ byte[] out = new CopyRecordTransformer(spec, limits).apply(input.getBytes(StandardCharsets.UTF_8));+ return new String(out, StandardCharsets.UTF_8);+ }++ @Test+ void truncateColumnsCutsOnlyLimitedColumns() {+ assertEquals("1|abc|xyzxyz\n",+ applyWithLimits("|", false, Arrays.asList(null, 3, null), "1|abcdef|xyzxyz\n"));+ }++ @Test+ void truncateColumnsNeverSplitsAMultiByteCharacter() {+ // The limit counts bytes, so the 2-byte e-acute does not fit twice into 3 bytes.+ assertEquals("a\n", applyWithLimits("|", false, List.of(2), "aébc\n"));+ assertEquals("é\n", applyWithLimits("|", false, List.of(3), "éééé\n"));+ assertEquals("éé\n", applyWithLimits("|", false, List.of(4), "ééé\n"));+ }++ @Test+ void truncateColumnsLeavesAValueThatFitsItsByteLimit() {+ assertEquals("éé\n", applyWithLimits("|", false, List.of(4), "éé\n"));+ }++ @Test+ void truncateColumnsKeepsCsvQuoting() {+ assertEquals("\"abc\",x\n", applyWithLimits(",", true, Arrays.asList(3, null), "\"abcdef\",x\n"));+ }++ @Test+ void truncateColumnsLeavesTheNullMarkerIntact() {+ assertEquals("\\N|x|y\n",+ applyWithLimits("|", false, Arrays.asList(1, 10, 10), "\\N|x|y\n"));+ }++ @Test+ void truncateColumnsKeepsAnEscapedBackslashWhole() {+ // Decoded value is ab, backslash, c. The 3-byte cut keeps the backslash, re-escaped.+ assertEquals("ab\\\\|hello\n",+ applyWithLimits("|", false, Arrays.asList(3, null), "ab\\\\c|hello\n"));+ }++ @Test+ void truncateColumnsCountsAnEscapedDelimiterAsOneByte() {+ assertEquals("a\\|b|x\n", applyWithLimits("|", false, Arrays.asList(3, null), "a\\|b|x\n"));+ }++ @Test+ void truncateColumnsNeverSplitsAnOctalEscape() {+ // Backslash 101, 102 and 103 decode to A, B and C.+ assertEquals("AB\n", applyWithLimits("|", false, List.of(2), "\\101\\102\\103\n"));+ }++ @Test+ void removeQuotesDoesNotCloseOnAnEscapedQuoteBeforeTheDelimiter() {+ assertEquals("a\\\"\\|b|x\n",+ apply("|", false, null, transforms(false, false, true, false, null), "\"a\\\"|b\"|x\n"));+ }++ @Test+ void removeQuotesRecognisesTheClosingQuoteBeforeCrlf() {+ assertEquals("a\nb\n",+ apply("|", false, null, transforms(false, false, true, false, null), "'a'\r\n'b'\n"));+ }++ @Test+ void csvQuotedLastFieldBeforeCrlfIsAccepted() {+ assertEquals("1,\"x\"\n",+ apply(",", true, null, transforms(false, true, false, false, null), "1,\"x\"\r\n"));+ }++ @Test+ void acceptInvalidCharsAloneDoesNotNeedFieldParsing() {+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "t", List.of(), "b", "k", "|", 0, false, false, null, null, false, false, false,+ transforms(false, false, false, false, '#'));+ byte[] out = new CopyRecordTransformer(spec, null).apply(new byte[]{'a', (byte) 0xFF, '\n'});+ assertEquals("a#\n", new String(out, StandardCharsets.UTF_8));+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.javaindex 75270541b0..e9b5e08b26 100644--- a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.java@@ -307,8 +307,8 @@ class CopyStatementParserTest {void unloadRejectsUnsupportedOptions() {assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/p/' PARQUET"));assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/p/' ENCRYPTED"));- assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/p/' REGION 'us-west-2'"));- assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/p/' EXTENSION 'csv'"));+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/p/' MASTER_SYMMETRIC_KEY 'k'"));+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/p/' KMS_KEY_ID 'k'"));assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/p/' ZSTD"));assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/p/' PARTITION BY (dt)"));}@@ -327,6 +327,185 @@ class CopyStatementParserTest {assertEquals("csv", u.nullAs());}+ @Test+ void copyAcceptsAndIgnoresRegion() {+ CopyStatementParser.S3CopyFrom c = copyFrom(+ "COPY t FROM 's3://b/k' REGION 'eu-west-1'");+ assertNotNull(c);+ assertNull(c.iamRoleArn());+ }++ @Test+ void copyMapsCredentialsIamRoleToIamRoleArn() {+ CopyStatementParser.S3CopyFrom c = copyFrom(+ "COPY t FROM 's3://b/k' CREDENTIALS 'aws_iam_role=arn:aws:iam::000000000000:role/r'");+ assertEquals("arn:aws:iam::000000000000:role/r", c.iamRoleArn());+ }++ @Test+ void copyAcceptsKeyCredentialsWithoutRole() {+ CopyStatementParser.S3CopyFrom c = copyFrom(+ "COPY t FROM 's3://b/k' ACCESS_KEY_ID 'test' SECRET_ACCESS_KEY 'test' SESSION_TOKEN 'test'");+ assertNotNull(c);+ assertNull(c.iamRoleArn());+ }++ @Test+ void copyAcceptsKeyCredentialsInsideCredentialsString() {+ CopyStatementParser.S3CopyFrom c = copyFrom(+ "COPY t FROM 's3://b/k' CREDENTIALS 'aws_access_key_id=test;aws_secret_access_key=test'");+ assertNotNull(c);+ assertNull(c.iamRoleArn());+ }++ @Test+ void copyRejectsConflictingOrIncompleteAuthorization() {+ assertNull(CopyStatementParser.parse(+ "COPY t FROM 's3://b/k' IAM_ROLE 'arn:aws:iam::000000000000:role/r' "+ + "CREDENTIALS 'aws_iam_role=arn:aws:iam::000000000000:role/r2'"));+ assertNull(CopyStatementParser.parse(+ "COPY t FROM 's3://b/k' IAM_ROLE 'arn:aws:iam::000000000000:role/r' "+ + "ACCESS_KEY_ID 'test' SECRET_ACCESS_KEY 'test'"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' ACCESS_KEY_ID 'test'"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' ACCESS_KEY_ID '' SECRET_ACCESS_KEY ''"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' ACCESS_KEY_ID 'test' SECRET_ACCESS_KEY ' '"));+ assertNull(CopyStatementParser.parse(+ "COPY t FROM 's3://b/k' ACCESS_KEY_ID 'test' SECRET_ACCESS_KEY 'test' SESSION_TOKEN ''"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' IAM_ROLE ''"));+ assertNull(CopyStatementParser.parse(+ "UNLOAD ('select 1') TO 's3://b/out/' ACCESS_KEY_ID '' SECRET_ACCESS_KEY ''"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' SESSION_TOKEN 'test'"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' REGION 'a' REGION 'b'"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' CREDENTIALS 'aws_access_key_id=test'"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' CREDENTIALS 'something_else=1'"));+ }++ @Test+ void copyStillRejectsEncryptionClauses() {+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' ENCRYPTED"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' MASTER_SYMMETRIC_KEY 'k'"));+ }++ @Test+ void unloadAcceptsRegionAndCredentials() {+ CopyStatementParser.S3Unload u = unload(+ "UNLOAD ('select 1') TO 's3://b/out/' REGION 'us-east-1' "+ + "CREDENTIALS 'aws_iam_role=arn:aws:iam::000000000000:role/r'");+ assertEquals("arn:aws:iam::000000000000:role/r", u.iamRoleArn());+ assertNotNull(unload("UNLOAD ('select 1') TO 's3://b/out/' ACCESS_KEY_ID 'test' SECRET_ACCESS_KEY 'test'"));+ }++ @Test+ void unloadParsesExtensionVerbatim() {+ CopyStatementParser.S3Unload u = unload(+ "UNLOAD ('select 1') TO 's3://b/out/' EXTENSION '.csv'");+ assertEquals(".csv", u.extension());+ assertNull(unload("UNLOAD ('select 1') TO 's3://b/out/'").extension());+ }++ @Test+ void unloadRejectsUnsafeOrRepeatedExtension() {+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' EXTENSION ''"));+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' EXTENSION 'a/b'"));+ assertNull(CopyStatementParser.parse(+ "UNLOAD ('select 1') TO 's3://b/out/' EXTENSION '.a' EXTENSION '.b'"));+ }++ @Test+ void copyParsesFieldContentOptions() {+ CopyStatementParser.S3CopyFrom c = copyFrom(+ "COPY t FROM 's3://b/k' EMPTYASNULL BLANKSASNULL REMOVEQUOTES ACCEPTINVCHARS AS '#'");+ assertTrue(c.transforms().emptyAsNull());+ assertTrue(c.transforms().blanksAsNull());+ assertTrue(c.transforms().removeQuotes());+ assertEquals('#', c.transforms().invalidCharReplacement());+ }++ @Test+ void copyParsesTruncateColumns() {+ assertTrue(copyFrom("COPY t FROM 's3://b/k' TRUNCATECOLUMNS").transforms().truncateColumns());+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' TRUNCATECOLUMNS TRUNCATECOLUMNS"));+ assertNull(CopyStatementParser.parse(+ "COPY t FROM 's3://b/k' FORMAT AS JSON 'auto' TRUNCATECOLUMNS"));+ }++ @Test+ void acceptInvCharsDefaultsToQuestionMark() {+ CopyStatementParser.S3CopyFrom c = copyFrom("COPY t FROM 's3://b/k' ACCEPTINVCHARS");+ assertEquals('?', c.transforms().invalidCharReplacement());+ assertEquals(CopyStatementParser.CopyTransforms.NONE, copyFrom("COPY t FROM 's3://b/k'").transforms());+ }++ @Test+ void acceptInvCharsReplacementCannotBeFramingSyntax() {+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' ACCEPTINVCHARS AS '|'"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' ACCEPTINVCHARS AS '\\'"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' CSV ACCEPTINVCHARS AS '\"'"));+ assertNotNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' ACCEPTINVCHARS AS '#'"));+ }++ @Test+ void copyRejectsInvalidTransformCombinations() {+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' CSV REMOVEQUOTES"));+ assertNull(CopyStatementParser.parse(+ "COPY t FROM 's3://b/k' FORMAT AS JSON 'auto' EMPTYASNULL"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' DELIMITER 'é' EMPTYASNULL"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' EMPTYASNULL EMPTYASNULL"));+ }++ @Test+ void unloadParsesEscapeOnlyWhenFramingStaysText() {+ assertTrue(unload("UNLOAD ('select 1') TO 's3://b/out/' ESCAPE").escape());+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' CSV ESCAPE"));+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' ADDQUOTES ESCAPE"));+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' HEADER ESCAPE"));+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' ESCAPE ESCAPE"));+ }++ @Test+ void unloadParsesCleanPathAndFailsOpenWhenItCouldBeUnsafe() {+ assertTrue(unload("UNLOAD ('select 1') TO 's3://b/out/' CLEANPATH").cleanPath());+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' CLEANPATH ALLOWOVERWRITE"));+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/' CLEANPATH"));+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' CLEANPATH CLEANPATH"));+ }++ @Test+ void unloadParsesServerSideEncryptionOptions() {+ CopyStatementParser.S3Unload kms = unload(+ "UNLOAD ('select 1') TO 's3://b/out/' ENCRYPTED KMS_KEY_ID 'key-1'");+ assertTrue(kms.encrypted());+ assertEquals("key-1", kms.sseKmsKeyId());++ CopyStatementParser.S3Unload reversed = unload(+ "UNLOAD ('select 1') TO 's3://b/out/' KMS_KEY_ID 'key-2' ENCRYPTED");+ assertEquals("key-2", reversed.sseKmsKeyId());++ CopyStatementParser.S3Unload auto = unload("UNLOAD ('select 1') TO 's3://b/out/' ENCRYPTED AUTO");+ assertTrue(auto.encrypted());+ assertNull(auto.sseKmsKeyId());+ assertFalse(unload("UNLOAD ('select 1') TO 's3://b/out/'").encrypted());+ }++ @Test+ void unloadEncryptionFailsOpenWhenItWouldNeedClientSideKeys() {+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' ENCRYPTED"));+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' KMS_KEY_ID 'k'"));+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' ENCRYPTED KMS_KEY_ID ' '"));+ assertNull(CopyStatementParser.parse(+ "UNLOAD ('select 1') TO 's3://b/out/' ENCRYPTED AUTO KMS_KEY_ID 'k'"));+ assertNull(CopyStatementParser.parse(+ "UNLOAD ('select 1') TO 's3://b/out/' ENCRYPTED MASTER_SYMMETRIC_KEY 'k'"));+ }++ @Test+ void unloadRejectsConflictingAuthorization() {+ assertNull(CopyStatementParser.parse(+ "UNLOAD ('select 1') TO 's3://b/out/' IAM_ROLE 'arn:aws:iam::000000000000:role/r' "+ + "ACCESS_KEY_ID 'test' SECRET_ACCESS_KEY 'test'"));+ assertNull(CopyStatementParser.parse("UNLOAD ('select 1') TO 's3://b/out/' REGION 'a' REGION 'b'"));+ }+@Testvoid parseCopy_jsonAuto_explicitColumns() {CopyStatementParser.S3CopyFrom c = copyFrom("COPY users (id, name) FROM 's3://mybucket/users.json' "diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.javaindex aad137b3c7..eddd2a7da1 100644--- a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java@@ -4,10 +4,13 @@ import io.github.hectorvent.floci.core.common.AwsException;import io.github.hectorvent.floci.services.iam.IamService;import io.github.hectorvent.floci.services.iam.model.IamRole;import io.github.hectorvent.floci.services.s3.S3Service;+import io.github.hectorvent.floci.services.s3.model.PutObjectOptions;import io.github.hectorvent.floci.services.s3.model.S3Object;import org.junit.jupiter.api.AfterEach;import org.junit.jupiter.api.BeforeEach;import org.junit.jupiter.api.Test;+import org.mockito.ArgumentCaptor;+import org.mockito.InOrder;import java.io.ByteArrayInputStream;import java.io.ByteArrayOutputStream;@@ -18,11 +21,13 @@ import java.net.Socket;import java.net.SocketTimeoutException;import java.nio.charset.StandardCharsets;import java.util.ArrayList;+import java.util.Arrays;import java.util.List;import java.util.Map;import java.util.Optional;import java.util.concurrent.ConcurrentHashMap;import java.util.concurrent.atomic.AtomicReference;+import java.util.function.Consumer;import java.util.zip.GZIPInputStream;import java.util.zip.GZIPOutputStream;@@ -36,7 +41,9 @@ import static org.mockito.ArgumentMatchers.any;import static org.mockito.ArgumentMatchers.anyInt;import static org.mockito.ArgumentMatchers.eq;import static org.mockito.ArgumentMatchers.isNull;+import static org.mockito.Mockito.doAnswer;import static org.mockito.Mockito.doThrow;+import static org.mockito.Mockito.inOrder;import static org.mockito.Mockito.mock;import static org.mockito.Mockito.never;import static org.mockito.Mockito.times;@@ -326,6 +333,152 @@ class S3CopySimulatorTest {S3CopySimulator.unloadBackendSql(spec));}+ private CopyStatementParser.S3Unload unloadSpecWithExtension(boolean gzip, boolean manifest, String extension) {+ return new CopyStatementParser.S3Unload("select a,b from t", "wh", "out/",+ "|", false, gzip, false, false, null, manifest, true, true, 0, null, extension);+ }++ private CopyStatementParser.S3Unload unloadSpecWithOptions(+ boolean manifest, boolean cleanPath, boolean encrypted, String kmsKeyId) {+ return new CopyStatementParser.S3Unload("select a,b from t", "wh", "out/", "|", false, false, false,+ false, null, manifest, false, true, 0, null, null, false, cleanPath, encrypted, kmsKeyId);+ }++ @Test+ void cleanPathDeletesOnlyObjectsUnderThePrefixBeforeWriting() throws Exception {+ S3Object old1 = new S3Object("wh", "out/old1", new byte[0], "text/plain");+ S3Object old2 = new S3Object("wh", "out/old2", new byte[0], "text/plain");+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("out/"), isNull(), anyInt(), any(), any()))+ .thenReturn(new S3Service.ListObjectsResult(List.of(old1, old2), List.of(), false, null));+ Map<String, byte[]> written = new ConcurrentHashMap<>();+ when(s3.putObject(eq("wh"), any(), any(), any(), any())).thenAnswer(inv -> {+ written.put(inv.getArgument(1), inv.getArgument(2));+ return null;+ });++ Thread backend = backendThread(() -> playUnloadBackend("1|alice\n"));+ S3CopySimulator.runUnload(simClient, simBackend, unloadSpecWithOptions(false, true, false, null),+ s3, null, 'I');+ joinBackend(backend);++ verify(s3).deleteObjectsWithPrefix(eq("wh"), eq("out/"), any());+ assertTrue(written.containsKey("out/0000_part_00"), written.keySet().toString());+ }++ @Test+ void cleanPathDeletesNothingUntilOutputArrivesAndDeletesBeforeTheFirstWrite() throws Exception {+ S3Object old1 = new S3Object("wh", "out/old1", new byte[0], "text/plain");+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("out/"), isNull(), anyInt(), any(), any()))+ .thenReturn(new S3Service.ListObjectsResult(List.of(old1), List.of(), false, null));++ try (S3CopySimulator.UnloadCollector collector =+ S3CopySimulator.prepareUnload(unloadSpecWithOptions(false, true, false, null), s3, null)) {+ verify(s3, never()).deleteObjectsWithPrefix(any(), any(), any());+ collector.complete();+ }++ InOrder order = inOrder(s3);+ order.verify(s3).deleteObjectsWithPrefix(eq("wh"), eq("out/"), any());+ order.verify(s3).putObject(eq("wh"), eq("out/0000_part_00"), any(), any(), any());+ }++ @Test+ void cleanPathAuthorizesEachDeleteThroughTheCallbackItHandsToS3() throws Exception {+ doAnswer(inv -> {+ Consumer<String> authorize = inv.getArgument(2);+ authorize.accept("out/late");+ return 1;+ }).when(s3).deleteObjectsWithPrefix(eq("wh"), eq("out/"), any());++ try (S3CopySimulator.UnloadCollector collector =+ S3CopySimulator.prepareUnload(unloadSpecWithOptions(false, true, false, null), s3, null)) {+ collector.complete();+ }++ verify(s3).authorizeAnonymousDeleteObject("wh", "out/late");+ }++ @Test+ void cleanPathKeepsObjectsWhenTheQueryNeverProducesOutput() throws Exception {+ S3Object old1 = new S3Object("wh", "out/old1", new byte[0], "text/plain");+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("out/"), isNull(), anyInt(), any(), any()))+ .thenReturn(new S3Service.ListObjectsResult(List.of(old1), List.of(), false, null));++ // Closed without complete(): the backend rejected the query, e.g. an unknown column.+ try (S3CopySimulator.UnloadCollector collector =+ S3CopySimulator.prepareUnload(unloadSpecWithOptions(false, true, false, null), s3, null)) {+ assertNotNull(collector);+ }++ verify(s3, never()).deleteObjectsWithPrefix(any(), any(), any());+ }++ @Test+ void cleanPathDeniedDeleteFailsBeforeAnythingIsWritten() {+ S3Object old1 = new S3Object("wh", "out/old1", new byte[0], "text/plain");+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("out/"), isNull(), anyInt(), any(), any()))+ .thenReturn(new S3Service.ListObjectsResult(List.of(old1), List.of(), false, null));+ doThrow(new AwsException("AccessDenied", "denied", 403))+ .when(s3).authorizeAnonymousDeleteObject("wh", "out/old1");++ assertThrows(S3CopySimulator.S3TransferException.class,+ () -> S3CopySimulator.prepareUnload(unloadSpecWithOptions(false, true, false, null), s3, null));++ verify(s3, never()).deleteObjectsWithPrefix(any(), any(), any());+ verify(s3, never()).putObject(any(), any(), any(), any(), any());+ }++ @Test+ void kmsKeyIdIsStoredOnDataAndManifestObjects() throws Exception {+ ArgumentCaptor<PutObjectOptions> options = ArgumentCaptor.forClass(PutObjectOptions.class);+ when(s3.putObject(eq("wh"), any(), any(), any(), any(), options.capture())).thenReturn(null);++ Thread backend = backendThread(() -> playUnloadBackend("1|alice\n"));+ S3CopySimulator.runUnload(simClient, simBackend, unloadSpecWithOptions(true, false, true, "key-1"),+ s3, null, 'I');+ joinBackend(backend);++ assertEquals(2, options.getAllValues().size());+ for (PutObjectOptions captured : options.getAllValues()) {+ assertEquals("aws:kms", captured.getServerSideEncryption());+ assertEquals("key-1", captured.getSseKmsKeyId());+ }+ }++ @Test+ void unloadAppendsExtensionToDataKeys() throws Exception {+ Map<String, byte[]> written = new ConcurrentHashMap<>();+ when(s3.putObject(eq("wh"), any(), any(), any(), any())).thenAnswer(inv -> {+ written.put(inv.getArgument(1), inv.getArgument(2));+ return null;+ });++ Thread backend = backendThread(() -> playUnloadBackend("1|alice\n"));+ S3CopySimulator.runUnload(simClient, simBackend, unloadSpecWithExtension(false, false, ".csv"),+ s3, null, 'I');+ joinBackend(backend);++ assertTrue(written.containsKey("out/0000_part_00.csv"), written.keySet().toString());+ }++ @Test+ void unloadWithExtensionDoesNotAddAGzipSuffixAndListsTheNameInManifest() throws Exception {+ Map<String, byte[]> written = new ConcurrentHashMap<>();+ when(s3.putObject(eq("wh"), any(), any(), any(), any())).thenAnswer(inv -> {+ written.put(inv.getArgument(1), inv.getArgument(2));+ return null;+ });++ Thread backend = backendThread(() -> playUnloadBackend("1|alice\n"));+ S3CopySimulator.runUnload(simClient, simBackend, unloadSpecWithExtension(true, true, "txt.gz"),+ s3, null, 'I');+ joinBackend(backend);++ assertTrue(written.containsKey("out/0000_part_00.txt.gz"), written.keySet().toString());+ String manifest = new String(written.get("out/manifest"), StandardCharsets.UTF_8);+ assertTrue(manifest.contains("s3://wh/out/0000_part_00.txt.gz"), manifest);+ }+@Testvoid unloadWritesSingleObjectAndForwardsCommandComplete() throws Exception {Map<String, byte[]> written = new ConcurrentHashMap<>();@@ -1364,6 +1517,107 @@ class S3CopySimulatorTest {assertEquals('d', rawBytes[0]);}+ private static String copyDataPayload(byte[] frames) {+ ByteArrayOutputStream payload = new ByteArrayOutputStream();+ int i = 0;+ while (i < frames.length) {+ int length = ((frames[i + 1] & 0xFF) << 24) | ((frames[i + 2] & 0xFF) << 16)+ | ((frames[i + 3] & 0xFF) << 8) | (frames[i + 4] & 0xFF);+ payload.write(frames, i + 5, length - 4);+ i += 1 + length;+ }+ return payload.toString(StandardCharsets.UTF_8);+ }++ @Test+ void streamCopyInput_appliesFieldTransforms() throws Exception {+ CopyStatementParser.CopyTransforms transforms =+ new CopyStatementParser.CopyTransforms(true, true, false, false, null);+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "people", List.of(), "wh", "p.txt", "|", 0, false, false, null, null,+ false, false, false, transforms);+ S3Object dataObj = new S3Object("wh", "p.txt",+ "1| \n2||x\n".getBytes(StandardCharsets.UTF_8), "text/plain");+ when(s3.getObject("wh", "p.txt")).thenReturn(dataObj);++ S3CopySimulator.CopyInput input = new S3CopySimulator.CopyInput(spec, List.of("p.txt"), s3, null, null);+ ByteArrayOutputStream out = new ByteArrayOutputStream();+ S3CopySimulator.streamCopyInput(input, out);++ assertEquals("1|\\N\n2|\\N|x\n", copyDataPayload(out.toByteArray()));+ }++ @Test+ void streamCopyInput_transformsAfterGzipDecodeAndHeaderSkip() throws Exception {+ CopyStatementParser.CopyTransforms transforms =+ new CopyStatementParser.CopyTransforms(false, true, false, false, null);+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "people", List.of(), "wh", "p.gz", "|", 1, true, false, null, null,+ false, false, false, transforms);+ ByteArrayOutputStream gz = new ByteArrayOutputStream();+ try (GZIPOutputStream gzip = new GZIPOutputStream(gz)) {+ gzip.write("id|name\n1|\n".getBytes(StandardCharsets.UTF_8));+ }+ when(s3.getObject("wh", "p.gz")).thenReturn(new S3Object("wh", "p.gz", gz.toByteArray(), "application/gzip"));++ S3CopySimulator.CopyInput input = new S3CopySimulator.CopyInput(spec, List.of("p.gz"), s3, null, null);+ ByteArrayOutputStream out = new ByteArrayOutputStream();+ S3CopySimulator.streamCopyInput(input, out);++ assertEquals("1|\\N\n", copyDataPayload(out.toByteArray()));+ }++ @Test+ void alignMaxBytes_followsExplicitColumnListAndFoldsUnquotedNames() {+ List<S3CopySimulator.ColumnInfo> catalog = List.of(+ new S3CopySimulator.ColumnInfo("id", null),+ new S3CopySimulator.ColumnInfo("name", 3),+ new S3CopySimulator.ColumnInfo("Note", 5));+ CopyStatementParser.S3CopyFrom explicit = new CopyStatementParser.S3CopyFrom(+ "t", List.of("NAME", "\"Note\""), "wh", "k", "|", 0, false, false, null, null);+ assertEquals(Arrays.asList(3, 5), S3CopySimulator.alignMaxBytes(explicit, catalog));++ CopyStatementParser.S3CopyFrom all = new CopyStatementParser.S3CopyFrom(+ "t", List.of(), "wh", "k", "|", 0, false, false, null, null);+ assertEquals(Arrays.asList(null, 3, 5), S3CopySimulator.alignMaxBytes(all, catalog));+ }++ @Test+ void streamCopyInput_withTruncateColumnsAndNoCatalogFailsAndNamesSimpleQueryMode() {+ CopyStatementParser.CopyTransforms transforms =+ new CopyStatementParser.CopyTransforms(false, false, false, true, null);+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "t", List.of(), "wh", "k", "|", 0, false, false, null, null, false, false, false, transforms);+ S3CopySimulator.CopyInput input = new S3CopySimulator.CopyInput(spec, List.of("k"), s3, null, null);++ S3CopySimulator.S3TransferException failure = assertThrows(S3CopySimulator.S3TransferException.class,+ () -> S3CopySimulator.streamCopyInput(input, new ByteArrayOutputStream()));+ assertTrue(failure.getMessage().contains("preferQueryMode=simple"), failure.getMessage());+ }++ @Test+ void streamCopyInput_withFieldTransformOverTheObjectLimitFailsTheStatementAndReleasesTheBudget() {+ CopyStatementParser.CopyTransforms transforms =+ new CopyStatementParser.CopyTransforms(true, false, false, false, null);+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "t", List.of(), "wh", "k", "|", 0, false, false, null, null, false, false, false, transforms);+ when(s3.getObject("wh", "k")).thenReturn(+ new S3Object("wh", "k", "1||x\n1||x\n".getBytes(StandardCharsets.UTF_8), "text/plain"));+ S3CopySimulator.CopyInput input = new S3CopySimulator.CopyInput(spec, List.of("k"), s3, null, null);++ long saved = S3CopySimulator.COPY_TRANSFORM_MAX_OBJECT_BYTES;+ int availableBefore = S3CopySimulator.COPY_TRANSFORM_HEAP_MIB.availablePermits();+ S3CopySimulator.COPY_TRANSFORM_MAX_OBJECT_BYTES = 8;+ try {+ S3CopySimulator.S3TransferException failure = assertThrows(S3CopySimulator.S3TransferException.class,+ () -> S3CopySimulator.streamCopyInput(input, new ByteArrayOutputStream()));+ assertTrue(failure.getMessage().contains("8-byte limit"), failure.getMessage());+ assertEquals(availableBefore, S3CopySimulator.COPY_TRANSFORM_HEAP_MIB.availablePermits());+ } finally {+ S3CopySimulator.COPY_TRANSFORM_MAX_OBJECT_BYTES = saved;+ }+ }+@Testvoid copyBackendSql_quotesDiscoveredColumnIdentifiers() {CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(diff --git a/src/test/java/io/github/hectorvent/floci/services/s3/S3ServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/s3/S3ServiceTest.javaindex 67490f6b5b..f60f35aba5 100644--- a/src/test/java/io/github/hectorvent/floci/services/s3/S3ServiceTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/s3/S3ServiceTest.java@@ -35,6 +35,10 @@ import java.util.HexFormat;import java.util.List;import java.util.Map;import java.util.Set;+import java.util.concurrent.CompletableFuture;+import java.util.concurrent.CountDownLatch;+import java.util.concurrent.FutureTask;+import java.util.concurrent.TimeUnit;import java.util.stream.Stream;import static org.junit.jupiter.api.Assertions.*;@@ -1634,4 +1638,72 @@ class S3ServiceTest {void authorizeSignedDeleteObjectIsANoOpWhenEnforceAuthIsOff() {assertDoesNotThrow(() -> s3Service.authorizeSignedDeleteObject("ASIAFAKEKEY00000001", "sessiontoken", "signed-del-bucket", "some/key"));}++ @Test+ void deleteObjectsWithPrefixRemovesOnlyTheMatchingKeys() {+ s3Service.createBucket("prefix-del", "us-east-1");+ s3Service.putObject("prefix-del", "out/a", new byte[]{1}, "text/plain", Map.of());+ s3Service.putObject("prefix-del", "out/b", new byte[]{2}, "text/plain", Map.of());+ s3Service.putObject("prefix-del", "keep/c", new byte[]{3}, "text/plain", Map.of());++ int deleted = s3Service.deleteObjectsWithPrefix("prefix-del", "out/", key -> { });++ assertEquals(2, deleted);+ assertFalse(s3Service.objectExists("prefix-del", "out/a"));+ assertFalse(s3Service.objectExists("prefix-del", "out/b"));+ assertTrue(s3Service.objectExists("prefix-del", "keep/c"));+ }++ @Test+ void deleteObjectsWithPrefixRemovesNothingWhenAnyKeyIsDenied() {+ s3Service.createBucket("prefix-deny", "us-east-1");+ s3Service.putObject("prefix-deny", "out/a", new byte[]{1}, "text/plain", Map.of());+ s3Service.putObject("prefix-deny", "out/b", new byte[]{2}, "text/plain", Map.of());++ assertThrows(AwsException.class, () -> s3Service.deleteObjectsWithPrefix("prefix-deny", "out/", key -> {+ if (key.equals("out/b")) {+ throw new AwsException("AccessDenied", "denied", 403);+ }+ }));++ assertTrue(s3Service.objectExists("prefix-deny", "out/a"));+ assertTrue(s3Service.objectExists("prefix-deny", "out/b"));+ }++ @Test+ void deleteObjectsWithPrefixHoldsOffAConcurrentPutUntilItIsDone() throws Exception {+ s3Service.createBucket("prefix-lock", "us-east-1");+ s3Service.putObject("prefix-lock", "out/old", new byte[]{1}, "text/plain", Map.of());+ CountDownLatch authorizing = new CountDownLatch(1);+ CountDownLatch release = new CountDownLatch(1);++ CompletableFuture<Integer> delete = CompletableFuture.supplyAsync(() ->+ s3Service.deleteObjectsWithPrefix("prefix-lock", "out/", key -> {+ authorizing.countDown();+ try {+ assertTrue(release.await(10, TimeUnit.SECONDS));+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();+ throw new IllegalStateException(e);+ }+ }));+ assertTrue(authorizing.await(10, TimeUnit.SECONDS));++ FutureTask<S3Object> put = new FutureTask<>(() ->+ s3Service.putObject("prefix-lock", "out/late", new byte[]{2}, "text/plain", Map.of()));+ Thread putThread = new Thread(put);+ putThread.start();+ long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10);+ while (putThread.getState() != Thread.State.BLOCKED && System.nanoTime() < deadline) {+ Thread.onSpinWait();+ }+ assertEquals(Thread.State.BLOCKED, putThread.getState());+ assertFalse(put.isDone());++ release.countDown();+ assertEquals(1, delete.get(10, TimeUnit.SECONDS));+ put.get(10, TimeUnit.SECONDS);+ assertFalse(s3Service.objectExists("prefix-lock", "out/old"));+ assertTrue(s3Service.objectExists("prefix-lock", "out/late"));+ }}