Light, fluffy, and always free - The AWS Local Emulator alternative
feat(redshift): emulate Redshift COPY FROM s3 over Simple Query protocol (#3100)
Vũ Thế Huy85180a4d88ae
diff --git a/docs/services/redshift.md b/docs/services/redshift.mdindex 01db4613f2..8d632a0085 100644--- a/docs/services/redshift.md+++ b/docs/services/redshift.md@@ -4,9 +4,9 @@**Management Endpoint:** `POST http://localhost:4566/` with `Action=` param**Data Endpoint:** Floci's auth proxy on the `Endpoint` and `Port` returned by `DescribeClusters` (PostgreSQL wire protocol)-Floci emulates Amazon Redshift by managing a real [PostgreSQL](https://www.postgresql.org/) Docker container per cluster behind a Redshift-shaped control plane. Each cluster sits behind a lightweight auth proxy on the Floci host, so the endpoint is reachable from outside Docker and the master password is validated at the proxy — a `ModifyCluster` password change takes effect for new connections immediately. Redshift speaks the PostgreSQL wire protocol, so the cluster endpoint returned by `DescribeClusters` works with any standard PostgreSQL driver (`psql`, JDBC, `psycopg`, …).+Floci emulates Amazon Redshift by managing a real [PostgreSQL](https://www.postgresql.org/) Docker container per cluster behind a Redshift-shaped control plane. Each cluster sits behind a lightweight auth proxy on the Floci host, so the endpoint is reachable from outside Docker and the master password is validated at the proxy, a `ModifyCluster` password change takes effect for new connections immediately. Redshift speaks the PostgreSQL wire protocol, so the cluster endpoint returned by `DescribeClusters` works with any standard PostgreSQL driver (`psql`, JDBC, `psycopg`, …).-> **Always read the host and port from `DescribeClusters`** rather than assuming a fixed port. PostgreSQL listens on `5432` *inside* the container; the port you connect to is dynamically assigned on the host and returned as `Clusters[0].Endpoint.Port`. Redshift's conventional port is `5439`, but the emulator does not bind it — use whatever `DescribeClusters` reports.+> **Always read the host and port from `DescribeClusters`** rather than assuming a fixed port. PostgreSQL listens on `5432` *inside* the container; the port you connect to is dynamically assigned on the host and returned as `Clusters[0].Endpoint.Port`. Redshift's conventional port is `5439`, but the emulator does not bind it: use whatever `DescribeClusters` reports.The container has **no persistent volume**: if the physical container survives a Floci restart it is adopted and its data is kept, but if the container itself is gone (host reboot, `docker rm`, a pruned dev box) the cluster comes back empty. Use `CreateClusterSnapshot` / `RestoreFromClusterSnapshot` to preserve data explicitly.@@ -106,7 +106,7 @@ aws redshift delete-cluster \```pythonimport psycopg-# Read host and port from DescribeClusters — the host port is dynamic.+# Read host and port from DescribeClusters: the host port is dynamic.host, port = "localhost", 32768 # e.g. Clusters[0].Endpoint.Address / .Portwith psycopg.connect(f"host={host} port={port} dbname=dev user=admin password=Secret123") as conn:conn.execute("CREATE TABLE people (name text)")@@ -145,18 +145,42 @@ Floci's Redshift auth proxy inspects frontend queries on the PostgreSQL wire pro- The rewrite only runs when the statement's first keyword is `CREATE TABLE` or `ALTER TABLE`. A `SELECT`, `INSERT`, function body, or string literal that merely contains one of these keywords is forwarded byte-for-byte. Single-quoted and dollar-quoted string literals are masked before the rewrite, so a keyword inside a quoted value (including in a later statement of a multi-statement query) is preserved.- Columns legitimately named `distkey`, `sortkey`, or `encode` survive.+### COPY from S3++`COPY <table> [(<columns>)] FROM 's3://<bucket>/<keyOrPrefix>' [options]` sent over the Simple+Query protocol is emulated: Floci reads the object (or every object under the prefix, in key+order) through its own S3 service and streams the rows into the backing PostgreSQL container with+`COPY ... FROM STDIN`.++- Supported options: `DELIMITER`, `FORMAT CSV` (or a bare `CSV`), `GZIP`, `IGNOREHEADER <n>` and+ `HEADER`, `NULL AS`, and an explicit column list.+- The default framing is pipe-delimited text, matching Redshift. `FORMAT CSV` switches to CSV with+ a comma default delimiter.+- `IGNOREHEADER` and `HEADER` skip lines from the first resolved object only.+- `GZIP` is the only input compression recognized; `BZIP2`, `LZOP` and `ZSTD` are not.+- S3 access is authorized as an unsigned request: with `FLOCI_SERVICES_S3_ENFORCE_AUTH` off it is+ unrestricted; with it on, bucket policy and public access settings apply.+- Any other clause (`FIXEDWIDTH`, `JSON`, `PARQUET`, `AVRO`, `ORC`, `MANIFEST`, `MAXERROR`,+ `DATEFORMAT`, `TIMEFORMAT`, `REGION`, `ENCODING`, `ESCAPE`, `REMOVEQUOTES`, `BLANKSASNULL`,+ `EMPTYASNULL`, `TRUNCATECOLUMNS`, `ACCEPTINVCHARS`, credentials clauses, and so on) is not+ recognized: the statement is forwarded unchanged and PostgreSQL returns its own error.+- A multi-statement query whose COPY is followed by another statement is not intercepted; send the+ COPY on its own.+- Extended Query protocol COPY (a JDBC `PreparedStatement`, or pgjdbc's default+ `preferQueryMode=extended`) is not intercepted. Use `preferQueryMode=simple`.+### Limitations- Emulation runs on the **Simple Query protocol** (`'Q'`) only. Extended Query protocol statements (`Parse`/`Bind`/`Execute`) pass through untouched, including anything a JDBC `PreparedStatement` sends, and, with the pgjdbc default `preferQueryMode=extended`, plain `Statement` calls too. Connect with `preferQueryMode=simple` to exercise the interceptor from JDBC.- The rewrite is textual (regex-based). It masks single-quoted string literals first, so `DEFAULT` / `CHECK` string values are safe, but it is **not** comment-aware and does not recognize escape strings (`E'...'`): an apostrophe inside a `--` or `/* */` comment can make the rewrite skip a Redshift clause. That fails safe: the statement then reaches PostgreSQL, which returns its own syntax error, but avoid apostrophes-in-comments in `CREATE TABLE` / `ALTER TABLE`.- A `rewrite` failure or any statement the interceptor does not recognize is forwarded unmodified (fail-open); PostgreSQL then rejects the Redshift-only syntax itself.- Simple Query ('Q') messages larger than 16 MiB bypass the interceptor and stream through verbatim without heap buffering; non-query traffic also streams through with no size limit.-- `COPY ... FROM 's3://...'` and `UNLOAD (...) TO 's3://...'` are **not** yet emulated (planned).+- `UNLOAD (...) TO 's3://...'` is **not** yet emulated (planned).## Out of Scope-- Real Redshift SQL semantics: the data plane is stock PostgreSQL. Redshift-only table DDL keywords (DISTSTYLE / DISTKEY / SORTKEY / ENCODE) are stripped so CREATE TABLE / ALTER TABLE executes (see [SQL Interceptor](#sql-interceptor)), but the distribution/sort behavior they request is not; COPY/UNLOAD from S3 and SUPER/SPECTRUM are not emulated.-- Multi-node clusters — `NodeType` and `NumberOfNodes` are stored as metadata; every cluster is a single PostgreSQL container.+- Real Redshift SQL semantics: the data plane is stock PostgreSQL. Redshift-only table DDL keywords (DISTSTYLE / DISTKEY / SORTKEY / ENCODE) are stripped so CREATE TABLE / ALTER TABLE executes (see [SQL Interceptor](#sql-interceptor)), but the distribution/sort behavior they request is not; UNLOAD to S3 and SUPER/SPECTRUM are not emulated.+- Multi-node clusters: `NodeType` and `NumberOfNodes` are stored as metadata; every cluster is a single PostgreSQL container.- Parameter groups apply no real engine settings; values are stored and echoed back only.- Subnet groups, VPC routing, and security groups are metadata only.- Resize, pause/resume, IAM authentication, snapshot schedules, and cross-region snapshot copy.diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.javanew file mode 100644index 0000000000..3c990300f1--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.java@@ -0,0 +1,275 @@+package io.github.hectorvent.floci.services.redshift.proxy;++import java.util.Arrays;+import java.util.List;+import java.util.regex.Matcher;+import java.util.regex.Pattern;++/**+ * Recognises a Simple Query statement that is an S3 {@code COPY <table> FROM 's3://...'}.+ * Any other statement, or a COPY that uses an option this simulator cannot honour, returns+ * {@code null} so the bridge falls back to DDL rewriting and PostgreSQL reports its own error.+ */+public final class CopyStatementParser {++ public record S3CopyFrom(+ String targetTable,+ List<String> columns,+ String bucket,+ String keyOrPrefix,+ String delimiter,+ int headerLines,+ boolean gzip,+ boolean csv,+ String nullAs) {+ }++ private static final Pattern COPY_PATTERN = Pattern.compile(+ "(?is)^\\s*COPY\\s+((?:\"[^\"]*\"|[^\\s(])+)\\s*(?:\\(([^)]*)\\))?\\s+FROM\\s*"+ + "['\"]s3://([^/'\"\\s]+)(?:/([^'\"\\s]*))?['\"]\\s*(.*)$");++ private static final Pattern QUALIFIED_NAME = Pattern.compile(+ "(?:[A-Za-z_][A-Za-z0-9_$]*|\"[^\"]+\")(?:\\.(?:[A-Za-z_][A-Za-z0-9_$]*|\"[^\"]+\"))?");+ private static final Pattern SIMPLE_NAME = Pattern.compile("[A-Za-z_][A-Za-z0-9_$]*|\"[^\"]+\"");++ private static final Pattern DELIMITER_PATTERN = Pattern.compile(+ "(?i)\\bDELIMITER\\s+(?:AS\\s+)?(?:'((?:[^']|'')*)'|\"([^\"]*)\"|([^\\s;]+))");+ private static final Pattern NULL_AS_PATTERN = Pattern.compile(+ "(?i)\\bNULL\\s+(?:AS\\s+)?(?:'((?:[^']|'')*)'|\"([^\"]*)\")");+ private static final Pattern IGNOREHEADER_PATTERN = Pattern.compile(+ "(?i)\\bIGNOREHEADER\\s+(?:AS\\s+)?(\\d+)\\b");+ private static final Pattern HEADER_PATTERN = Pattern.compile("(?i)\\bHEADER\\b");+ private static final Pattern GZIP_PATTERN = Pattern.compile("(?i)\\bGZIP\\b");+ private static final Pattern CSV_PATTERN = Pattern.compile("(?i)\\b(?:FORMAT\\s+(?:AS\\s+)?)?CSV\\b");++ /**+ * Options this simulator does not implement. A COPY carrying any of these is not intercepted:+ * the original statement is forwarded so PostgreSQL rejects it, rather than the simulator+ * silently loading the data with the wrong framing.+ */+ private static final Pattern UNSUPPORTED_CLAUSE = Pattern.compile(+ "(?i)\\b(FIXEDWIDTH|PARQUET|AVRO|ORC|JSON|SHAPEFILE|BZIP2|LZOP|ZSTD|MANIFEST|MAXERROR"+ + "|DATEFORMAT|TIMEFORMAT|ENCRYPTED|ENCODING|REGION|CREDENTIALS|IAM_ROLE"+ + "|ACCESS_KEY_ID|SECRET_ACCESS_KEY|SESSION_TOKEN|MASTER_SYMMETRIC_KEY|KMS_KEY_ID"+ + "|ACCEPTINVCHARS|ACCEPTANYDATE|BLANKSASNULL|EMPTYASNULL|FILLRECORD|TRIMBLANKS"+ + "|TRUNCATECOLUMNS|IGNOREBLANKLINES|ESCAPE|REMOVEQUOTES|EXPLICIT_IDS|COMPUPDATE"+ + "|STATUPDATE|NOLOAD|ROUNDEC|QUOTE|SSH|READRATIO|COMPROWS|DIMENSION)\\b");++ /** A {@code ;} followed by another statement: only a lone trailing {@code ;} is tolerated. */+ private static final Pattern TRAILING_STATEMENT = Pattern.compile(";\\s*\\S");++ private CopyStatementParser() {+ }++ public static S3CopyFrom parse(String sql) {+ if (sql == null || sql.isBlank()) {+ return null;+ }+ String cleaned = stripLeadingComments(sql.trim());+ Matcher matcher = COPY_PATTERN.matcher(cleaned);+ if (!matcher.matches()) {+ return null;+ }++ String table = matcher.group(1).trim();+ if (!QUALIFIED_NAME.matcher(table).matches()) {+ return null;+ }++ List<String> columns = List.of();+ String columnsGroup = matcher.group(2);+ if (columnsGroup != null && !columnsGroup.isBlank()) {+ columns = Arrays.stream(columnsGroup.split(","))+ .map(String::trim)+ .filter(s -> !s.isEmpty())+ .toList();+ if (columns.stream().anyMatch(c -> !SIMPLE_NAME.matcher(c).matches())) {+ return null;+ }+ }++ String bucket = matcher.group(3);+ String keyOrPrefix = matcher.group(4) != null ? matcher.group(4) : "";+ String options = matcher.group(5) != null ? matcher.group(5) : "";++ // Scan for keywords and separators on a copy with string literals blanked out, so a+ // value like NULL AS 'json' or DELIMITER ';' cannot trip a keyword or the trailing check.+ String flagScan = blankQuoted(options);+ if (UNSUPPORTED_CLAUSE.matcher(flagScan).find() || TRAILING_STATEMENT.matcher(flagScan).find()) {+ return null;+ }++ int semiIdx = flagScan.indexOf(';');+ if (semiIdx >= 0) {+ if (!flagScan.substring(semiIdx + 1).isBlank()) {+ return null;+ }+ options = options.substring(0, semiIdx);+ }++ boolean csv = false;+ boolean gzip = false;+ String nullAs = null;+ String delimiter = null;+ int headerLines = 0;++ boolean seenCsv = false;+ boolean seenGzip = false;+ boolean seenNull = false;+ boolean seenDelimiter = false;+ boolean seenHeader = false;++ Matcher csvMatcher = CSV_PATTERN.matcher(options);+ Matcher gzipMatcher = GZIP_PATTERN.matcher(options);+ Matcher ignoreHeaderMatcher = IGNOREHEADER_PATTERN.matcher(options);+ Matcher headerMatcher = HEADER_PATTERN.matcher(options);+ Matcher delimiterMatcher = DELIMITER_PATTERN.matcher(options);+ Matcher nullMatcher = NULL_AS_PATTERN.matcher(options);++ int offset = 0;+ int len = options.length();+ while (offset < len) {+ while (offset < len && Character.isWhitespace(options.charAt(offset))) {+ offset++;+ }+ if (offset >= len) {+ break;+ }++ if (matchClause(csvMatcher, offset, len)) {+ if (seenCsv) {+ return null;+ }+ seenCsv = true;+ csv = true;+ offset = csvMatcher.end();+ } else if (matchClause(gzipMatcher, offset, len)) {+ if (seenGzip) {+ return null;+ }+ seenGzip = true;+ gzip = true;+ offset = gzipMatcher.end();+ } else if (matchClause(ignoreHeaderMatcher, offset, len)) {+ if (seenHeader) {+ return null;+ }+ seenHeader = true;+ headerLines = Math.max(0, Integer.parseInt(ignoreHeaderMatcher.group(1)));+ offset = ignoreHeaderMatcher.end();+ } else if (matchClause(headerMatcher, offset, len)) {+ if (seenHeader) {+ return null;+ }+ seenHeader = true;+ headerLines = 1;+ offset = headerMatcher.end();+ } else if (matchClause(delimiterMatcher, offset, len)) {+ if (seenDelimiter) {+ return null;+ }+ seenDelimiter = true;+ delimiter = extractDelimiterValue(delimiterMatcher);+ offset = delimiterMatcher.end();+ } else if (matchClause(nullMatcher, offset, len)) {+ if (seenNull) {+ return null;+ }+ seenNull = true;+ nullAs = extractNullValue(nullMatcher);+ offset = nullMatcher.end();+ } else {+ return null;+ }+ }++ if (delimiter == null) {+ delimiter = csv ? "," : "|";+ }++ return new S3CopyFrom(table, columns, bucket, keyOrPrefix, delimiter, headerLines, gzip, csv, nullAs);+ }++ private static boolean matchClause(Matcher m, int start, int end) {+ m.region(start, end);+ return m.lookingAt();+ }++ private static String extractDelimiterValue(Matcher matcher) {+ String value = matcher.group(1) != null ? matcher.group(1)+ : matcher.group(2) != null ? matcher.group(2)+ : matcher.group(3);+ if (value != null) {+ value = unescape(value);+ }+ return "\\t".equals(value) || "\t".equals(value) ? "\t" : value;+ }++ private static String extractNullValue(Matcher matcher) {+ String value = matcher.group(1) != null ? matcher.group(1) : matcher.group(2);+ return unescape(value);+ }++ private static String unescape(String s) {+ if (s == null) {+ return null;+ }+ return s.replace("''", "'").replace("\\\\", "\\");+ }++ /** Replace every character inside a single- or double-quoted run with a space. */+ private static String blankQuoted(String s) {+ StringBuilder out = new StringBuilder(s.length());+ boolean inSingle = false;+ boolean inDouble = false;+ for (int i = 0; i < s.length(); i++) {+ char c = s.charAt(i);+ if (inSingle) {+ if (c == '\'') {+ if (i + 1 < s.length() && s.charAt(i + 1) == '\'') {+ out.append(" ");+ i++;+ continue;+ }+ inSingle = false;+ }+ out.append(' ');+ } else if (inDouble) {+ if (c == '"') {+ inDouble = false;+ }+ out.append(' ');+ } else if (c == '\'') {+ inSingle = true;+ out.append(' ');+ } else if (c == '"') {+ inDouble = true;+ out.append(' ');+ } else {+ out.append(c);+ }+ }+ return out.toString();+ }++ private static String stripLeadingComments(String sql) {+ String current = sql;+ while (true) {+ if (current.startsWith("--")) {+ int newline = current.indexOf('\n');+ if (newline < 0) {+ return "";+ }+ current = current.substring(newline + 1).stripLeading();+ } else if (current.startsWith("/*")) {+ int close = current.indexOf("*/");+ if (close < 0) {+ return "";+ }+ current = current.substring(close + 2).stripLeading();+ } else {+ return current;+ }+ }+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftAuthProxy.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftAuthProxy.javaindex 7819377f1e..a2591315c1 100644--- a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftAuthProxy.java+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftAuthProxy.java@@ -4,6 +4,7 @@ import io.github.hectorvent.floci.services.rds.proxy.PostgresProtocolHandler;import io.github.hectorvent.floci.services.rds.proxy.RdsAuthProxy;import io.github.hectorvent.floci.services.rds.proxy.RdsProxyTlsCertificates;import io.github.hectorvent.floci.services.rds.proxy.RdsSigV4Validator;+import io.github.hectorvent.floci.services.s3.S3Service;import org.jboss.logging.Logger;import java.io.IOException;@@ -32,6 +33,7 @@ public class RedshiftAuthProxy {private final RdsSigV4Validator sigV4;private final RdsProxyTlsCertificates tlsCertificates;private final RdsAuthProxy.PasswordValidator passwordValidator;+ private final S3Service s3Service;private volatile boolean running;private ServerSocket serverSocket;@@ -39,7 +41,8 @@ public class RedshiftAuthProxy {public RedshiftAuthProxy(String clusterKey, String backendHost, int backendPort,String masterUsername, String masterPassword, String dbName,RdsSigV4Validator sigV4, RdsProxyTlsCertificates tlsCertificates,- RdsAuthProxy.PasswordValidator passwordValidator) {+ RdsAuthProxy.PasswordValidator passwordValidator,+ S3Service s3Service) {this.clusterKey = clusterKey;this.backendHost = backendHost;this.backendPort = backendPort;@@ -49,6 +52,7 @@ public class RedshiftAuthProxy {this.sigV4 = sigV4;this.tlsCertificates = tlsCertificates;this.passwordValidator = passwordValidator;+ this.s3Service = s3Service;}public void start(int proxyPort) throws IOException {@@ -145,7 +149,7 @@ public class RedshiftAuthProxy {if (session != null) {// Redshift-only DDL (DISTKEY/SORTKEY/ENCODE/...) is rewritten for the plain// PostgreSQL backend on the way through; every other message is relayed verbatim.- new RedshiftInterceptingBridge(session.client(), backend).run();+ new RedshiftInterceptingBridge(session.client(), backend, s3Service).run();}} catch (Exception e) {LOG.debugv("Redshift connection error for cluster {0}: {1}", clusterKey, e.getMessage());diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftInterceptingBridge.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftInterceptingBridge.javaindex c554f873b8..85d495e4db 100644--- a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftInterceptingBridge.java+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftInterceptingBridge.java@@ -1,117 +1,292 @@package io.github.hectorvent.floci.services.redshift.proxy;+import io.github.hectorvent.floci.services.s3.S3Service;import org.jboss.logging.Logger;import java.io.IOException;import java.io.InputStream;import java.io.OutputStream;import java.net.Socket;-import java.util.concurrent.atomic.AtomicBoolean;+import java.net.SocketTimeoutException;+import java.util.concurrent.TimeUnit;+import java.util.concurrent.atomic.AtomicInteger;+import java.util.concurrent.locks.LockSupport;+import java.util.concurrent.locks.ReentrantLock;+import java.util.function.IntConsumer;/**- * Replaces the transparent {@code PostgresProtocolHandler.bridge} for Redshift connections so- * Simple Query ({@code 'Q'}) DDL can be rewritten for a plain PostgreSQL backend before it is- * forwarded.+ * Replaces the transparent {@code bridge()} for Redshift connections so Simple Query ({@code 'Q'})+ * traffic can be inspected before it reaches the backing PostgreSQL container.*- * <p><b>Model.</b> Backend→client is a verbatim byte pump on a virtual thread, exactly as a- * plain proxy: extended-protocol pipelines, {@code COPY … FROM STDIN}, asynchronous- * {@code NotificationResponse} and backend EOF all flow through untouched. Client→backend is- * a framed loop: every frontend message is forwarded opaque <em>except</em> a {@code 'Q'}, whose- * SQL is run through {@link RedshiftSqlInterceptor#rewrite} and re-encoded only if it changed.+ * <p><b>Model.</b> Backend→client is a byte pump on a virtual thread. Client→backend is a+ * framed loop: every frontend message is forwarded opaque except a {@code 'Q'}, whose SQL is run+ * through {@link RedshiftSqlInterceptor#rewrite} and re-encoded only if it changed.*- * <p><b>Fail-open.</b> A {@code rewrite} failure, or any statement the interceptor leaves- * untouched, forwards the original bytes; PostgreSQL then returns its own error. The bridge never- * closes the connection because of a rewrite failure.+ * <p><b>Backend ownership.</b> A future interceptor that injects its own query and reads the reply+ * (the S3 COPY simulator) must own the backend exclusively and only start when no earlier request+ * is still being answered. {@link #backendLock} serialises pump reads against such an exchange, and+ * {@link #outstandingResponses} plus {@link #pumpBetweenMessages} gate it to a safe boundary. If the+ * backend cannot be caught idle within {@link #PUMP_PARK_WAIT_MS}, the exchange is skipped and the+ * original {@code 'Q'} is forwarded (fail-open).*/public class RedshiftInterceptingBridge {private static final Logger LOG = Logger.getLogger(RedshiftInterceptingBridge.class);+ private static final int PUMP_READ_TIMEOUT_MS = 200;+ private static final int CLIENT_READ_TIMEOUT_MS = 10_000;+ private static final long PUMP_PARK_WAIT_MS = 2_000L;+ /** Bounded read timeout while an exchange owns the backend, so a stalled backend cannot hang the session. */+ private static final int EXCHANGE_READ_TIMEOUT_MS = 60_000;+ private static final long BUSY_BACKOFF_NANOS = 2_000_000L;+private final Socket client;private final Socket backend;- private final AtomicBoolean closed = new AtomicBoolean();+ private final S3Service s3Service;++ private final ReentrantLock backendLock = new ReentrantLock(true);+ private volatile boolean pumpBetweenMessages = true;+ private final AtomicInteger outstandingResponses = new AtomicInteger(0);+ private volatile boolean pumpFinished = false;+ /** Transaction-status byte from the backend's most recent {@code ReadyForQuery}; {@code 'I'} until one is seen. */+ private volatile char lastBackendStatus = 'I';- public RedshiftInterceptingBridge(Socket client, Socket backend) {+ public RedshiftInterceptingBridge(Socket client, Socket backend, S3Service s3Service) {this.client = client;this.backend = backend;+ this.s3Service = s3Service;+ }++ @FunctionalInterface+ interface BackendExchange {+ /** @return {@code true} if the exchange handled the query; {@code false} to forward the original. */+ boolean run() throws IOException;}public void run() {try {- InputStream clientIn = client.getInputStream();- OutputStream backendOut = backend.getOutputStream();-+ backend.setSoTimeout(PUMP_READ_TIMEOUT_MS);+ client.setSoTimeout(CLIENT_READ_TIMEOUT_MS);Thread.ofVirtual().name("redshift-pump-backend-to-client").start(this::pumpBackendToClient);+ InputStream clientIn = client.getInputStream();+ OutputStream backendOut = backend.getOutputStream();PostgresWireDecoder decoder = new PostgresWireDecoder(clientIn);+while (true) {- PostgresWireDecoder.FrontendMessage msg = decoder.nextMessage(backendOut);+ PostgresWireDecoder.FrontendMessage msg;+ try {+ msg = decoder.nextMessage();+ } catch (SocketTimeoutException e) {+ if (decoder.isBetweenMessages()) {+ continue;+ }+ LOG.warnv("Client socket timed out mid-message: {0}", e.getMessage());+ break;+ }if (msg == null) {- break; // client EOF+ break;}if (!msg.isQuery()) {- if (msg.type() == 'X') { // Terminate+ if (msg.type() == 'S') {+ outstandingResponses.incrementAndGet();+ }+ backendOut.write(msg.toPacketBytes());+ backendOut.flush();+ if (msg.type() == 'X') {break;}continue;}String sql = msg.getSql();++ CopyStatementParser.S3CopyFrom copyFrom = null;+ try {+ copyFrom = CopyStatementParser.parse(sql);+ } catch (RuntimeException e) {+ LOG.warnv("CopyStatementParser failed, forwarding original query: {0}", e.getMessage());+ }++ if (copyFrom != null) {+ CopyStatementParser.S3CopyFrom spec = copyFrom;+ boolean intercepted = runWithBackendOwned(+ () -> S3CopySimulator.runCopyFrom(client, backend, spec, s3Service, lastBackendStatus,+ status -> lastBackendStatus = (char) status));+ if (intercepted) {+ continue;+ }+ }+byte[] toBackend = msg.toPacketBytes();try {String rewritten = RedshiftSqlInterceptor.rewrite(sql);- if (rewritten != sql) { // identity: rewrite returns the same instance when nothing matched+ if (rewritten != sql) {toBackend = PostgresWireDecoder.encodeQuery(rewritten);}} catch (RuntimeException e) {LOG.warnv("RedshiftSqlInterceptor failed, forwarding original query: {0}", e.getMessage());toBackend = msg.toPacketBytes();}+ outstandingResponses.incrementAndGet();backendOut.write(toBackend);backendOut.flush();- // The response returns over the untouched pump.}} catch (IOException e) {LOG.debugv(e, "RedshiftInterceptingBridge client loop ended");} catch (Exception e) {LOG.warnv(e, "Unexpected error in RedshiftInterceptingBridge");} finally {- closeBoth();+ closeQuietly(client, "client");+ closeQuietly(backend, "backend");+ }+ }++ /**+ * Take exclusive ownership of an idle backend at a wire-message boundary, run {@code exchange},+ * then release it. Returns {@code false} if the backend could not be caught idle within+ * {@link #PUMP_PARK_WAIT_MS}; the caller must then forward the original {@code 'Q'} unmodified.+ */+ boolean runWithBackendOwned(BackendExchange exchange) throws IOException {+ long deadlineNanos = System.nanoTime() + PUMP_PARK_WAIT_MS * 1_000_000L;+ while (true) {+ if (pumpFinished) {+ return false;+ }+ boolean locked;+ try {+ locked = backendLock.tryLock(PUMP_READ_TIMEOUT_MS, TimeUnit.MILLISECONDS);+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();+ throw new IOException("interrupted while acquiring the backend lock", e);+ }+ if (!locked) {+ if (System.nanoTime() >= deadlineNanos) {+ LOG.warn("could not take the backend stream in time; forwarding the COPY unintercepted");+ return false;+ }+ continue;+ }+ boolean backendBusy;+ try {+ backendBusy = !pumpBetweenMessages || outstandingResponses.get() > 0;+ if (!backendBusy || pumpFinished) {+ backend.setSoTimeout(EXCHANGE_READ_TIMEOUT_MS);+ try {+ return exchange.run();+ } finally {+ try {+ backend.setSoTimeout(PUMP_READ_TIMEOUT_MS);+ } catch (IOException e) {+ LOG.debugv(e, "could not restore backend read timeout (socket closing)");+ }+ }+ }+ } finally {+ backendLock.unlock();+ }+ if (System.nanoTime() >= deadlineNanos) {+ LOG.warn("backend did not go idle in time; forwarding the COPY unintercepted");+ return false;+ }+ LockSupport.parkNanos(BUSY_BACKOFF_NANOS);}}private void pumpBackendToClient() {+ WireFrameTracker tracker = new WireFrameTracker(status -> {+ lastBackendStatus = (char) status;+ outstandingResponses.updateAndGet(v -> Math.max(0, v - 1));+ });try {InputStream backendIn = backend.getInputStream();OutputStream clientOut = client.getOutputStream();- byte[] buf = new byte[8192];- int n;- while ((n = backendIn.read(buf)) != -1) {- clientOut.write(buf, 0, n);- clientOut.flush();+ byte[] buffer = new byte[8192];+ while (true) {+ backendLock.lockInterruptibly();+ try {+ int read;+ try {+ read = backendIn.read(buffer);+ } catch (SocketTimeoutException e) {+ continue;+ }+ if (read == -1) {+ break;+ }+ clientOut.write(buffer, 0, read);+ clientOut.flush();+ tracker.consume(buffer, 0, read);+ pumpBetweenMessages = tracker.betweenMessages();+ } finally {+ backendLock.unlock();+ }}} catch (IOException e) {LOG.debugv(e, "backend->client pump ended");+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();} finally {- closeBoth();- }- }-- private void closeBoth() {- if (closed.compareAndSet(false, true)) {+ pumpFinished = true;closeQuietly(client, "client");closeQuietly(backend, "backend");}}- private void closeQuietly(Socket s, String which) {+ private void closeQuietly(Socket socket, String which) {try {- if (s != null && !s.isClosed()) {- s.close();+ if (socket != null && !socket.isClosed()) {+ socket.close();}} catch (IOException e) {LOG.debugv(e, "error closing {0} socket", which);}}++ /**+ * Tracks PostgreSQL wire-message boundaries in a byte stream without buffering it. Each backend+ * message is {@code type(1) . length(int32, includes itself) . body}. {@link #betweenMessages()}+ * is true exactly when the next byte would begin a new message; {@code onReadyForQuery} fires+ * once per completed {@code 'Z'} with its one-byte transaction-status payload.+ */+ static final class WireFrameTracker {+ private final IntConsumer onReadyForQuery;+ private int headerBytesSeen = 0;+ private final byte[] header = new byte[5];+ private long bodyRemaining = 0;+ private char pendingType = 0;++ WireFrameTracker(IntConsumer onReadyForQuery) {+ this.onReadyForQuery = onReadyForQuery;+ }++ boolean betweenMessages() {+ return headerBytesSeen == 0 && bodyRemaining == 0;+ }++ void consume(byte[] buf, int off, int len) {+ for (int i = off; i < off + len; i++) {+ if (bodyRemaining > 0) {+ int current = buf[i] & 0xFF;+ bodyRemaining--;+ if (bodyRemaining == 0 && pendingType == 'Z') {+ onReadyForQuery.accept(current);+ }+ continue;+ }+ header[headerBytesSeen++] = buf[i];+ if (headerBytesSeen == 5) {+ long msgLen = ((header[1] & 0xFFL) << 24) | ((header[2] & 0xFFL) << 16)+ | ((header[3] & 0xFFL) << 8) | (header[4] & 0xFFL);+ pendingType = (char) (header[0] & 0xFF);+ bodyRemaining = Math.max(0, msgLen - 4);+ headerBytesSeen = 0;+ if (bodyRemaining == 0 && pendingType == 'Z') {+ onReadyForQuery.accept('I');+ }+ }+ }+ }+ }}diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManager.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManager.javaindex a51b8c7ae2..aebaa1d340 100644--- a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManager.java+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManager.java@@ -3,6 +3,7 @@ package io.github.hectorvent.floci.services.redshift.proxy;import io.github.hectorvent.floci.services.rds.proxy.RdsAuthProxy;import io.github.hectorvent.floci.services.rds.proxy.RdsProxyTlsCertificates;import io.github.hectorvent.floci.services.rds.proxy.RdsSigV4Validator;+import io.github.hectorvent.floci.services.s3.S3Service;import io.quarkus.runtime.ShutdownEvent;import jakarta.enterprise.context.ApplicationScoped;import jakarta.enterprise.event.Observes;@@ -24,6 +25,7 @@ public class RedshiftProxyManager {private final RdsSigV4Validator sigV4Validator;private final RdsProxyTlsCertificates tlsCertificates;+ private final S3Service s3Service;private final ConcurrentHashMap<String, RedshiftAuthProxy> proxies = new ConcurrentHashMap<>();/*** Proxies whose listener could not be closed during a failed start or replace. The@@ -34,9 +36,11 @@ public class RedshiftProxyManager {private final ConcurrentHashMap<String, RedshiftAuthProxy> unclosableProxies = new ConcurrentHashMap<>();@Inject- public RedshiftProxyManager(RdsSigV4Validator sigV4Validator, RdsProxyTlsCertificates tlsCertificates) {+ public RedshiftProxyManager(RdsSigV4Validator sigV4Validator, RdsProxyTlsCertificates tlsCertificates,+ S3Service s3Service) {this.sigV4Validator = sigV4Validator;this.tlsCertificates = tlsCertificates;+ this.s3Service = s3Service;}public synchronized void startProxy(String relayKey, int proxyPort,@@ -50,7 +54,7 @@ public class RedshiftProxyManager {tlsCertificates.ensureHost(advertisedHost);RedshiftAuthProxy proxy = new RedshiftAuthProxy(relayKey, backendHost, backendPort, masterUsername, masterPassword, dbName,- sigV4Validator, tlsCertificates, passwordValidator);+ sigV4Validator, tlsCertificates, passwordValidator, s3Service);try {proxy.start(proxyPort);} catch (IOException | RuntimeException e) {diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.javanew file mode 100644index 0000000000..3ff45c0f6d--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.java@@ -0,0 +1,404 @@+package io.github.hectorvent.floci.services.redshift.proxy;++import io.github.hectorvent.floci.core.common.AwsException;+import io.github.hectorvent.floci.services.s3.S3Service;+import io.github.hectorvent.floci.services.s3.model.S3Object;+import org.jboss.logging.Logger;++import java.io.ByteArrayInputStream;+import java.io.ByteArrayOutputStream;+import java.io.IOException;+import java.io.InputStream;+import java.io.OutputStream;+import java.net.Socket;+import java.nio.charset.StandardCharsets;+import java.util.ArrayList;+import java.util.List;+import java.util.zip.GZIPInputStream;++/**+ * Emulates {@code COPY <table> FROM 's3://bucket/keyOrPrefix'} by reading the S3 object (or every+ * object under the prefix, in key order) through {@link S3Service} and streaming it into the+ * backing PostgreSQL container over a fabricated {@code COPY <table> FROM STDIN} exchange.+ *+ * <p>The caller must own the backend socket exclusively for the whole call: this class writes a+ * query and reads its response frames directly. Every exit path leaves the client with exactly one+ * response for its original {@code 'Q'} (a {@code CommandComplete}/{@code ReadyForQuery} on success,+ * a single {@code ErrorResponse}/{@code ReadyForQuery} on failure) and never closes the connection.+ */+public final class S3CopySimulator {++ private static final Logger LOG = Logger.getLogger(S3CopySimulator.class);++ private static final int LIST_PAGE_SIZE = 1000;+ private static final int CHUNK = 8192;++ private static final String SQLSTATE_INSUFFICIENT_PRIVILEGE = "42501";+ private static final String SQLSTATE_INTERNAL = "XX000";++ private S3CopySimulator() {+ }++ /**+ * @param txStatus the transaction-status byte from the client's last {@code ReadyForQuery}+ * ({@code 'I'} idle, {@code 'T'} in a block, {@code 'E'} failed block); a+ * synthesized error reports {@code 'I'} outside a block and {@code 'E'} inside.+ * @return {@code true} when the exchange was handled (success or a clean error sent to the+ * client). Part 4 always handles; the value exists so a later interceptor can decline.+ */+ public static boolean runCopyFrom(Socket client, Socket backend,+ CopyStatementParser.S3CopyFrom spec, S3Service s3,+ char txStatus) throws IOException {+ return runCopyFrom(client, backend, spec, s3, txStatus, null);+ }++ public static boolean runCopyFrom(Socket client, Socket backend,+ CopyStatementParser.S3CopyFrom spec, S3Service s3,+ char txStatus, java.util.function.IntConsumer onStatusChange) throws IOException {+ try {+ s3.authorizeAnonymousListBucket(spec.bucket());+ } catch (AwsException e) {+ LOG.debugv(e, "ListBucket denied for COPY from s3://{0}/{1}", spec.bucket(), spec.keyOrPrefix());+ sendAccessDenied(client, backend, spec, txStatus, onStatusChange);+ return true;+ }++ List<String> keys;+ try {+ keys = resolveKeys(spec, s3);+ } catch (AwsException e) {+ LOG.warnv(e, "failed to list s3://{0}/{1} for COPY", spec.bucket(), spec.keyOrPrefix());+ sendError(client, backend, SQLSTATE_INTERNAL,+ "S3 COPY could not list s3://" + spec.bucket() + "/" + spec.keyOrPrefix(), txStatus, onStatusChange);+ return true;+ }++ try {+ for (String key : keys) {+ s3.authorizeAnonymousGetObject(spec.bucket(), key);+ }+ } catch (AwsException e) {+ LOG.debugv(e, "GetObject denied for COPY from s3://{0}/{1}", spec.bucket(), spec.keyOrPrefix());+ sendAccessDenied(client, backend, spec, txStatus, onStatusChange);+ return true;+ }++ if (keys.isEmpty()) {+ sendError(client, backend, SQLSTATE_INTERNAL,+ "S3 object s3://" + spec.bucket() + "/" + spec.keyOrPrefix() + " not found", txStatus, onStatusChange);+ return true;+ }++ OutputStream backendOut = backend.getOutputStream();+ backendOut.write(PostgresWireDecoder.encodeQuery(fabricateCopy(spec)));+ backendOut.flush();++ PostgresWireDecoder backendDecoder = new PostgresWireDecoder(backend.getInputStream());+ PostgresWireDecoder.FrontendMessage first;+ try {+ first = nextNonAsync(backendDecoder, client);+ } catch (IOException e) {+ LOG.warnv(e, "backend read failed while awaiting CopyInResponse");+ closeQuietly(backend);+ sendError(client, null, SQLSTATE_INTERNAL, "S3 COPY failed: backend closed or timed out", txStatus, onStatusChange);+ closeQuietly(client);+ return true;+ }+ if (first == null) {+ LOG.warn("backend closed before answering the fabricated COPY");+ closeQuietly(backend);+ sendError(client, null, SQLSTATE_INTERNAL, "S3 COPY failed: backend closed before COPY started", txStatus, onStatusChange);+ closeQuietly(client);+ return true;+ }+ if (first.type() != 'G') {+ // Backend rejected the COPY itself (e.g. no such table). Its ErrorResponse and the+ // ReadyForQuery that follows are the client's one response.+ forward(client, first);+ drainToReadyForQuery(backendDecoder, client, onStatusChange);+ return true;+ }++ // The CopyIn stream is open. Any failure from here is resolved with exactly one response:+ // a CopyFail to the backend, whose ErrorResponse/ReadyForQuery is relayed to the client;+ // or, if the backend is unreachable, one synthesized ErrorResponse/ReadyForQuery.+ try {+ streamObjects(spec, s3, keys, backendOut);+ writeCopyDone(backendOut);+ drainToReadyForQuery(backendDecoder, client, onStatusChange);+ } catch (RuntimeException | IOException e) {+ LOG.warnv(e, "S3 COPY streaming failed; aborting the open CopyIn");+ abortOpenCopyIn(client, backend, backendOut, backendDecoder, e, txStatus, onStatusChange);+ }+ return true;+ }++ private static void abortOpenCopyIn(Socket client, Socket backend, OutputStream backendOut,+ PostgresWireDecoder backendDecoder, Exception cause,+ char txStatus, java.util.function.IntConsumer onStatusChange) throws IOException {+ try {+ writeCopyFail(backendOut, cause.getMessage());+ drainToReadyForQuery(backendDecoder, client, onStatusChange);+ } catch (IOException backendGone) {+ LOG.debugv(backendGone, "backend unreachable while aborting CopyIn; synthesizing client error");+ closeQuietly(backend);+ String detail = cause.getMessage() != null ? cause.getMessage() : cause.toString();+ sendError(client, null, SQLSTATE_INTERNAL, "S3 COPY failed: " + detail, txStatus, onStatusChange);+ closeQuietly(client);+ }+ }++ private static List<String> resolveKeys(CopyStatementParser.S3CopyFrom spec, S3Service s3) {+ List<String> keys = new ArrayList<>();+ if (s3.objectExists(spec.bucket(), spec.keyOrPrefix())) {+ keys.add(spec.keyOrPrefix());+ return keys;+ }+ String continuationToken = null;+ do {+ S3Service.ListObjectsResult result = s3.listObjectsWithPrefixes(+ spec.bucket(), spec.keyOrPrefix(), null, LIST_PAGE_SIZE, continuationToken, null);+ if (result != null && result.objects() != null) {+ for (S3Object object : result.objects()) {+ keys.add(object.getKey());+ }+ }+ continuationToken = (result != null && result.isTruncated()) ? result.nextContinuationToken() : null;+ } while (continuationToken != null);++ keys.sort(String::compareTo);+ return keys;+ }++ private static String fabricateCopy(CopyStatementParser.S3CopyFrom spec) {+ StringBuilder sql = new StringBuilder("COPY ").append(spec.targetTable());+ if (spec.columns() != null && !spec.columns().isEmpty()) {+ sql.append(" (").append(String.join(", ", spec.columns())).append(")");+ }+ sql.append(" FROM STDIN WITH (FORMAT ").append(spec.csv() ? "csv" : "text");+ String delimiter = spec.delimiter() != null ? spec.delimiter() : (spec.csv() ? "," : "|");+ sql.append(", DELIMITER '").append(quoteLiteral(delimiter)).append("'");+ if (spec.nullAs() != null) {+ sql.append(", NULL '").append(quoteLiteral(spec.nullAs())).append("'");+ }+ sql.append(")");+ return sql.toString();+ }++ /**+ * Quote a value for a single-quoted SQL string literal. PostgreSQL defaults to+ * {@code standard_conforming_strings = on}, where a backslash is an ordinary character, so only+ * the quote itself is doubled: {@code NULL AS '\N'} must reach the backend as {@code '\N'}.+ */+ private static String quoteLiteral(String value) {+ return value.replace("'", "''");+ }++ private static void streamObjects(CopyStatementParser.S3CopyFrom spec, S3Service s3,+ List<String> keys, OutputStream backendOut) throws IOException {+ byte[] buffer = new byte[CHUNK];+ for (int i = 0; i < keys.size(); i++) {+ S3Object object = s3.getObject(spec.bucket(), keys.get(i));+ byte[] data = object != null && object.getData() != null ? object.getData() : new byte[0];+ InputStream in = new ByteArrayInputStream(data);+ if (spec.gzip()) {+ in = new GZIPInputStream(in);+ }+ if (i == 0 && spec.headerLines() > 0) {+ skipLines(in, spec.headerLines());+ }+ int read;+ boolean endsWithNewline = false;+ boolean hasData = false;+ while ((read = in.read(buffer)) != -1) {+ if (read > 0) {+ hasData = true;+ endsWithNewline = (buffer[read - 1] == '\n');+ writeCopyData(backendOut, buffer, read);+ }+ }+ in.close();+ if (hasData && !endsWithNewline) {+ writeCopyData(backendOut, new byte[]{'\n'}, 1);+ }+ }+ backendOut.flush();+ }++ private static void skipLines(InputStream in, int lines) throws IOException {+ int seen = 0;+ int b;+ while (seen < lines && (b = in.read()) != -1) {+ if (b == '\n') {+ seen++;+ }+ }+ }++ private static void writeCopyData(OutputStream out, byte[] buf, int len) throws IOException {+ out.write('d');+ out.write(intBytes(4 + len));+ out.write(buf, 0, len);+ }++ private static void writeCopyDone(OutputStream out) throws IOException {+ out.write('c');+ out.write(intBytes(4));+ out.flush();+ }++ private static void writeCopyFail(OutputStream out, String reason) throws IOException {+ byte[] message = (reason == null ? "S3 COPY aborted" : reason).getBytes(StandardCharsets.UTF_8);+ out.write('f');+ out.write(intBytes(4 + message.length + 1));+ out.write(message);+ out.write(0);+ out.flush();+ }++ /** Read the next backend message, relaying asynchronous messages to the client as it goes. */+ private static PostgresWireDecoder.FrontendMessage nextNonAsync(PostgresWireDecoder decoder, Socket client)+ throws IOException {+ OutputStream clientOut = client.getOutputStream();+ while (true) {+ PostgresWireDecoder.FrontendMessage message = decoder.nextMessage();+ if (message == null) {+ return null;+ }+ char type = message.type();+ if (type == 'N' || type == 'A' || type == 'S') {+ clientOut.write(message.toPacketBytes());+ clientOut.flush();+ continue;+ }+ return message;+ }+ }++ private static void drainToReadyForQuery(PostgresWireDecoder decoder, Socket client,+ java.util.function.IntConsumer onStatusChange) throws IOException {+ OutputStream clientOut = client.getOutputStream();+ while (true) {+ PostgresWireDecoder.FrontendMessage message = decoder.nextMessage();+ if (message == null) {+ return;+ }+ clientOut.write(message.toPacketBytes());+ clientOut.flush();+ if (message.type() == 'Z') {+ if (onStatusChange != null && message.body().length > 0) {+ onStatusChange.accept(message.body()[0]);+ }+ return;+ }+ }+ }++ private static void forward(Socket client, PostgresWireDecoder.FrontendMessage message) throws IOException {+ client.getOutputStream().write(message.toPacketBytes());+ client.getOutputStream().flush();+ }++ private static void sendAccessDenied(Socket client, Socket backend, CopyStatementParser.S3CopyFrom spec,+ char txStatus, java.util.function.IntConsumer onStatusChange) throws IOException {+ sendError(client, backend, SQLSTATE_INSUFFICIENT_PRIVILEGE,+ "S3 access denied for s3://" + spec.bucket() + "/" + spec.keyOrPrefix(), txStatus, onStatusChange);+ }++ private static void sendError(Socket client, Socket backend, String sqlState, String message,+ char txStatus, java.util.function.IntConsumer onStatusChange) throws IOException {+ if (txStatus == 'T' && backend != null && !backend.isClosed()) {+ if (!failBackendTransaction(backend, onStatusChange)) {+ closeQuietly(backend);+ // Backend transaction state could not be confirmed; do not report an unconfirmed+ // failed transaction ('E') or leave a desynchronized backend in the pump.+ try {+ OutputStream out = client.getOutputStream();+ byte[] body = errorBody(sqlState, message);+ out.write('E');+ out.write(intBytes(4 + body.length));+ out.write(body);+ out.flush();+ } catch (IOException e) {+ LOG.debugv(e, "failed to send ErrorResponse to client before closing socket");+ } finally {+ closeQuietly(client);+ }+ return;+ }+ }+ OutputStream out = client.getOutputStream();+ byte[] body = errorBody(sqlState, message);+ out.write('E');+ out.write(intBytes(4 + body.length));+ out.write(body);+ byte status = (txStatus == 'I' || txStatus == 0) ? (byte) 'I' : (byte) 'E';+ out.write(new byte[]{'Z', 0, 0, 0, 5, status});+ out.flush();+ if (onStatusChange != null) {+ onStatusChange.accept(status);+ }+ }++ private static boolean failBackendTransaction(Socket backend, java.util.function.IntConsumer onStatusChange) {+ try {+ OutputStream out = backend.getOutputStream();+ out.write(PostgresWireDecoder.encodeQuery("(FLOCI_ABORT_TX)"));+ out.flush();+ PostgresWireDecoder decoder = new PostgresWireDecoder(backend.getInputStream());+ while (true) {+ PostgresWireDecoder.FrontendMessage msg = decoder.nextMessage();+ if (msg == null) {+ LOG.warn("backend closed before transaction abort could be synchronized");+ return false;+ }+ if (msg.type() == 'Z') {+ char status = (msg.body().length > 0) ? (char) msg.body()[0] : 0;+ if (onStatusChange != null && status != 0) {+ onStatusChange.accept(status);+ }+ if (status != 'E') {+ LOG.warnv("backend returned unexpected transaction status {0} instead of 'E' after abort query", status);+ return false;+ }+ return true;+ }+ }+ } catch (IOException e) {+ LOG.warnv(e, "failed to synchronize backend transaction abort state");+ return false;+ }+ }++ private static void closeQuietly(Socket s) {+ try {+ if (s != null && !s.isClosed()) {+ s.close();+ }+ } catch (IOException e) {+ LOG.debugv(e, "failed to close socket: {0}", s);+ }+ }++ private static byte[] errorBody(String sqlState, String message) {+ ByteArrayOutputStream bytes = new ByteArrayOutputStream();+ writeField(bytes, 'S', "ERROR");+ writeField(bytes, 'C', sqlState);+ writeField(bytes, 'M', message);+ bytes.write(0);+ return bytes.toByteArray();+ }++ private static void writeField(ByteArrayOutputStream bytes, char tag, String value) {+ bytes.write(tag);+ byte[] raw = value.getBytes(StandardCharsets.UTF_8);+ bytes.write(raw, 0, raw.length);+ bytes.write(0);+ }++ private static byte[] intBytes(int value) {+ return new byte[]{+ (byte) (value >>> 24), (byte) (value >>> 16), (byte) (value >>> 8), (byte) value+ };+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java b/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.javaindex 426a527972..de6a63d597 100644--- a/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java+++ b/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java@@ -612,6 +612,11 @@ public class S3Service implements Resettable, ResourceProvider {authorizeGetObject(bucketName, key, null, RequestAuthorization.unsigned());}+ /** Authorize an unsigned {@code s3:ListBucket}; see {@link #authorizeAnonymousGetObject}. */+ public void authorizeAnonymousListBucket(String bucketName) {+ authorizeListBucket(bucketName, RequestAuthorization.unsigned());+ }+public void authorizeCloudFrontOacGetObject(String bucketName, String key, String distributionArn) {authorizeCloudFrontGetObject(diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.javaindex b2a3b854e3..9b12d1ff44 100644--- a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java@@ -27,6 +27,9 @@ class RedshiftInterceptorIntegrationTest {@InjectRedshiftService service;+ @Inject+ io.github.hectorvent.floci.services.s3.S3Service s3;+private String clusterId;@BeforeAll@@ -106,4 +109,115 @@ class RedshiftInterceptorIntegrationTest {}}}++ @Test+ void copyFromS3SingleObjectLoadsRows() throws Exception {+ clusterId = "it-copy-single";+ Cluster cluster = service.createCluster(clusterId, "dc2.large", "admin", "Secret123");++ String bucket = "redshift-copy-it";+ s3.createBucket(bucket, "us-east-1");+ s3.putObject(bucket, "people/p1.txt",+ "1|alice\n2|bob\n".getBytes(java.nio.charset.StandardCharsets.UTF_8), "text/plain", java.util.Map.of());++ try (Connection c = waitForConnection(cluster, "admin", "Secret123")) {+ c.createStatement().execute("CREATE TABLE people (id int, name text)");+ c.createStatement().execute("COPY people FROM 's3://redshift-copy-it/people/p1.txt'");+ try (ResultSet rs = c.createStatement().executeQuery("SELECT count(*) FROM people")) {+ assertTrue(rs.next());+ assertEquals(2, rs.getInt(1));+ }+ }+ }++ @Test+ void copyFromS3PrefixConcatenatesObjects() throws Exception {+ clusterId = "it-copy-prefix";+ Cluster cluster = service.createCluster(clusterId, "dc2.large", "admin", "Secret123");++ String bucket = "redshift-copy-it-prefix";+ s3.createBucket(bucket, "us-east-1");+ s3.putObject(bucket, "d/a", "1|a\n".getBytes(java.nio.charset.StandardCharsets.UTF_8), "text/plain", java.util.Map.of());+ s3.putObject(bucket, "d/b", "2|b\n".getBytes(java.nio.charset.StandardCharsets.UTF_8), "text/plain", java.util.Map.of());++ try (Connection c = waitForConnection(cluster, "admin", "Secret123")) {+ c.createStatement().execute("CREATE TABLE t (id int, v text)");+ c.createStatement().execute("COPY t FROM 's3://redshift-copy-it-prefix/d/'");+ try (ResultSet rs = c.createStatement().executeQuery("SELECT count(*) FROM t")) {+ assertTrue(rs.next());+ assertEquals(2, rs.getInt(1));+ }+ }+ }++ @Test+ void copyFromS3GzipObjectLoadsRows() throws Exception {+ clusterId = "it-copy-gzip";+ Cluster cluster = service.createCluster(clusterId, "dc2.large", "admin", "Secret123");++ String bucket = "redshift-copy-it-gzip";+ s3.createBucket(bucket, "us-east-1");+ java.io.ByteArrayOutputStream raw = new java.io.ByteArrayOutputStream();+ try (java.util.zip.GZIPOutputStream gz = new java.util.zip.GZIPOutputStream(raw)) {+ gz.write("10|x\n11|y\n".getBytes(java.nio.charset.StandardCharsets.UTF_8));+ }+ s3.putObject(bucket, "g/data.gz", raw.toByteArray(), "application/gzip", java.util.Map.of());++ try (Connection c = waitForConnection(cluster, "admin", "Secret123")) {+ c.createStatement().execute("CREATE TABLE g (id int, v text)");+ c.createStatement().execute("COPY g FROM 's3://redshift-copy-it-gzip/g/data.gz' GZIP");+ try (ResultSet rs = c.createStatement().executeQuery("SELECT count(*) FROM g")) {+ assertTrue(rs.next());+ assertEquals(2, rs.getInt(1));+ }+ }+ }++ @Test+ void copyFromMissingObjectSurfacesASqlError() throws Exception {+ clusterId = "it-copy-missing";+ Cluster cluster = service.createCluster(clusterId, "dc2.large", "admin", "Secret123");++ String bucket = "redshift-copy-it-missing";+ s3.createBucket(bucket, "us-east-1");++ try (Connection c = waitForConnection(cluster, "admin", "Secret123")) {+ c.createStatement().execute("CREATE TABLE t2 (id int)");+ SQLException ex = org.junit.jupiter.api.Assertions.assertThrows(+ SQLException.class,+ () -> c.createStatement().execute("COPY t2 FROM 's3://redshift-copy-it-missing/does/not/exist'"));+ assertTrue(+ ex.getMessage().toLowerCase().contains("not found"), ex.getMessage());+ }+ }++ @Test+ void copyFromMissingObjectInTransactionAbortsTransaction() throws Exception {+ clusterId = "it-copy-tx-abort";+ Cluster cluster = service.createCluster(clusterId, "dc2.large", "admin", "Secret123");++ String bucket = "redshift-copy-it-tx";+ s3.createBucket(bucket, "us-east-1");++ try (Connection c = waitForConnection(cluster, "admin", "Secret123")) {+ c.createStatement().execute("CREATE TABLE tx_test (id int)");+ c.setAutoCommit(false);+ Statement st = c.createStatement();+ st.execute("INSERT INTO tx_test VALUES (1)");+ org.junit.jupiter.api.Assertions.assertThrows(+ SQLException.class,+ () -> st.execute("COPY tx_test FROM 's3://redshift-copy-it-tx/does/not/exist'"));+ // The backend must now be in the aborted transaction state ('E').+ // Subsequent statements in this transaction block must fail.+ org.junit.jupiter.api.Assertions.assertThrows(+ SQLException.class,+ () -> st.execute("INSERT INTO tx_test VALUES (2)"));+ c.rollback();+ c.setAutoCommit(true);+ try (ResultSet rs = c.createStatement().executeQuery("SELECT count(*) FROM tx_test")) {+ assertTrue(rs.next());+ assertEquals(0, rs.getInt(1));+ }+ }+ }}diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.javanew file mode 100644index 0000000000..e6a8cb3c72--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.java@@ -0,0 +1,171 @@+package io.github.hectorvent.floci.services.redshift.proxy;++import org.junit.jupiter.api.Test;++import java.util.List;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertNull;+import static org.junit.jupiter.api.Assertions.assertTrue;++class CopyStatementParserTest {++ @Test+ void parsesMinimalCopyFromKey() {+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse(+ "COPY sales FROM 's3://warehouse/data/sales.txt'");+ assertEquals("sales", c.targetTable());+ assertEquals(List.of(), c.columns());+ assertEquals("warehouse", c.bucket());+ assertEquals("data/sales.txt", c.keyOrPrefix());+ assertEquals("|", c.delimiter());+ assertEquals(0, c.headerLines());+ assertFalse(c.gzip());+ assertFalse(c.csv());+ assertNull(c.nullAs());+ }++ @Test+ void parsesColumnsOptionsAndPrefix() {+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse(+ "COPY public.events (id, ts, note) FROM 's3://bkt/evt/' "+ + "GZIP DELIMITER ',' IGNOREHEADER 2 NULL AS '\\\\N' FORMAT AS CSV");+ assertEquals("public.events", c.targetTable());+ assertEquals(List.of("id", "ts", "note"), c.columns());+ assertEquals("bkt", c.bucket());+ assertEquals("evt/", c.keyOrPrefix());+ assertTrue(c.gzip());+ assertTrue(c.csv());+ assertEquals(",", c.delimiter());+ assertEquals(2, c.headerLines());+ assertEquals("\\N", c.nullAs());+ }++ @Test+ void defaultsDelimiterToCommaForCsvAndTreatsHeaderKeywordAsOneLine() {+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse(+ "COPY t FROM 's3://b/k' CSV HEADER");+ assertEquals(",", c.delimiter());+ assertEquals(1, c.headerLines());+ }++ @Test+ void decodesTabDelimiterToken() {+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse(+ "COPY t FROM 's3://b/k' DELIMITER '\\\\t'");+ assertEquals("\t", c.delimiter());+ }++ @Test+ void bucketOnlyPathGivesEmptyPrefix() {+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse("COPY t FROM 's3://only-bucket'");+ assertEquals("only-bucket", c.bucket());+ assertEquals("", c.keyOrPrefix());+ }++ @Test+ void stripsLeadingComments() {+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse(+ "-- load nightly\n/* batch */ COPY t FROM 's3://b/k'");+ assertEquals("t", c.targetTable());+ }++ @Test+ void rejectsInjectionInTableName() {+ assertNull(CopyStatementParser.parse("COPY t; DROP TABLE u; FROM 's3://b/k'"));+ assertNull(CopyStatementParser.parse("COPY (SELECT 1) FROM 's3://b/k'"));+ }++ @Test+ void rejectsInjectionInColumnList() {+ assertNull(CopyStatementParser.parse("COPY t (id, x) FROM STDIN) --) FROM 's3://b/k'"));+ assertNull(CopyStatementParser.parse("COPY t (id, ts::text) FROM 's3://b/k'"));+ }++ @Test+ void returnsNullForNonCopyAndForCopyWithoutS3() {+ assertNull(CopyStatementParser.parse("SELECT 1"));+ assertNull(CopyStatementParser.parse("CREATE TABLE t (id int) DISTKEY (id)"));+ assertNull(CopyStatementParser.parse("COPY t FROM STDIN"));+ assertNull(CopyStatementParser.parse("COPY t TO 's3://b/k'"));+ assertNull(CopyStatementParser.parse(null));+ assertNull(CopyStatementParser.parse(" "));+ }++ @Test+ void quotedIdentifiersSurvive() {+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse(+ "COPY \"My Schema\".\"Tab\" (\"col one\") FROM 's3://b/k'");+ assertEquals("\"My Schema\".\"Tab\"", c.targetTable());+ assertEquals(List.of("\"col one\""), c.columns());+ }++ @Test+ void explicitDelimiterOverridesCsvDefault() {+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse(+ "COPY t FROM 's3://b/k' CSV DELIMITER '\t'");+ assertEquals("\t", c.delimiter());+ assertTrue(c.csv());+ }++ @Test+ void returnsNullForUnsupportedClauses() {+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' FORMAT AS PARQUET"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' JSON 'auto'"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' FIXEDWIDTH 'a:1,b:2'"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' GZIP MAXERROR 10"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' IAM_ROLE 'arn:aws:iam::0:role/r'"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' DATEFORMAT 'YYYY-MM-DD'"));+ }++ @Test+ void returnsNullWhenAStatementFollowsTheCopy() {+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k'; DROP TABLE staging"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' GZIP; SELECT 1"));+ }++ @Test+ void toleratesALoneTrailingSemicolon() {+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse("COPY t FROM 's3://b/k' GZIP;");+ assertTrue(c.gzip());+ }++ @Test+ void aQuotedKeywordInAnOptionValueDoesNotChangeParsing() {+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse(+ "COPY t FROM 's3://b/k' NULL AS 'csv' DELIMITER ';'");+ assertFalse(c.csv());+ assertEquals(";", c.delimiter());+ assertEquals("csv", c.nullAs());+ }++ @Test+ void rejectsTyposInOptionKeywords() {+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' DELIMETER ','"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' GZIPP"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' HEADERR"));+ }++ @Test+ void rejectsLeftoverOrUnknownTokens() {+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' GZIP EXTRA_TOKEN"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' SOME_UNKNOWN_OPTION"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' CSV FOO"));+ }++ @Test+ void rejectsDuplicateOrConflictingOptions() {+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' GZIP GZIP"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' CSV FORMAT CSV"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' DELIMITER ',' DELIMITER '\t'"));+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' HEADER IGNOREHEADER 2"));+ }++ @Test+ void parsesEscapedSingleQuoteInDelimiter() {+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse(+ "COPY t FROM 's3://b/k' DELIMITER ''''");+ assertEquals("'", c.delimiter());+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftAuthProxyTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftAuthProxyTest.javaindex 804e5d3a08..a0f396f648 100644--- a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftAuthProxyTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftAuthProxyTest.java@@ -67,7 +67,8 @@ class RedshiftAuthProxyTest {int proxyPort = freePort();proxy = new RedshiftAuthProxy("111111111111:c1", "localhost", fakeBackend.getLocalPort(),"admin", "Secret123", "dev",- mock(RdsSigV4Validator.class), realTls(), (user, pw) -> true);+ mock(RdsSigV4Validator.class), realTls(), (user, pw) -> true,+ mock(io.github.hectorvent.floci.services.s3.S3Service.class));proxy.start(proxyPort);try (Socket client = new Socket("localhost", proxyPort)) {@@ -120,7 +121,8 @@ class RedshiftAuthProxyTest {int proxyPort = freePort();proxy = new RedshiftAuthProxy("111111111111:c1", "localhost", fakeBackend.getLocalPort(),"admin", "Secret123", "dev",- mock(RdsSigV4Validator.class), realTls(), (user, pw) -> true);+ mock(RdsSigV4Validator.class), realTls(), (user, pw) -> true,+ mock(io.github.hectorvent.floci.services.s3.S3Service.class));proxy.start(proxyPort);// Connect, then drop without ever sending a startup packet.@@ -150,7 +152,8 @@ class RedshiftAuthProxyTest {proxy = new RedshiftAuthProxy("111111111111:c1", "localhost", fakeBackend.getLocalPort(),"admin", "Secret123", "dev",- mock(RdsSigV4Validator.class), realTls(), (user, pw) -> true);+ mock(RdsSigV4Validator.class), realTls(), (user, pw) -> true,+ mock(io.github.hectorvent.floci.services.s3.S3Service.class));proxy.start(proxyPort); // must not throw despite the port being busy at firstassertTrue(portAccepts(proxyPort), "proxy never bound the port after the squatter released it");@@ -162,7 +165,8 @@ class RedshiftAuthProxyTest {int proxyPort = freePort();proxy = new RedshiftAuthProxy("111111111111:c1", "localhost", fakeBackend.getLocalPort(),"admin", "old", "dev",- mock(RdsSigV4Validator.class), realTls(), (user, pw) -> true);+ mock(RdsSigV4Validator.class), realTls(), (user, pw) -> true,+ mock(io.github.hectorvent.floci.services.s3.S3Service.class));proxy.start(proxyPort);proxy.updateMasterPassword("rotated");diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftInterceptingBridgeTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftInterceptingBridgeTest.javaindex b11b9fbe29..a7fdf5c879 100644--- a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftInterceptingBridgeTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftInterceptingBridgeTest.java@@ -1,13 +1,18 @@package io.github.hectorvent.floci.services.redshift.proxy;+import io.github.hectorvent.floci.services.s3.S3Service;+import io.github.hectorvent.floci.services.s3.model.S3Object;import org.jboss.logging.Logger;import org.junit.jupiter.api.AfterEach;import org.junit.jupiter.api.Test;+import org.mockito.Mockito;import java.io.IOException;+import java.io.OutputStream;import java.net.ServerSocket;import java.net.Socket;import java.nio.charset.StandardCharsets;+import java.util.concurrent.atomic.AtomicReference;import static org.junit.jupiter.api.Assertions.assertArrayEquals;import static org.junit.jupiter.api.Assertions.assertEquals;@@ -27,6 +32,8 @@ class RedshiftInterceptingBridgeTest {private Socket testBackendEnd; // test reads what the bridge forwarded, writes backend repliesprivate Thread bridgeThread;+ private S3Service s3Stub;+private void startBridge() throws IOException {clientListener = new ServerSocket(0);testClientEnd = new Socket("localhost", clientListener.getLocalPort());@@ -36,7 +43,8 @@ class RedshiftInterceptingBridgeTest {bridgeBackendEnd = new Socket("localhost", backendListener.getLocalPort());testBackendEnd = backendListener.accept();- RedshiftInterceptingBridge bridge = new RedshiftInterceptingBridge(bridgeClientEnd, bridgeBackendEnd);+ s3Stub = Mockito.mock(S3Service.class);+ RedshiftInterceptingBridge bridge = new RedshiftInterceptingBridge(bridgeClientEnd, bridgeBackendEnd, s3Stub);bridgeThread = Thread.ofVirtual().name("bridge-under-test").start(bridge::run);}@@ -138,4 +146,70 @@ class RedshiftInterceptingBridgeTest {assertFalse(bridgeThread.isAlive(), "bridge did not stop after Terminate");assertEquals(-1, testClientEnd.getInputStream().read(), "bridge left the client socket open");}++ @Test+ void interceptsCopyFromS3AndDoesNotForwardTheOriginalQuery() throws Exception {+ startBridge();+ Mockito.when(s3Stub.objectExists("wh", "k")).thenReturn(true);+ Mockito.when(s3Stub.getObject("wh", "k")).thenReturn(+ new S3Object("wh", "k", "1|a\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));++ AtomicReference<Throwable> backendFailure = new AtomicReference<>();+ // fake backend: expect a fabricated Query, answer 'G', then after CopyDone answer 'C'+'Z'+ Thread backend = Thread.ofVirtual().start(() -> {+ try {+ PostgresWireDecoder in = new PostgresWireDecoder(testBackendEnd.getInputStream());+ PostgresWireDecoder.FrontendMessage q = in.nextMessage();+ assertEquals('Q', q.type());+ assertTrue(q.getSql().contains("FROM STDIN"), q.getSql());+ OutputStream out = testBackendEnd.getOutputStream();+ out.write(new byte[]{'G', 0, 0, 0, 7, 0, 0, 0});+ out.flush();+ while (true) {+ PostgresWireDecoder.FrontendMessage m = in.nextMessage();+ if (m == null || m.type() == 'c') {+ break;+ }+ }+ byte[] tag = "COPY 1\0".getBytes(StandardCharsets.US_ASCII);+ out.write('C');+ out.write(new byte[]{0, 0, 0, (byte) (4 + tag.length)});+ out.write(tag);+ out.write(new byte[]{'Z', 0, 0, 0, 5, 'I'});+ out.flush();+ } catch (Throwable t) {+ backendFailure.compareAndSet(null, t);+ }+ });++ testClientEnd.getOutputStream().write(+ PostgresWireDecoder.encodeQuery("COPY t FROM 's3://wh/k'"));+ testClientEnd.getOutputStream().flush();++ PostgresWireDecoder.FrontendMessage toClient =+ new PostgresWireDecoder(testClientEnd.getInputStream()).nextMessage();+ assertEquals('C', toClient.type());+ backend.join();+ if (backendFailure.get() != null) {+ throw new AssertionError("fake backend failed", backendFailure.get());+ }+ }++ @Test+ void fallsOpenAndForwardsCopyWhenBackendHasAnOutstandingResponse() throws Exception {+ startBridge();+ // send a normal query first and never answer it, so outstandingResponses stays > 0+ testClientEnd.getOutputStream().write(PostgresWireDecoder.encodeQuery("SELECT 1"));+ testClientEnd.getOutputStream().flush();+ PostgresWireDecoder backendIn = new PostgresWireDecoder(testBackendEnd.getInputStream());+ assertEquals('Q', backendIn.nextMessage().type()); // SELECT 1 arrived, unanswered++ testClientEnd.getOutputStream().write(PostgresWireDecoder.encodeQuery("COPY t FROM 's3://wh/k'"));+ testClientEnd.getOutputStream().flush();++ // within the ~2s deadline the bridge gives up and forwards the COPY verbatim+ PostgresWireDecoder.FrontendMessage forwarded = backendIn.nextMessage();+ assertEquals('Q', forwarded.type());+ assertTrue(forwarded.getSql().contains("s3://wh/k"), forwarded.getSql());+ }}diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManagerTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManagerTest.javaindex 0c07c91cb7..d62c6658f8 100644--- a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManagerTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManagerTest.java@@ -27,7 +27,8 @@ class RedshiftProxyManagerTest {private RedshiftProxyManager newManager() {return new RedshiftProxyManager(- mock(RdsSigV4Validator.class), mock(RdsProxyTlsCertificates.class));+ mock(RdsSigV4Validator.class), mock(RdsProxyTlsCertificates.class),+ mock(io.github.hectorvent.floci.services.s3.S3Service.class));}private static void start(RedshiftProxyManager manager, String key, int proxyPort) {diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.javanew file mode 100644index 0000000000..28b9a8aeaa--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java@@ -0,0 +1,549 @@+package io.github.hectorvent.floci.services.redshift.proxy;++import io.github.hectorvent.floci.core.common.AwsException;+import io.github.hectorvent.floci.services.s3.S3Service;+import io.github.hectorvent.floci.services.s3.model.S3Object;+import org.junit.jupiter.api.AfterEach;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Test;++import java.io.ByteArrayOutputStream;+import java.io.IOException;+import java.io.OutputStream;+import java.net.ServerSocket;+import java.net.Socket;+import java.net.SocketTimeoutException;+import java.nio.charset.StandardCharsets;+import java.util.List;+import java.util.concurrent.atomic.AtomicReference;+import java.util.zip.GZIPOutputStream;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertNull;+import static org.junit.jupiter.api.Assertions.assertThrows;+import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.mockito.ArgumentMatchers.any;+import static org.mockito.ArgumentMatchers.anyInt;+import static org.mockito.ArgumentMatchers.eq;+import static org.mockito.ArgumentMatchers.isNull;+import static org.mockito.Mockito.doThrow;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.when;++class S3CopySimulatorTest {++ private ServerSocket listener;+ private Socket simClient; // simulator's "client" end+ private Socket simBackend; // simulator's "backend" end+ private Socket testClient; // test reads ErrorResponse / CommandComplete here+ private Socket testBackend; // test plays PostgreSQL here+ private S3Service s3;++ private final AtomicReference<Throwable> backendFailure = new AtomicReference<>();++ @BeforeEach+ void setUp() throws IOException {+ listener = new ServerSocket(0);+ simClient = new Socket("localhost", listener.getLocalPort());+ testClient = listener.accept();++ ServerSocket backendListener = new ServerSocket(0);+ simBackend = new Socket("localhost", backendListener.getLocalPort());+ testBackend = backendListener.accept();+ backendListener.close();++ s3 = mock(S3Service.class);+ }++ @AfterEach+ void tearDown() throws IOException {+ for (Socket s : new Socket[]{simClient, simBackend, testClient, testBackend}) {+ if (s != null && !s.isClosed()) {+ s.close();+ }+ }+ if (listener != null && !listener.isClosed()) {+ listener.close();+ }+ }++ @FunctionalInterface+ private interface BackendScript {+ void run() throws IOException;+ }++ /** Run a fake-backend script on a virtual thread, capturing any failure for the test thread. */+ private Thread backendThread(BackendScript script) {+ return Thread.ofVirtual().start(() -> {+ try {+ script.run();+ } catch (Throwable t) {+ backendFailure.compareAndSet(null, t);+ }+ });+ }++ private void joinBackend(Thread t) throws InterruptedException {+ t.join();+ Throwable failure = backendFailure.get();+ if (failure != null) {+ throw new AssertionError("fake backend failed: " + failure, failure);+ }+ }++ private static byte[] gzip(String s) throws IOException {+ ByteArrayOutputStream bytes = new ByteArrayOutputStream();+ try (GZIPOutputStream gz = new GZIPOutputStream(bytes)) {+ gz.write(s.getBytes(StandardCharsets.US_ASCII));+ }+ return bytes.toByteArray();+ }++ private static byte[] intBytes(int v) {+ return new byte[]{(byte) (v >>> 24), (byte) (v >>> 16), (byte) (v >>> 8), (byte) v};+ }++ private PostgresWireDecoder.FrontendMessage nextFromSimulatorToBackend() throws IOException {+ return new PostgresWireDecoder(testBackend.getInputStream()).nextMessage();+ }++ /** Play a minimal happy-path PostgreSQL backend: 'G' then, after CopyDone, 'C' and 'Z'. */+ private void playHappyBackend(ByteArrayOutputStream capturedCopyData) throws IOException {+ OutputStream out = testBackend.getOutputStream();+ PostgresWireDecoder in = new PostgresWireDecoder(testBackend.getInputStream());+ PostgresWireDecoder.FrontendMessage query = in.nextMessage();+ assertEquals('Q', query.type());+ // CopyInResponse 'G': body = format(1) + columnCount(2) = 3 bytes, all zero => length 7+ out.write(new byte[]{'G', 0, 0, 0, 7, 0, 0, 0});+ out.flush();+ while (true) {+ PostgresWireDecoder.FrontendMessage m = in.nextMessage();+ if (m == null || m.type() == 'c') {+ break;+ }+ if (m.type() == 'd') {+ capturedCopyData.write(m.body());+ }+ }+ byte[] tag = "COPY 1\0".getBytes(StandardCharsets.US_ASCII);+ out.write('C');+ out.write(intBytes(4 + tag.length));+ out.write(tag);+ out.write(new byte[]{'Z', 0, 0, 0, 5, 'I'});+ out.flush();+ }++ private String captureFabricatedThenComplete() throws IOException {+ OutputStream out = testBackend.getOutputStream();+ PostgresWireDecoder in = new PostgresWireDecoder(testBackend.getInputStream());+ PostgresWireDecoder.FrontendMessage q = in.nextMessage();+ assertEquals('Q', q.type());+ out.write(new byte[]{'G', 0, 0, 0, 7, 0, 0, 0});+ out.flush();+ while (true) {+ PostgresWireDecoder.FrontendMessage m = in.nextMessage();+ if (m == null || m.type() == 'c') {+ break;+ }+ }+ out.write(new byte[]{'Z', 0, 0, 0, 5, 'I'});+ out.flush();+ return q.getSql();+ }++ @Test+ void loadsSingleKeyAsPipeTextAndForwardsCommandComplete() throws Exception {+ when(s3.objectExists("wh", "d/a.txt")).thenReturn(true);+ when(s3.getObject("wh", "d/a.txt")).thenReturn(+ new S3Object("wh", "d/a.txt", "1|alice\n2|bob\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));++ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "people", List.of(), "wh", "d/a.txt", "|", 0, false, false, null);++ ByteArrayOutputStream copyData = new ByteArrayOutputStream();+ Thread backend = backendThread(() -> playHappyBackend(copyData));++ boolean handled = S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');+ joinBackend(backend);++ assertTrue(handled);+ assertEquals("1|alice\n2|bob\n", copyData.toString(StandardCharsets.US_ASCII));++ PostgresWireDecoder.FrontendMessage toClient = new PostgresWireDecoder(testClient.getInputStream()).nextMessage();+ assertEquals('C', toClient.type());+ }++ @Test+ void fabricatedQueryUsesTextFormatAndPipeDelimiterByDefault() throws Exception {+ when(s3.objectExists("wh", "k")).thenReturn(true);+ when(s3.getObject("wh", "k")).thenReturn(+ new S3Object("wh", "k", "x\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "t", List.of("a", "b"), "wh", "k", "|", 0, false, false, null);++ AtomicReference<String> fabricated = new AtomicReference<>();+ Thread backend = backendThread(() -> fabricated.set(captureFabricatedThenComplete()));++ S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');+ joinBackend(backend);++ String sql = fabricated.get();+ assertTrue(sql.contains("COPY t (a, b) FROM STDIN"), sql);+ assertTrue(sql.toUpperCase().contains("FORMAT TEXT"), sql);+ assertTrue(sql.contains("DELIMITER '|'"), sql);+ assertFalse(sql.toUpperCase().contains("HEADER"), sql);+ }++ @Test+ void fabricatedQueryUsesCsvFormatAndCommaDelimiterByDefault() throws Exception {+ when(s3.objectExists("wh", "k")).thenReturn(true);+ when(s3.getObject("wh", "k")).thenReturn(+ new S3Object("wh", "k", "x\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "t", List.of("a", "b"), "wh", "k", null, 0, false, true, null);++ AtomicReference<String> fabricated = new AtomicReference<>();+ Thread backend = backendThread(() -> fabricated.set(captureFabricatedThenComplete()));++ S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');+ joinBackend(backend);++ String sql = fabricated.get();+ assertTrue(sql.contains("COPY t (a, b) FROM STDIN"), sql);+ assertTrue(sql.toUpperCase().contains("FORMAT CSV"), sql);+ assertTrue(sql.contains("DELIMITER ','"), sql);+ }++ @Test+ void nullAsBackslashNIsPassedThroughLiterally() throws Exception {+ when(s3.objectExists("wh", "k")).thenReturn(true);+ when(s3.getObject("wh", "k")).thenReturn(+ new S3Object("wh", "k", "x\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "t", List.of(), "wh", "k", "|", 0, false, false, "\\N");++ AtomicReference<String> fabricated = new AtomicReference<>();+ Thread backend = backendThread(() -> fabricated.set(captureFabricatedThenComplete()));++ S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');+ joinBackend(backend);++ String sql = fabricated.get();+ assertTrue(sql.contains("NULL '\\N'"), sql);+ assertFalse(sql.contains("NULL '\\\\N'"), sql);+ }++ @Test+ void concatenatesPrefixObjectsInKeyOrderAndSkipsHeaderOnFirstOnly() throws Exception {+ when(s3.objectExists("wh", "p/")).thenReturn(false);+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("p/"), isNull(), anyInt(), any(), any())).thenReturn(+ new S3Service.ListObjectsResult(List.of(+ new S3Object("wh", "p/1", "h1|h2\n1|a\n".getBytes(StandardCharsets.US_ASCII), "text/plain"),+ new S3Object("wh", "p/2", "2|b\n".getBytes(StandardCharsets.US_ASCII), "text/plain")),+ List.of(), false, null));+ when(s3.getObject("wh", "p/1")).thenReturn(+ new S3Object("wh", "p/1", "h1|h2\n1|a\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));+ when(s3.getObject("wh", "p/2")).thenReturn(+ new S3Object("wh", "p/2", "2|b\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));++ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "t", List.of(), "wh", "p/", "|", 1, false, false, null);++ ByteArrayOutputStream copyData = new ByteArrayOutputStream();+ Thread backend = backendThread(() -> playHappyBackend(copyData));+ S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');+ joinBackend(backend);++ assertEquals("1|a\n2|b\n", copyData.toString(StandardCharsets.US_ASCII));+ }++ @Test+ void decompressesGzipObjects() throws Exception {+ when(s3.objectExists("wh", "k.gz")).thenReturn(true);+ when(s3.getObject("wh", "k.gz")).thenReturn(+ new S3Object("wh", "k.gz", gzip("1|a\n2|b\n"), "application/gzip"));+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "t", List.of(), "wh", "k.gz", "|", 0, true, false, null);++ ByteArrayOutputStream copyData = new ByteArrayOutputStream();+ Thread backend = backendThread(() -> playHappyBackend(copyData));+ S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');+ joinBackend(backend);++ assertEquals("1|a\n2|b\n", copyData.toString(StandardCharsets.US_ASCII));+ }++ @Test+ void deniedBucketSendsInsufficientPrivilegeErrorResponse() throws Exception {+ doThrow(new AwsException("AccessDenied", "no", 403))+ .when(s3).authorizeAnonymousListBucket("wh");+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "t", List.of(), "wh", "k", "|", 0, false, false, null);++ boolean handled = S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');+ assertTrue(handled);++ PostgresWireDecoder in = new PostgresWireDecoder(testClient.getInputStream());+ PostgresWireDecoder.FrontendMessage err = in.nextMessage();+ assertEquals('E', err.type());+ assertTrue(new String(err.body(), StandardCharsets.US_ASCII).contains("42501"));+ assertEquals('Z', in.nextMessage().type());+ }++ @Test+ void missingKeySendsNotFoundErrorResponse() throws Exception {+ when(s3.objectExists("wh", "missing")).thenReturn(false);+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("missing"), isNull(), anyInt(), any(), any())).thenReturn(+ new S3Service.ListObjectsResult(List.of(), List.of(), false, null));+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "t", List.of(), "wh", "missing", "|", 0, false, false, null);++ boolean handled = S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');+ assertTrue(handled);++ PostgresWireDecoder in = new PostgresWireDecoder(testClient.getInputStream());+ PostgresWireDecoder.FrontendMessage err = in.nextMessage();+ assertEquals('E', err.type());+ assertTrue(new String(err.body(), StandardCharsets.US_ASCII).contains("not found"));+ assertEquals('Z', in.nextMessage().type());+ }++ @Test+ void backendErrorInsteadOfCopyInIsForwardedToClient() throws Exception {+ when(s3.objectExists("wh", "k")).thenReturn(true);+ when(s3.getObject("wh", "k")).thenReturn(+ new S3Object("wh", "k", "1\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "nosuch", List.of(), "wh", "k", "|", 0, false, false, null);++ Thread backend = backendThread(() -> {+ new PostgresWireDecoder(testBackend.getInputStream()).nextMessage();+ byte[] msg = "SERROR\0C42P01\0Mrelation \"nosuch\" does not exist\0\0".getBytes(StandardCharsets.US_ASCII);+ OutputStream out = testBackend.getOutputStream();+ out.write('E');+ out.write(intBytes(4 + msg.length));+ out.write(msg);+ out.write(new byte[]{'Z', 0, 0, 0, 5, 'I'});+ out.flush();+ });++ boolean handled = S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');+ joinBackend(backend);+ assertTrue(handled);++ PostgresWireDecoder in = new PostgresWireDecoder(testClient.getInputStream());+ assertEquals('E', in.nextMessage().type());+ assertEquals('Z', in.nextMessage().type());+ }++ @Test+ void midCopyFailureSendsExactlyOneErrorAndReadyForQuery() throws Exception {+ when(s3.objectExists("wh", "k")).thenReturn(true);+ when(s3.getObject("wh", "k")).thenThrow(new AwsException("InternalError", "boom", 500));+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "t", List.of(), "wh", "k", "|", 0, false, false, null);++ Thread backend = backendThread(() -> {+ OutputStream out = testBackend.getOutputStream();+ PostgresWireDecoder in = new PostgresWireDecoder(testBackend.getInputStream());+ assertEquals('Q', in.nextMessage().type());+ out.write(new byte[]{'G', 0, 0, 0, 7, 0, 0, 0});+ out.flush();+ // wait for the simulator's CopyFail 'f'+ PostgresWireDecoder.FrontendMessage m;+ while ((m = in.nextMessage()) != null && m.type() != 'f') {+ // discard any CopyData already buffered+ }+ assertEquals('f', m == null ? 0 : m.type());+ byte[] err = "SERROR\0C57014\0MCOPY aborted\0\0".getBytes(StandardCharsets.US_ASCII);+ out.write('E');+ out.write(intBytes(4 + err.length));+ out.write(err);+ out.write(new byte[]{'Z', 0, 0, 0, 5, 'I'});+ out.flush();+ });++ S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');+ joinBackend(backend);++ PostgresWireDecoder in = new PostgresWireDecoder(testClient.getInputStream());+ assertEquals('E', in.nextMessage().type());+ assertEquals('Z', in.nextMessage().type());+ testClient.setSoTimeout(400);+ assertThrows(SocketTimeoutException.class, () -> testClient.getInputStream().read(),+ "no second response should follow");+ }++ @Test+ void backendEofBeforeCopyInResponseSendsErrorToClient() throws Exception {+ when(s3.objectExists("wh", "k")).thenReturn(true);+ when(s3.getObject("wh", "k")).thenReturn(+ new S3Object("wh", "k", "1\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "t", List.of(), "wh", "k", "|", 0, false, false, null);++ Thread backend = backendThread(() -> {+ new PostgresWireDecoder(testBackend.getInputStream()).nextMessage();+ testBackend.close();+ });++ boolean handled = S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');+ joinBackend(backend);+ assertTrue(handled);++ PostgresWireDecoder in = new PostgresWireDecoder(testClient.getInputStream());+ assertEquals('E', in.nextMessage().type());+ assertEquals('Z', in.nextMessage().type());+ }++ @Test+ void synthesizedErrorReportsFailedTransactionStatusWhenInABlock() throws Exception {+ when(s3.objectExists("wh", "missing")).thenReturn(false);+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("missing"), isNull(), anyInt(), any(), any())).thenReturn(+ new S3Service.ListObjectsResult(List.of(), List.of(), false, null));+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "t", List.of(), "wh", "missing", "|", 0, false, false, null);++ Thread backend = backendThread(() -> {+ PostgresWireDecoder in = new PostgresWireDecoder(testBackend.getInputStream());+ PostgresWireDecoder.FrontendMessage q = in.nextMessage();+ assertEquals('Q', q.type());+ assertTrue(q.getSql().contains("FLOCI_ABORT_TX"));+ OutputStream out = testBackend.getOutputStream();+ byte[] err = "SERROR\0C42601\0Msyntax error\0\0".getBytes(StandardCharsets.US_ASCII);+ out.write('E');+ out.write(intBytes(4 + err.length));+ out.write(err);+ out.write(new byte[]{'Z', 0, 0, 0, 5, 'E'});+ out.flush();+ });++ S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'T');+ joinBackend(backend);++ PostgresWireDecoder in = new PostgresWireDecoder(testClient.getInputStream());+ assertEquals('E', in.nextMessage().type());+ PostgresWireDecoder.FrontendMessage ready = in.nextMessage();+ assertEquals('Z', ready.type());+ assertEquals('E', (char) ready.body()[0]);+ }++ @Test+ void paginatesPrefixListingAcrossMultiplePages() throws Exception {+ when(s3.objectExists("wh", "multi/")).thenReturn(false);+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("multi/"), isNull(), anyInt(), isNull(), isNull())).thenReturn(+ new S3Service.ListObjectsResult(List.of(+ new S3Object("wh", "multi/1", "1|a\n".getBytes(StandardCharsets.US_ASCII), "text/plain")),+ List.of(), true, "tok-1"));+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("multi/"), isNull(), anyInt(), eq("tok-1"), isNull())).thenReturn(+ new S3Service.ListObjectsResult(List.of(+ new S3Object("wh", "multi/2", "2|b\n".getBytes(StandardCharsets.US_ASCII), "text/plain")),+ List.of(), false, null));++ when(s3.getObject("wh", "multi/1")).thenReturn(+ new S3Object("wh", "multi/1", "1|a\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));+ when(s3.getObject("wh", "multi/2")).thenReturn(+ new S3Object("wh", "multi/2", "2|b\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));++ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "t", List.of(), "wh", "multi/", "|", 0, false, false, null);++ ByteArrayOutputStream copyData = new ByteArrayOutputStream();+ Thread backend = backendThread(() -> playHappyBackend(copyData));+ S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');+ joinBackend(backend);++ assertEquals("1|a\n2|b\n", copyData.toString(StandardCharsets.US_ASCII));+ }++ @Test+ void backendTimeoutAwaitingCopyInResponseClosesBackend() throws Exception {+ when(s3.objectExists("wh", "k")).thenReturn(true);+ when(s3.getObject("wh", "k")).thenReturn(+ new S3Object("wh", "k", "1\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "t", List.of(), "wh", "k", "|", 0, false, false, null);++ simBackend.setSoTimeout(100);++ Thread backend = backendThread(() -> {+ new PostgresWireDecoder(testBackend.getInputStream()).nextMessage();+ try {+ Thread.sleep(300);+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();+ }+ });++ boolean handled = S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');+ joinBackend(backend);+ assertTrue(handled);++ assertTrue(simBackend.isClosed(), "simBackend must be closed upon timeout");++ PostgresWireDecoder in = new PostgresWireDecoder(testClient.getInputStream());+ assertEquals('E', in.nextMessage().type());+ assertEquals('Z', in.nextMessage().type());+ }++ @Test+ void failedBackendTransactionAbortClosesBackendAndClientWithoutReportingFailedTransaction() throws Exception {+ when(s3.objectExists("wh", "missing")).thenReturn(false);+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("missing"), isNull(), anyInt(), any(), any())).thenReturn(+ new S3Service.ListObjectsResult(List.of(), List.of(), false, null));+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "t", List.of(), "wh", "missing", "|", 0, false, false, null);++ // Backend closes socket when abort query arrives, simulating dropped connection+ Thread backend = backendThread(() -> {+ PostgresWireDecoder in = new PostgresWireDecoder(testBackend.getInputStream());+ PostgresWireDecoder.FrontendMessage q = in.nextMessage();+ assertEquals('Q', q.type());+ assertTrue(q.getSql().contains("FLOCI_ABORT_TX"));+ testBackend.close();+ });++ S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'T');+ joinBackend(backend);++ assertTrue(simBackend.isClosed(), "simBackend must be closed upon synchronization failure");+ assertTrue(simClient.isClosed(), "simClient must be closed upon synchronization failure");++ PostgresWireDecoder in = new PostgresWireDecoder(testClient.getInputStream());+ PostgresWireDecoder.FrontendMessage err = in.nextMessage();+ assertEquals('E', err.type());+ assertNull(in.nextMessage(), "Client must not receive an unconfirmed ReadyForQuery");+ }++ @Test+ void backendReturnsNonETransactionStatusAfterAbortClosesBackendAndClient() throws Exception {+ when(s3.objectExists("wh", "missing")).thenReturn(false);+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("missing"), isNull(), anyInt(), any(), any())).thenReturn(+ new S3Service.ListObjectsResult(List.of(), List.of(), false, null));+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(+ "t", List.of(), "wh", "missing", "|", 0, false, false, null);++ // Fake backend returns ReadyForQuery with 'T' (e.g. abort query unexpectedly succeeded)+ Thread backend = backendThread(() -> {+ PostgresWireDecoder in = new PostgresWireDecoder(testBackend.getInputStream());+ PostgresWireDecoder.FrontendMessage q = in.nextMessage();+ assertEquals('Q', q.type());+ assertTrue(q.getSql().contains("FLOCI_ABORT_TX"));+ OutputStream out = testBackend.getOutputStream();+ out.write(new byte[]{'Z', 0, 0, 0, 5, 'T'});+ out.flush();+ });++ S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'T');+ joinBackend(backend);++ assertTrue(simBackend.isClosed(), "simBackend must be closed when abort returns non-E status");+ assertTrue(simClient.isClosed(), "simClient must be closed when abort returns non-E status");++ PostgresWireDecoder in = new PostgresWireDecoder(testClient.getInputStream());+ PostgresWireDecoder.FrontendMessage err = in.nextMessage();+ assertEquals('E', err.type());+ assertNull(in.nextMessage(), "Client must not receive an unconfirmed ReadyForQuery");+ }+}