Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(redshift): emulate Redshift COPY FROM s3 over Simple Query protocol (#3100)

Vũ Thế Huy85180a4d88ae
diff --git a/docs/services/redshift.md b/docs/services/redshift.md
index 01db4613f2..8d632a0085 100644
--- a/docs/services/redshift.md
+++ b/docs/services/redshift.md
@@ -4,9 +4,9 @@
**Management Endpoint:** `POST http://localhost:4566/` with `Action=` param
**Data Endpoint:** Floci's auth proxy on the `Endpoint` and `Port` returned by `DescribeClusters` (PostgreSQL wire protocol)
-Floci emulates Amazon Redshift by managing a real [PostgreSQL](https://www.postgresql.org/) Docker container per cluster behind a Redshift-shaped control plane. Each cluster sits behind a lightweight auth proxy on the Floci host, so the endpoint is reachable from outside Docker and the master password is validated at the proxy — a `ModifyCluster` password change takes effect for new connections immediately. Redshift speaks the PostgreSQL wire protocol, so the cluster endpoint returned by `DescribeClusters` works with any standard PostgreSQL driver (`psql`, JDBC, `psycopg`, …).
+Floci emulates Amazon Redshift by managing a real [PostgreSQL](https://www.postgresql.org/) Docker container per cluster behind a Redshift-shaped control plane. Each cluster sits behind a lightweight auth proxy on the Floci host, so the endpoint is reachable from outside Docker and the master password is validated at the proxy, a `ModifyCluster` password change takes effect for new connections immediately. Redshift speaks the PostgreSQL wire protocol, so the cluster endpoint returned by `DescribeClusters` works with any standard PostgreSQL driver (`psql`, JDBC, `psycopg`, …).
-> **Always read the host and port from `DescribeClusters`** rather than assuming a fixed port. PostgreSQL listens on `5432` *inside* the container; the port you connect to is dynamically assigned on the host and returned as `Clusters[0].Endpoint.Port`. Redshift's conventional port is `5439`, but the emulator does not bind it — use whatever `DescribeClusters` reports.
+> **Always read the host and port from `DescribeClusters`** rather than assuming a fixed port. PostgreSQL listens on `5432` *inside* the container; the port you connect to is dynamically assigned on the host and returned as `Clusters[0].Endpoint.Port`. Redshift's conventional port is `5439`, but the emulator does not bind it: use whatever `DescribeClusters` reports.
The container has **no persistent volume**: if the physical container survives a Floci restart it is adopted and its data is kept, but if the container itself is gone (host reboot, `docker rm`, a pruned dev box) the cluster comes back empty. Use `CreateClusterSnapshot` / `RestoreFromClusterSnapshot` to preserve data explicitly.
@@ -106,7 +106,7 @@ aws redshift delete-cluster \
```python
import psycopg
-# Read host and port from DescribeClusters — the host port is dynamic.
+# Read host and port from DescribeClusters: the host port is dynamic.
host, port = "localhost", 32768 # e.g. Clusters[0].Endpoint.Address / .Port
with psycopg.connect(f"host={host} port={port} dbname=dev user=admin password=Secret123") as conn:
conn.execute("CREATE TABLE people (name text)")
@@ -145,18 +145,42 @@ Floci's Redshift auth proxy inspects frontend queries on the PostgreSQL wire pro
- The rewrite only runs when the statement's first keyword is `CREATE TABLE` or `ALTER TABLE`. A `SELECT`, `INSERT`, function body, or string literal that merely contains one of these keywords is forwarded byte-for-byte. Single-quoted and dollar-quoted string literals are masked before the rewrite, so a keyword inside a quoted value (including in a later statement of a multi-statement query) is preserved.
- Columns legitimately named `distkey`, `sortkey`, or `encode` survive.
+### COPY from S3
+
+`COPY <table> [(<columns>)] FROM 's3://<bucket>/<keyOrPrefix>' [options]` sent over the Simple
+Query protocol is emulated: Floci reads the object (or every object under the prefix, in key
+order) through its own S3 service and streams the rows into the backing PostgreSQL container with
+`COPY ... FROM STDIN`.
+
+- Supported options: `DELIMITER`, `FORMAT CSV` (or a bare `CSV`), `GZIP`, `IGNOREHEADER <n>` and
+ `HEADER`, `NULL AS`, and an explicit column list.
+- The default framing is pipe-delimited text, matching Redshift. `FORMAT CSV` switches to CSV with
+ a comma default delimiter.
+- `IGNOREHEADER` and `HEADER` skip lines from the first resolved object only.
+- `GZIP` is the only input compression recognized; `BZIP2`, `LZOP` and `ZSTD` are not.
+- S3 access is authorized as an unsigned request: with `FLOCI_SERVICES_S3_ENFORCE_AUTH` off it is
+ unrestricted; with it on, bucket policy and public access settings apply.
+- Any other clause (`FIXEDWIDTH`, `JSON`, `PARQUET`, `AVRO`, `ORC`, `MANIFEST`, `MAXERROR`,
+ `DATEFORMAT`, `TIMEFORMAT`, `REGION`, `ENCODING`, `ESCAPE`, `REMOVEQUOTES`, `BLANKSASNULL`,
+ `EMPTYASNULL`, `TRUNCATECOLUMNS`, `ACCEPTINVCHARS`, credentials clauses, and so on) is not
+ recognized: the statement is forwarded unchanged and PostgreSQL returns its own error.
+- A multi-statement query whose COPY is followed by another statement is not intercepted; send the
+ COPY on its own.
+- Extended Query protocol COPY (a JDBC `PreparedStatement`, or pgjdbc's default
+ `preferQueryMode=extended`) is not intercepted. Use `preferQueryMode=simple`.
+
### Limitations
- Emulation runs on the **Simple Query protocol** (`'Q'`) only. Extended Query protocol statements (`Parse`/`Bind`/`Execute`) pass through untouched, including anything a JDBC `PreparedStatement` sends, and, with the pgjdbc default `preferQueryMode=extended`, plain `Statement` calls too. Connect with `preferQueryMode=simple` to exercise the interceptor from JDBC.
- The rewrite is textual (regex-based). It masks single-quoted string literals first, so `DEFAULT` / `CHECK` string values are safe, but it is **not** comment-aware and does not recognize escape strings (`E'...'`): an apostrophe inside a `--` or `/* */` comment can make the rewrite skip a Redshift clause. That fails safe: the statement then reaches PostgreSQL, which returns its own syntax error, but avoid apostrophes-in-comments in `CREATE TABLE` / `ALTER TABLE`.
- A `rewrite` failure or any statement the interceptor does not recognize is forwarded unmodified (fail-open); PostgreSQL then rejects the Redshift-only syntax itself.
- Simple Query ('Q') messages larger than 16 MiB bypass the interceptor and stream through verbatim without heap buffering; non-query traffic also streams through with no size limit.
-- `COPY ... FROM 's3://...'` and `UNLOAD (...) TO 's3://...'` are **not** yet emulated (planned).
+- `UNLOAD (...) TO 's3://...'` is **not** yet emulated (planned).
## Out of Scope
-- Real Redshift SQL semantics: the data plane is stock PostgreSQL. Redshift-only table DDL keywords (DISTSTYLE / DISTKEY / SORTKEY / ENCODE) are stripped so CREATE TABLE / ALTER TABLE executes (see [SQL Interceptor](#sql-interceptor)), but the distribution/sort behavior they request is not; COPY/UNLOAD from S3 and SUPER/SPECTRUM are not emulated.
-- Multi-node clusters — `NodeType` and `NumberOfNodes` are stored as metadata; every cluster is a single PostgreSQL container.
+- Real Redshift SQL semantics: the data plane is stock PostgreSQL. Redshift-only table DDL keywords (DISTSTYLE / DISTKEY / SORTKEY / ENCODE) are stripped so CREATE TABLE / ALTER TABLE executes (see [SQL Interceptor](#sql-interceptor)), but the distribution/sort behavior they request is not; UNLOAD to S3 and SUPER/SPECTRUM are not emulated.
+- Multi-node clusters: `NodeType` and `NumberOfNodes` are stored as metadata; every cluster is a single PostgreSQL container.
- Parameter groups apply no real engine settings; values are stored and echoed back only.
- Subnet groups, VPC routing, and security groups are metadata only.
- Resize, pause/resume, IAM authentication, snapshot schedules, and cross-region snapshot copy.
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.java
new file mode 100644
index 0000000000..3c990300f1
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.java
@@ -0,0 +1,275 @@
+package io.github.hectorvent.floci.services.redshift.proxy;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Recognises a Simple Query statement that is an S3 {@code COPY <table> FROM 's3://...'}.
+ * Any other statement, or a COPY that uses an option this simulator cannot honour, returns
+ * {@code null} so the bridge falls back to DDL rewriting and PostgreSQL reports its own error.
+ */
+public final class CopyStatementParser {
+
+ public record S3CopyFrom(
+ String targetTable,
+ List<String> columns,
+ String bucket,
+ String keyOrPrefix,
+ String delimiter,
+ int headerLines,
+ boolean gzip,
+ boolean csv,
+ String nullAs) {
+ }
+
+ private static final Pattern COPY_PATTERN = Pattern.compile(
+ "(?is)^\\s*COPY\\s+((?:\"[^\"]*\"|[^\\s(])+)\\s*(?:\\(([^)]*)\\))?\\s+FROM\\s*"
+ + "['\"]s3://([^/'\"\\s]+)(?:/([^'\"\\s]*))?['\"]\\s*(.*)$");
+
+ private static final Pattern QUALIFIED_NAME = Pattern.compile(
+ "(?:[A-Za-z_][A-Za-z0-9_$]*|\"[^\"]+\")(?:\\.(?:[A-Za-z_][A-Za-z0-9_$]*|\"[^\"]+\"))?");
+ private static final Pattern SIMPLE_NAME = Pattern.compile("[A-Za-z_][A-Za-z0-9_$]*|\"[^\"]+\"");
+
+ private static final Pattern DELIMITER_PATTERN = Pattern.compile(
+ "(?i)\\bDELIMITER\\s+(?:AS\\s+)?(?:'((?:[^']|'')*)'|\"([^\"]*)\"|([^\\s;]+))");
+ private static final Pattern NULL_AS_PATTERN = Pattern.compile(
+ "(?i)\\bNULL\\s+(?:AS\\s+)?(?:'((?:[^']|'')*)'|\"([^\"]*)\")");
+ private static final Pattern IGNOREHEADER_PATTERN = Pattern.compile(
+ "(?i)\\bIGNOREHEADER\\s+(?:AS\\s+)?(\\d+)\\b");
+ private static final Pattern HEADER_PATTERN = Pattern.compile("(?i)\\bHEADER\\b");
+ private static final Pattern GZIP_PATTERN = Pattern.compile("(?i)\\bGZIP\\b");
+ private static final Pattern CSV_PATTERN = Pattern.compile("(?i)\\b(?:FORMAT\\s+(?:AS\\s+)?)?CSV\\b");
+
+ /**
+ * Options this simulator does not implement. A COPY carrying any of these is not intercepted:
+ * the original statement is forwarded so PostgreSQL rejects it, rather than the simulator
+ * silently loading the data with the wrong framing.
+ */
+ private static final Pattern UNSUPPORTED_CLAUSE = Pattern.compile(
+ "(?i)\\b(FIXEDWIDTH|PARQUET|AVRO|ORC|JSON|SHAPEFILE|BZIP2|LZOP|ZSTD|MANIFEST|MAXERROR"
+ + "|DATEFORMAT|TIMEFORMAT|ENCRYPTED|ENCODING|REGION|CREDENTIALS|IAM_ROLE"
+ + "|ACCESS_KEY_ID|SECRET_ACCESS_KEY|SESSION_TOKEN|MASTER_SYMMETRIC_KEY|KMS_KEY_ID"
+ + "|ACCEPTINVCHARS|ACCEPTANYDATE|BLANKSASNULL|EMPTYASNULL|FILLRECORD|TRIMBLANKS"
+ + "|TRUNCATECOLUMNS|IGNOREBLANKLINES|ESCAPE|REMOVEQUOTES|EXPLICIT_IDS|COMPUPDATE"
+ + "|STATUPDATE|NOLOAD|ROUNDEC|QUOTE|SSH|READRATIO|COMPROWS|DIMENSION)\\b");
+
+ /** A {@code ;} followed by another statement: only a lone trailing {@code ;} is tolerated. */
+ private static final Pattern TRAILING_STATEMENT = Pattern.compile(";\\s*\\S");
+
+ private CopyStatementParser() {
+ }
+
+ public static S3CopyFrom parse(String sql) {
+ if (sql == null || sql.isBlank()) {
+ return null;
+ }
+ String cleaned = stripLeadingComments(sql.trim());
+ Matcher matcher = COPY_PATTERN.matcher(cleaned);
+ if (!matcher.matches()) {
+ return null;
+ }
+
+ String table = matcher.group(1).trim();
+ if (!QUALIFIED_NAME.matcher(table).matches()) {
+ return null;
+ }
+
+ List<String> columns = List.of();
+ String columnsGroup = matcher.group(2);
+ if (columnsGroup != null && !columnsGroup.isBlank()) {
+ columns = Arrays.stream(columnsGroup.split(","))
+ .map(String::trim)
+ .filter(s -> !s.isEmpty())
+ .toList();
+ if (columns.stream().anyMatch(c -> !SIMPLE_NAME.matcher(c).matches())) {
+ return null;
+ }
+ }
+
+ String bucket = matcher.group(3);
+ String keyOrPrefix = matcher.group(4) != null ? matcher.group(4) : "";
+ String options = matcher.group(5) != null ? matcher.group(5) : "";
+
+ // Scan for keywords and separators on a copy with string literals blanked out, so a
+ // value like NULL AS 'json' or DELIMITER ';' cannot trip a keyword or the trailing check.
+ String flagScan = blankQuoted(options);
+ if (UNSUPPORTED_CLAUSE.matcher(flagScan).find() || TRAILING_STATEMENT.matcher(flagScan).find()) {
+ return null;
+ }
+
+ int semiIdx = flagScan.indexOf(';');
+ if (semiIdx >= 0) {
+ if (!flagScan.substring(semiIdx + 1).isBlank()) {
+ return null;
+ }
+ options = options.substring(0, semiIdx);
+ }
+
+ boolean csv = false;
+ boolean gzip = false;
+ String nullAs = null;
+ String delimiter = null;
+ int headerLines = 0;
+
+ boolean seenCsv = false;
+ boolean seenGzip = false;
+ boolean seenNull = false;
+ boolean seenDelimiter = false;
+ boolean seenHeader = false;
+
+ Matcher csvMatcher = CSV_PATTERN.matcher(options);
+ Matcher gzipMatcher = GZIP_PATTERN.matcher(options);
+ Matcher ignoreHeaderMatcher = IGNOREHEADER_PATTERN.matcher(options);
+ Matcher headerMatcher = HEADER_PATTERN.matcher(options);
+ Matcher delimiterMatcher = DELIMITER_PATTERN.matcher(options);
+ Matcher nullMatcher = NULL_AS_PATTERN.matcher(options);
+
+ int offset = 0;
+ int len = options.length();
+ while (offset < len) {
+ while (offset < len && Character.isWhitespace(options.charAt(offset))) {
+ offset++;
+ }
+ if (offset >= len) {
+ break;
+ }
+
+ if (matchClause(csvMatcher, offset, len)) {
+ if (seenCsv) {
+ return null;
+ }
+ seenCsv = true;
+ csv = true;
+ offset = csvMatcher.end();
+ } else if (matchClause(gzipMatcher, offset, len)) {
+ if (seenGzip) {
+ return null;
+ }
+ seenGzip = true;
+ gzip = true;
+ offset = gzipMatcher.end();
+ } else if (matchClause(ignoreHeaderMatcher, offset, len)) {
+ if (seenHeader) {
+ return null;
+ }
+ seenHeader = true;
+ headerLines = Math.max(0, Integer.parseInt(ignoreHeaderMatcher.group(1)));
+ offset = ignoreHeaderMatcher.end();
+ } else if (matchClause(headerMatcher, offset, len)) {
+ if (seenHeader) {
+ return null;
+ }
+ seenHeader = true;
+ headerLines = 1;
+ offset = headerMatcher.end();
+ } else if (matchClause(delimiterMatcher, offset, len)) {
+ if (seenDelimiter) {
+ return null;
+ }
+ seenDelimiter = true;
+ delimiter = extractDelimiterValue(delimiterMatcher);
+ offset = delimiterMatcher.end();
+ } else if (matchClause(nullMatcher, offset, len)) {
+ if (seenNull) {
+ return null;
+ }
+ seenNull = true;
+ nullAs = extractNullValue(nullMatcher);
+ offset = nullMatcher.end();
+ } else {
+ return null;
+ }
+ }
+
+ if (delimiter == null) {
+ delimiter = csv ? "," : "|";
+ }
+
+ return new S3CopyFrom(table, columns, bucket, keyOrPrefix, delimiter, headerLines, gzip, csv, nullAs);
+ }
+
+ private static boolean matchClause(Matcher m, int start, int end) {
+ m.region(start, end);
+ return m.lookingAt();
+ }
+
+ private static String extractDelimiterValue(Matcher matcher) {
+ String value = matcher.group(1) != null ? matcher.group(1)
+ : matcher.group(2) != null ? matcher.group(2)
+ : matcher.group(3);
+ if (value != null) {
+ value = unescape(value);
+ }
+ return "\\t".equals(value) || "\t".equals(value) ? "\t" : value;
+ }
+
+ private static String extractNullValue(Matcher matcher) {
+ String value = matcher.group(1) != null ? matcher.group(1) : matcher.group(2);
+ return unescape(value);
+ }
+
+ private static String unescape(String s) {
+ if (s == null) {
+ return null;
+ }
+ return s.replace("''", "'").replace("\\\\", "\\");
+ }
+
+ /** Replace every character inside a single- or double-quoted run with a space. */
+ private static String blankQuoted(String s) {
+ StringBuilder out = new StringBuilder(s.length());
+ boolean inSingle = false;
+ boolean inDouble = false;
+ for (int i = 0; i < s.length(); i++) {
+ char c = s.charAt(i);
+ if (inSingle) {
+ if (c == '\'') {
+ if (i + 1 < s.length() && s.charAt(i + 1) == '\'') {
+ out.append(" ");
+ i++;
+ continue;
+ }
+ inSingle = false;
+ }
+ out.append(' ');
+ } else if (inDouble) {
+ if (c == '"') {
+ inDouble = false;
+ }
+ out.append(' ');
+ } else if (c == '\'') {
+ inSingle = true;
+ out.append(' ');
+ } else if (c == '"') {
+ inDouble = true;
+ out.append(' ');
+ } else {
+ out.append(c);
+ }
+ }
+ return out.toString();
+ }
+
+ private static String stripLeadingComments(String sql) {
+ String current = sql;
+ while (true) {
+ if (current.startsWith("--")) {
+ int newline = current.indexOf('\n');
+ if (newline < 0) {
+ return "";
+ }
+ current = current.substring(newline + 1).stripLeading();
+ } else if (current.startsWith("/*")) {
+ int close = current.indexOf("*/");
+ if (close < 0) {
+ return "";
+ }
+ current = current.substring(close + 2).stripLeading();
+ } else {
+ return current;
+ }
+ }
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftAuthProxy.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftAuthProxy.java
index 7819377f1e..a2591315c1 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftAuthProxy.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftAuthProxy.java
@@ -4,6 +4,7 @@ import io.github.hectorvent.floci.services.rds.proxy.PostgresProtocolHandler;
import io.github.hectorvent.floci.services.rds.proxy.RdsAuthProxy;
import io.github.hectorvent.floci.services.rds.proxy.RdsProxyTlsCertificates;
import io.github.hectorvent.floci.services.rds.proxy.RdsSigV4Validator;
+import io.github.hectorvent.floci.services.s3.S3Service;
import org.jboss.logging.Logger;
import java.io.IOException;
@@ -32,6 +33,7 @@ public class RedshiftAuthProxy {
private final RdsSigV4Validator sigV4;
private final RdsProxyTlsCertificates tlsCertificates;
private final RdsAuthProxy.PasswordValidator passwordValidator;
+ private final S3Service s3Service;
private volatile boolean running;
private ServerSocket serverSocket;
@@ -39,7 +41,8 @@ public class RedshiftAuthProxy {
public RedshiftAuthProxy(String clusterKey, String backendHost, int backendPort,
String masterUsername, String masterPassword, String dbName,
RdsSigV4Validator sigV4, RdsProxyTlsCertificates tlsCertificates,
- RdsAuthProxy.PasswordValidator passwordValidator) {
+ RdsAuthProxy.PasswordValidator passwordValidator,
+ S3Service s3Service) {
this.clusterKey = clusterKey;
this.backendHost = backendHost;
this.backendPort = backendPort;
@@ -49,6 +52,7 @@ public class RedshiftAuthProxy {
this.sigV4 = sigV4;
this.tlsCertificates = tlsCertificates;
this.passwordValidator = passwordValidator;
+ this.s3Service = s3Service;
}
public void start(int proxyPort) throws IOException {
@@ -145,7 +149,7 @@ public class RedshiftAuthProxy {
if (session != null) {
// Redshift-only DDL (DISTKEY/SORTKEY/ENCODE/...) is rewritten for the plain
// PostgreSQL backend on the way through; every other message is relayed verbatim.
- new RedshiftInterceptingBridge(session.client(), backend).run();
+ new RedshiftInterceptingBridge(session.client(), backend, s3Service).run();
}
} catch (Exception e) {
LOG.debugv("Redshift connection error for cluster {0}: {1}", clusterKey, e.getMessage());
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftInterceptingBridge.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftInterceptingBridge.java
index c554f873b8..85d495e4db 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftInterceptingBridge.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftInterceptingBridge.java
@@ -1,117 +1,292 @@
package io.github.hectorvent.floci.services.redshift.proxy;
+import io.github.hectorvent.floci.services.s3.S3Service;
import org.jboss.logging.Logger;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.Socket;
-import java.util.concurrent.atomic.AtomicBoolean;
+import java.net.SocketTimeoutException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.LockSupport;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.function.IntConsumer;
/**
- * Replaces the transparent {@code PostgresProtocolHandler.bridge} for Redshift connections so
- * Simple Query ({@code 'Q'}) DDL can be rewritten for a plain PostgreSQL backend before it is
- * forwarded.
+ * Replaces the transparent {@code bridge()} for Redshift connections so Simple Query ({@code 'Q'})
+ * traffic can be inspected before it reaches the backing PostgreSQL container.
*
- * <p><b>Model.</b> Backend&rarr;client is a verbatim byte pump on a virtual thread, exactly as a
- * plain proxy: extended-protocol pipelines, {@code COPY … FROM STDIN}, asynchronous
- * {@code NotificationResponse} and backend EOF all flow through untouched. Client&rarr;backend is
- * a framed loop: every frontend message is forwarded opaque <em>except</em> a {@code 'Q'}, whose
- * SQL is run through {@link RedshiftSqlInterceptor#rewrite} and re-encoded only if it changed.
+ * <p><b>Model.</b> Backend&rarr;client is a byte pump on a virtual thread. Client&rarr;backend is a
+ * framed loop: every frontend message is forwarded opaque except a {@code 'Q'}, whose SQL is run
+ * through {@link RedshiftSqlInterceptor#rewrite} and re-encoded only if it changed.
*
- * <p><b>Fail-open.</b> A {@code rewrite} failure, or any statement the interceptor leaves
- * untouched, forwards the original bytes; PostgreSQL then returns its own error. The bridge never
- * closes the connection because of a rewrite failure.
+ * <p><b>Backend ownership.</b> A future interceptor that injects its own query and reads the reply
+ * (the S3 COPY simulator) must own the backend exclusively and only start when no earlier request
+ * is still being answered. {@link #backendLock} serialises pump reads against such an exchange, and
+ * {@link #outstandingResponses} plus {@link #pumpBetweenMessages} gate it to a safe boundary. If the
+ * backend cannot be caught idle within {@link #PUMP_PARK_WAIT_MS}, the exchange is skipped and the
+ * original {@code 'Q'} is forwarded (fail-open).
*/
public class RedshiftInterceptingBridge {
private static final Logger LOG = Logger.getLogger(RedshiftInterceptingBridge.class);
+ private static final int PUMP_READ_TIMEOUT_MS = 200;
+ private static final int CLIENT_READ_TIMEOUT_MS = 10_000;
+ private static final long PUMP_PARK_WAIT_MS = 2_000L;
+ /** Bounded read timeout while an exchange owns the backend, so a stalled backend cannot hang the session. */
+ private static final int EXCHANGE_READ_TIMEOUT_MS = 60_000;
+ private static final long BUSY_BACKOFF_NANOS = 2_000_000L;
+
private final Socket client;
private final Socket backend;
- private final AtomicBoolean closed = new AtomicBoolean();
+ private final S3Service s3Service;
+
+ private final ReentrantLock backendLock = new ReentrantLock(true);
+ private volatile boolean pumpBetweenMessages = true;
+ private final AtomicInteger outstandingResponses = new AtomicInteger(0);
+ private volatile boolean pumpFinished = false;
+ /** Transaction-status byte from the backend's most recent {@code ReadyForQuery}; {@code 'I'} until one is seen. */
+ private volatile char lastBackendStatus = 'I';
- public RedshiftInterceptingBridge(Socket client, Socket backend) {
+ public RedshiftInterceptingBridge(Socket client, Socket backend, S3Service s3Service) {
this.client = client;
this.backend = backend;
+ this.s3Service = s3Service;
+ }
+
+ @FunctionalInterface
+ interface BackendExchange {
+ /** @return {@code true} if the exchange handled the query; {@code false} to forward the original. */
+ boolean run() throws IOException;
}
public void run() {
try {
- InputStream clientIn = client.getInputStream();
- OutputStream backendOut = backend.getOutputStream();
-
+ backend.setSoTimeout(PUMP_READ_TIMEOUT_MS);
+ client.setSoTimeout(CLIENT_READ_TIMEOUT_MS);
Thread.ofVirtual().name("redshift-pump-backend-to-client").start(this::pumpBackendToClient);
+ InputStream clientIn = client.getInputStream();
+ OutputStream backendOut = backend.getOutputStream();
PostgresWireDecoder decoder = new PostgresWireDecoder(clientIn);
+
while (true) {
- PostgresWireDecoder.FrontendMessage msg = decoder.nextMessage(backendOut);
+ PostgresWireDecoder.FrontendMessage msg;
+ try {
+ msg = decoder.nextMessage();
+ } catch (SocketTimeoutException e) {
+ if (decoder.isBetweenMessages()) {
+ continue;
+ }
+ LOG.warnv("Client socket timed out mid-message: {0}", e.getMessage());
+ break;
+ }
if (msg == null) {
- break; // client EOF
+ break;
}
if (!msg.isQuery()) {
- if (msg.type() == 'X') { // Terminate
+ if (msg.type() == 'S') {
+ outstandingResponses.incrementAndGet();
+ }
+ backendOut.write(msg.toPacketBytes());
+ backendOut.flush();
+ if (msg.type() == 'X') {
break;
}
continue;
}
String sql = msg.getSql();
+
+ CopyStatementParser.S3CopyFrom copyFrom = null;
+ try {
+ copyFrom = CopyStatementParser.parse(sql);
+ } catch (RuntimeException e) {
+ LOG.warnv("CopyStatementParser failed, forwarding original query: {0}", e.getMessage());
+ }
+
+ if (copyFrom != null) {
+ CopyStatementParser.S3CopyFrom spec = copyFrom;
+ boolean intercepted = runWithBackendOwned(
+ () -> S3CopySimulator.runCopyFrom(client, backend, spec, s3Service, lastBackendStatus,
+ status -> lastBackendStatus = (char) status));
+ if (intercepted) {
+ continue;
+ }
+ }
+
byte[] toBackend = msg.toPacketBytes();
try {
String rewritten = RedshiftSqlInterceptor.rewrite(sql);
- if (rewritten != sql) { // identity: rewrite returns the same instance when nothing matched
+ if (rewritten != sql) {
toBackend = PostgresWireDecoder.encodeQuery(rewritten);
}
} catch (RuntimeException e) {
LOG.warnv("RedshiftSqlInterceptor failed, forwarding original query: {0}", e.getMessage());
toBackend = msg.toPacketBytes();
}
+ outstandingResponses.incrementAndGet();
backendOut.write(toBackend);
backendOut.flush();
- // The response returns over the untouched pump.
}
} catch (IOException e) {
LOG.debugv(e, "RedshiftInterceptingBridge client loop ended");
} catch (Exception e) {
LOG.warnv(e, "Unexpected error in RedshiftInterceptingBridge");
} finally {
- closeBoth();
+ closeQuietly(client, "client");
+ closeQuietly(backend, "backend");
+ }
+ }
+
+ /**
+ * Take exclusive ownership of an idle backend at a wire-message boundary, run {@code exchange},
+ * then release it. Returns {@code false} if the backend could not be caught idle within
+ * {@link #PUMP_PARK_WAIT_MS}; the caller must then forward the original {@code 'Q'} unmodified.
+ */
+ boolean runWithBackendOwned(BackendExchange exchange) throws IOException {
+ long deadlineNanos = System.nanoTime() + PUMP_PARK_WAIT_MS * 1_000_000L;
+ while (true) {
+ if (pumpFinished) {
+ return false;
+ }
+ boolean locked;
+ try {
+ locked = backendLock.tryLock(PUMP_READ_TIMEOUT_MS, TimeUnit.MILLISECONDS);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new IOException("interrupted while acquiring the backend lock", e);
+ }
+ if (!locked) {
+ if (System.nanoTime() >= deadlineNanos) {
+ LOG.warn("could not take the backend stream in time; forwarding the COPY unintercepted");
+ return false;
+ }
+ continue;
+ }
+ boolean backendBusy;
+ try {
+ backendBusy = !pumpBetweenMessages || outstandingResponses.get() > 0;
+ if (!backendBusy || pumpFinished) {
+ backend.setSoTimeout(EXCHANGE_READ_TIMEOUT_MS);
+ try {
+ return exchange.run();
+ } finally {
+ try {
+ backend.setSoTimeout(PUMP_READ_TIMEOUT_MS);
+ } catch (IOException e) {
+ LOG.debugv(e, "could not restore backend read timeout (socket closing)");
+ }
+ }
+ }
+ } finally {
+ backendLock.unlock();
+ }
+ if (System.nanoTime() >= deadlineNanos) {
+ LOG.warn("backend did not go idle in time; forwarding the COPY unintercepted");
+ return false;
+ }
+ LockSupport.parkNanos(BUSY_BACKOFF_NANOS);
}
}
private void pumpBackendToClient() {
+ WireFrameTracker tracker = new WireFrameTracker(status -> {
+ lastBackendStatus = (char) status;
+ outstandingResponses.updateAndGet(v -> Math.max(0, v - 1));
+ });
try {
InputStream backendIn = backend.getInputStream();
OutputStream clientOut = client.getOutputStream();
- byte[] buf = new byte[8192];
- int n;
- while ((n = backendIn.read(buf)) != -1) {
- clientOut.write(buf, 0, n);
- clientOut.flush();
+ byte[] buffer = new byte[8192];
+ while (true) {
+ backendLock.lockInterruptibly();
+ try {
+ int read;
+ try {
+ read = backendIn.read(buffer);
+ } catch (SocketTimeoutException e) {
+ continue;
+ }
+ if (read == -1) {
+ break;
+ }
+ clientOut.write(buffer, 0, read);
+ clientOut.flush();
+ tracker.consume(buffer, 0, read);
+ pumpBetweenMessages = tracker.betweenMessages();
+ } finally {
+ backendLock.unlock();
+ }
}
} catch (IOException e) {
LOG.debugv(e, "backend->client pump ended");
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
} finally {
- closeBoth();
- }
- }
-
- private void closeBoth() {
- if (closed.compareAndSet(false, true)) {
+ pumpFinished = true;
closeQuietly(client, "client");
closeQuietly(backend, "backend");
}
}
- private void closeQuietly(Socket s, String which) {
+ private void closeQuietly(Socket socket, String which) {
try {
- if (s != null && !s.isClosed()) {
- s.close();
+ if (socket != null && !socket.isClosed()) {
+ socket.close();
}
} catch (IOException e) {
LOG.debugv(e, "error closing {0} socket", which);
}
}
+
+ /**
+ * Tracks PostgreSQL wire-message boundaries in a byte stream without buffering it. Each backend
+ * message is {@code type(1) . length(int32, includes itself) . body}. {@link #betweenMessages()}
+ * is true exactly when the next byte would begin a new message; {@code onReadyForQuery} fires
+ * once per completed {@code 'Z'} with its one-byte transaction-status payload.
+ */
+ static final class WireFrameTracker {
+ private final IntConsumer onReadyForQuery;
+ private int headerBytesSeen = 0;
+ private final byte[] header = new byte[5];
+ private long bodyRemaining = 0;
+ private char pendingType = 0;
+
+ WireFrameTracker(IntConsumer onReadyForQuery) {
+ this.onReadyForQuery = onReadyForQuery;
+ }
+
+ boolean betweenMessages() {
+ return headerBytesSeen == 0 && bodyRemaining == 0;
+ }
+
+ void consume(byte[] buf, int off, int len) {
+ for (int i = off; i < off + len; i++) {
+ if (bodyRemaining > 0) {
+ int current = buf[i] & 0xFF;
+ bodyRemaining--;
+ if (bodyRemaining == 0 && pendingType == 'Z') {
+ onReadyForQuery.accept(current);
+ }
+ continue;
+ }
+ header[headerBytesSeen++] = buf[i];
+ if (headerBytesSeen == 5) {
+ long msgLen = ((header[1] & 0xFFL) << 24) | ((header[2] & 0xFFL) << 16)
+ | ((header[3] & 0xFFL) << 8) | (header[4] & 0xFFL);
+ pendingType = (char) (header[0] & 0xFF);
+ bodyRemaining = Math.max(0, msgLen - 4);
+ headerBytesSeen = 0;
+ if (bodyRemaining == 0 && pendingType == 'Z') {
+ onReadyForQuery.accept('I');
+ }
+ }
+ }
+ }
+ }
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManager.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManager.java
index a51b8c7ae2..aebaa1d340 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManager.java
@@ -3,6 +3,7 @@ package io.github.hectorvent.floci.services.redshift.proxy;
import io.github.hectorvent.floci.services.rds.proxy.RdsAuthProxy;
import io.github.hectorvent.floci.services.rds.proxy.RdsProxyTlsCertificates;
import io.github.hectorvent.floci.services.rds.proxy.RdsSigV4Validator;
+import io.github.hectorvent.floci.services.s3.S3Service;
import io.quarkus.runtime.ShutdownEvent;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.enterprise.event.Observes;
@@ -24,6 +25,7 @@ public class RedshiftProxyManager {
private final RdsSigV4Validator sigV4Validator;
private final RdsProxyTlsCertificates tlsCertificates;
+ private final S3Service s3Service;
private final ConcurrentHashMap<String, RedshiftAuthProxy> proxies = new ConcurrentHashMap<>();
/**
* Proxies whose listener could not be closed during a failed start or replace. The
@@ -34,9 +36,11 @@ public class RedshiftProxyManager {
private final ConcurrentHashMap<String, RedshiftAuthProxy> unclosableProxies = new ConcurrentHashMap<>();
@Inject
- public RedshiftProxyManager(RdsSigV4Validator sigV4Validator, RdsProxyTlsCertificates tlsCertificates) {
+ public RedshiftProxyManager(RdsSigV4Validator sigV4Validator, RdsProxyTlsCertificates tlsCertificates,
+ S3Service s3Service) {
this.sigV4Validator = sigV4Validator;
this.tlsCertificates = tlsCertificates;
+ this.s3Service = s3Service;
}
public synchronized void startProxy(String relayKey, int proxyPort,
@@ -50,7 +54,7 @@ public class RedshiftProxyManager {
tlsCertificates.ensureHost(advertisedHost);
RedshiftAuthProxy proxy = new RedshiftAuthProxy(
relayKey, backendHost, backendPort, masterUsername, masterPassword, dbName,
- sigV4Validator, tlsCertificates, passwordValidator);
+ sigV4Validator, tlsCertificates, passwordValidator, s3Service);
try {
proxy.start(proxyPort);
} catch (IOException | RuntimeException e) {
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.java
new file mode 100644
index 0000000000..3ff45c0f6d
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.java
@@ -0,0 +1,404 @@
+package io.github.hectorvent.floci.services.redshift.proxy;
+
+import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.services.s3.S3Service;
+import io.github.hectorvent.floci.services.s3.model.S3Object;
+import org.jboss.logging.Logger;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.Socket;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.zip.GZIPInputStream;
+
+/**
+ * Emulates {@code COPY <table> FROM 's3://bucket/keyOrPrefix'} by reading the S3 object (or every
+ * object under the prefix, in key order) through {@link S3Service} and streaming it into the
+ * backing PostgreSQL container over a fabricated {@code COPY <table> FROM STDIN} exchange.
+ *
+ * <p>The caller must own the backend socket exclusively for the whole call: this class writes a
+ * query and reads its response frames directly. Every exit path leaves the client with exactly one
+ * response for its original {@code 'Q'} (a {@code CommandComplete}/{@code ReadyForQuery} on success,
+ * a single {@code ErrorResponse}/{@code ReadyForQuery} on failure) and never closes the connection.
+ */
+public final class S3CopySimulator {
+
+ private static final Logger LOG = Logger.getLogger(S3CopySimulator.class);
+
+ private static final int LIST_PAGE_SIZE = 1000;
+ private static final int CHUNK = 8192;
+
+ private static final String SQLSTATE_INSUFFICIENT_PRIVILEGE = "42501";
+ private static final String SQLSTATE_INTERNAL = "XX000";
+
+ private S3CopySimulator() {
+ }
+
+ /**
+ * @param txStatus the transaction-status byte from the client's last {@code ReadyForQuery}
+ * ({@code 'I'} idle, {@code 'T'} in a block, {@code 'E'} failed block); a
+ * synthesized error reports {@code 'I'} outside a block and {@code 'E'} inside.
+ * @return {@code true} when the exchange was handled (success or a clean error sent to the
+ * client). Part 4 always handles; the value exists so a later interceptor can decline.
+ */
+ public static boolean runCopyFrom(Socket client, Socket backend,
+ CopyStatementParser.S3CopyFrom spec, S3Service s3,
+ char txStatus) throws IOException {
+ return runCopyFrom(client, backend, spec, s3, txStatus, null);
+ }
+
+ public static boolean runCopyFrom(Socket client, Socket backend,
+ CopyStatementParser.S3CopyFrom spec, S3Service s3,
+ char txStatus, java.util.function.IntConsumer onStatusChange) throws IOException {
+ try {
+ s3.authorizeAnonymousListBucket(spec.bucket());
+ } catch (AwsException e) {
+ LOG.debugv(e, "ListBucket denied for COPY from s3://{0}/{1}", spec.bucket(), spec.keyOrPrefix());
+ sendAccessDenied(client, backend, spec, txStatus, onStatusChange);
+ return true;
+ }
+
+ List<String> keys;
+ try {
+ keys = resolveKeys(spec, s3);
+ } catch (AwsException e) {
+ LOG.warnv(e, "failed to list s3://{0}/{1} for COPY", spec.bucket(), spec.keyOrPrefix());
+ sendError(client, backend, SQLSTATE_INTERNAL,
+ "S3 COPY could not list s3://" + spec.bucket() + "/" + spec.keyOrPrefix(), txStatus, onStatusChange);
+ return true;
+ }
+
+ try {
+ for (String key : keys) {
+ s3.authorizeAnonymousGetObject(spec.bucket(), key);
+ }
+ } catch (AwsException e) {
+ LOG.debugv(e, "GetObject denied for COPY from s3://{0}/{1}", spec.bucket(), spec.keyOrPrefix());
+ sendAccessDenied(client, backend, spec, txStatus, onStatusChange);
+ return true;
+ }
+
+ if (keys.isEmpty()) {
+ sendError(client, backend, SQLSTATE_INTERNAL,
+ "S3 object s3://" + spec.bucket() + "/" + spec.keyOrPrefix() + " not found", txStatus, onStatusChange);
+ return true;
+ }
+
+ OutputStream backendOut = backend.getOutputStream();
+ backendOut.write(PostgresWireDecoder.encodeQuery(fabricateCopy(spec)));
+ backendOut.flush();
+
+ PostgresWireDecoder backendDecoder = new PostgresWireDecoder(backend.getInputStream());
+ PostgresWireDecoder.FrontendMessage first;
+ try {
+ first = nextNonAsync(backendDecoder, client);
+ } catch (IOException e) {
+ LOG.warnv(e, "backend read failed while awaiting CopyInResponse");
+ closeQuietly(backend);
+ sendError(client, null, SQLSTATE_INTERNAL, "S3 COPY failed: backend closed or timed out", txStatus, onStatusChange);
+ closeQuietly(client);
+ return true;
+ }
+ if (first == null) {
+ LOG.warn("backend closed before answering the fabricated COPY");
+ closeQuietly(backend);
+ sendError(client, null, SQLSTATE_INTERNAL, "S3 COPY failed: backend closed before COPY started", txStatus, onStatusChange);
+ closeQuietly(client);
+ return true;
+ }
+ if (first.type() != 'G') {
+ // Backend rejected the COPY itself (e.g. no such table). Its ErrorResponse and the
+ // ReadyForQuery that follows are the client's one response.
+ forward(client, first);
+ drainToReadyForQuery(backendDecoder, client, onStatusChange);
+ return true;
+ }
+
+ // The CopyIn stream is open. Any failure from here is resolved with exactly one response:
+ // a CopyFail to the backend, whose ErrorResponse/ReadyForQuery is relayed to the client;
+ // or, if the backend is unreachable, one synthesized ErrorResponse/ReadyForQuery.
+ try {
+ streamObjects(spec, s3, keys, backendOut);
+ writeCopyDone(backendOut);
+ drainToReadyForQuery(backendDecoder, client, onStatusChange);
+ } catch (RuntimeException | IOException e) {
+ LOG.warnv(e, "S3 COPY streaming failed; aborting the open CopyIn");
+ abortOpenCopyIn(client, backend, backendOut, backendDecoder, e, txStatus, onStatusChange);
+ }
+ return true;
+ }
+
+ private static void abortOpenCopyIn(Socket client, Socket backend, OutputStream backendOut,
+ PostgresWireDecoder backendDecoder, Exception cause,
+ char txStatus, java.util.function.IntConsumer onStatusChange) throws IOException {
+ try {
+ writeCopyFail(backendOut, cause.getMessage());
+ drainToReadyForQuery(backendDecoder, client, onStatusChange);
+ } catch (IOException backendGone) {
+ LOG.debugv(backendGone, "backend unreachable while aborting CopyIn; synthesizing client error");
+ closeQuietly(backend);
+ String detail = cause.getMessage() != null ? cause.getMessage() : cause.toString();
+ sendError(client, null, SQLSTATE_INTERNAL, "S3 COPY failed: " + detail, txStatus, onStatusChange);
+ closeQuietly(client);
+ }
+ }
+
+ private static List<String> resolveKeys(CopyStatementParser.S3CopyFrom spec, S3Service s3) {
+ List<String> keys = new ArrayList<>();
+ if (s3.objectExists(spec.bucket(), spec.keyOrPrefix())) {
+ keys.add(spec.keyOrPrefix());
+ return keys;
+ }
+ String continuationToken = null;
+ do {
+ S3Service.ListObjectsResult result = s3.listObjectsWithPrefixes(
+ spec.bucket(), spec.keyOrPrefix(), null, LIST_PAGE_SIZE, continuationToken, null);
+ if (result != null && result.objects() != null) {
+ for (S3Object object : result.objects()) {
+ keys.add(object.getKey());
+ }
+ }
+ continuationToken = (result != null && result.isTruncated()) ? result.nextContinuationToken() : null;
+ } while (continuationToken != null);
+
+ keys.sort(String::compareTo);
+ return keys;
+ }
+
+ private static String fabricateCopy(CopyStatementParser.S3CopyFrom spec) {
+ StringBuilder sql = new StringBuilder("COPY ").append(spec.targetTable());
+ if (spec.columns() != null && !spec.columns().isEmpty()) {
+ sql.append(" (").append(String.join(", ", spec.columns())).append(")");
+ }
+ sql.append(" FROM STDIN WITH (FORMAT ").append(spec.csv() ? "csv" : "text");
+ String delimiter = spec.delimiter() != null ? spec.delimiter() : (spec.csv() ? "," : "|");
+ sql.append(", DELIMITER '").append(quoteLiteral(delimiter)).append("'");
+ if (spec.nullAs() != null) {
+ sql.append(", NULL '").append(quoteLiteral(spec.nullAs())).append("'");
+ }
+ sql.append(")");
+ return sql.toString();
+ }
+
+ /**
+ * Quote a value for a single-quoted SQL string literal. PostgreSQL defaults to
+ * {@code standard_conforming_strings = on}, where a backslash is an ordinary character, so only
+ * the quote itself is doubled: {@code NULL AS '\N'} must reach the backend as {@code '\N'}.
+ */
+ private static String quoteLiteral(String value) {
+ return value.replace("'", "''");
+ }
+
+ private static void streamObjects(CopyStatementParser.S3CopyFrom spec, S3Service s3,
+ List<String> keys, OutputStream backendOut) throws IOException {
+ byte[] buffer = new byte[CHUNK];
+ for (int i = 0; i < keys.size(); i++) {
+ S3Object object = s3.getObject(spec.bucket(), keys.get(i));
+ byte[] data = object != null && object.getData() != null ? object.getData() : new byte[0];
+ InputStream in = new ByteArrayInputStream(data);
+ if (spec.gzip()) {
+ in = new GZIPInputStream(in);
+ }
+ if (i == 0 && spec.headerLines() > 0) {
+ skipLines(in, spec.headerLines());
+ }
+ int read;
+ boolean endsWithNewline = false;
+ boolean hasData = false;
+ while ((read = in.read(buffer)) != -1) {
+ if (read > 0) {
+ hasData = true;
+ endsWithNewline = (buffer[read - 1] == '\n');
+ writeCopyData(backendOut, buffer, read);
+ }
+ }
+ in.close();
+ if (hasData && !endsWithNewline) {
+ writeCopyData(backendOut, new byte[]{'\n'}, 1);
+ }
+ }
+ backendOut.flush();
+ }
+
+ private static void skipLines(InputStream in, int lines) throws IOException {
+ int seen = 0;
+ int b;
+ while (seen < lines && (b = in.read()) != -1) {
+ if (b == '\n') {
+ seen++;
+ }
+ }
+ }
+
+ private static void writeCopyData(OutputStream out, byte[] buf, int len) throws IOException {
+ out.write('d');
+ out.write(intBytes(4 + len));
+ out.write(buf, 0, len);
+ }
+
+ private static void writeCopyDone(OutputStream out) throws IOException {
+ out.write('c');
+ out.write(intBytes(4));
+ out.flush();
+ }
+
+ private static void writeCopyFail(OutputStream out, String reason) throws IOException {
+ byte[] message = (reason == null ? "S3 COPY aborted" : reason).getBytes(StandardCharsets.UTF_8);
+ out.write('f');
+ out.write(intBytes(4 + message.length + 1));
+ out.write(message);
+ out.write(0);
+ out.flush();
+ }
+
+ /** Read the next backend message, relaying asynchronous messages to the client as it goes. */
+ private static PostgresWireDecoder.FrontendMessage nextNonAsync(PostgresWireDecoder decoder, Socket client)
+ throws IOException {
+ OutputStream clientOut = client.getOutputStream();
+ while (true) {
+ PostgresWireDecoder.FrontendMessage message = decoder.nextMessage();
+ if (message == null) {
+ return null;
+ }
+ char type = message.type();
+ if (type == 'N' || type == 'A' || type == 'S') {
+ clientOut.write(message.toPacketBytes());
+ clientOut.flush();
+ continue;
+ }
+ return message;
+ }
+ }
+
+ private static void drainToReadyForQuery(PostgresWireDecoder decoder, Socket client,
+ java.util.function.IntConsumer onStatusChange) throws IOException {
+ OutputStream clientOut = client.getOutputStream();
+ while (true) {
+ PostgresWireDecoder.FrontendMessage message = decoder.nextMessage();
+ if (message == null) {
+ return;
+ }
+ clientOut.write(message.toPacketBytes());
+ clientOut.flush();
+ if (message.type() == 'Z') {
+ if (onStatusChange != null && message.body().length > 0) {
+ onStatusChange.accept(message.body()[0]);
+ }
+ return;
+ }
+ }
+ }
+
+ private static void forward(Socket client, PostgresWireDecoder.FrontendMessage message) throws IOException {
+ client.getOutputStream().write(message.toPacketBytes());
+ client.getOutputStream().flush();
+ }
+
+ private static void sendAccessDenied(Socket client, Socket backend, CopyStatementParser.S3CopyFrom spec,
+ char txStatus, java.util.function.IntConsumer onStatusChange) throws IOException {
+ sendError(client, backend, SQLSTATE_INSUFFICIENT_PRIVILEGE,
+ "S3 access denied for s3://" + spec.bucket() + "/" + spec.keyOrPrefix(), txStatus, onStatusChange);
+ }
+
+ private static void sendError(Socket client, Socket backend, String sqlState, String message,
+ char txStatus, java.util.function.IntConsumer onStatusChange) throws IOException {
+ if (txStatus == 'T' && backend != null && !backend.isClosed()) {
+ if (!failBackendTransaction(backend, onStatusChange)) {
+ closeQuietly(backend);
+ // Backend transaction state could not be confirmed; do not report an unconfirmed
+ // failed transaction ('E') or leave a desynchronized backend in the pump.
+ try {
+ OutputStream out = client.getOutputStream();
+ byte[] body = errorBody(sqlState, message);
+ out.write('E');
+ out.write(intBytes(4 + body.length));
+ out.write(body);
+ out.flush();
+ } catch (IOException e) {
+ LOG.debugv(e, "failed to send ErrorResponse to client before closing socket");
+ } finally {
+ closeQuietly(client);
+ }
+ return;
+ }
+ }
+ OutputStream out = client.getOutputStream();
+ byte[] body = errorBody(sqlState, message);
+ out.write('E');
+ out.write(intBytes(4 + body.length));
+ out.write(body);
+ byte status = (txStatus == 'I' || txStatus == 0) ? (byte) 'I' : (byte) 'E';
+ out.write(new byte[]{'Z', 0, 0, 0, 5, status});
+ out.flush();
+ if (onStatusChange != null) {
+ onStatusChange.accept(status);
+ }
+ }
+
+ private static boolean failBackendTransaction(Socket backend, java.util.function.IntConsumer onStatusChange) {
+ try {
+ OutputStream out = backend.getOutputStream();
+ out.write(PostgresWireDecoder.encodeQuery("(FLOCI_ABORT_TX)"));
+ out.flush();
+ PostgresWireDecoder decoder = new PostgresWireDecoder(backend.getInputStream());
+ while (true) {
+ PostgresWireDecoder.FrontendMessage msg = decoder.nextMessage();
+ if (msg == null) {
+ LOG.warn("backend closed before transaction abort could be synchronized");
+ return false;
+ }
+ if (msg.type() == 'Z') {
+ char status = (msg.body().length > 0) ? (char) msg.body()[0] : 0;
+ if (onStatusChange != null && status != 0) {
+ onStatusChange.accept(status);
+ }
+ if (status != 'E') {
+ LOG.warnv("backend returned unexpected transaction status {0} instead of 'E' after abort query", status);
+ return false;
+ }
+ return true;
+ }
+ }
+ } catch (IOException e) {
+ LOG.warnv(e, "failed to synchronize backend transaction abort state");
+ return false;
+ }
+ }
+
+ private static void closeQuietly(Socket s) {
+ try {
+ if (s != null && !s.isClosed()) {
+ s.close();
+ }
+ } catch (IOException e) {
+ LOG.debugv(e, "failed to close socket: {0}", s);
+ }
+ }
+
+ private static byte[] errorBody(String sqlState, String message) {
+ ByteArrayOutputStream bytes = new ByteArrayOutputStream();
+ writeField(bytes, 'S', "ERROR");
+ writeField(bytes, 'C', sqlState);
+ writeField(bytes, 'M', message);
+ bytes.write(0);
+ return bytes.toByteArray();
+ }
+
+ private static void writeField(ByteArrayOutputStream bytes, char tag, String value) {
+ bytes.write(tag);
+ byte[] raw = value.getBytes(StandardCharsets.UTF_8);
+ bytes.write(raw, 0, raw.length);
+ bytes.write(0);
+ }
+
+ private static byte[] intBytes(int value) {
+ return new byte[]{
+ (byte) (value >>> 24), (byte) (value >>> 16), (byte) (value >>> 8), (byte) value
+ };
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java b/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java
index 426a527972..de6a63d597 100644
--- a/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java
+++ b/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java
@@ -612,6 +612,11 @@ public class S3Service implements Resettable, ResourceProvider {
authorizeGetObject(bucketName, key, null, RequestAuthorization.unsigned());
}
+ /** Authorize an unsigned {@code s3:ListBucket}; see {@link #authorizeAnonymousGetObject}. */
+ public void authorizeAnonymousListBucket(String bucketName) {
+ authorizeListBucket(bucketName, RequestAuthorization.unsigned());
+ }
+
public void authorizeCloudFrontOacGetObject(
String bucketName, String key, String distributionArn) {
authorizeCloudFrontGetObject(
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java
index b2a3b854e3..9b12d1ff44 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java
@@ -27,6 +27,9 @@ class RedshiftInterceptorIntegrationTest {
@Inject
RedshiftService service;
+ @Inject
+ io.github.hectorvent.floci.services.s3.S3Service s3;
+
private String clusterId;
@BeforeAll
@@ -106,4 +109,115 @@ class RedshiftInterceptorIntegrationTest {
}
}
}
+
+ @Test
+ void copyFromS3SingleObjectLoadsRows() throws Exception {
+ clusterId = "it-copy-single";
+ Cluster cluster = service.createCluster(clusterId, "dc2.large", "admin", "Secret123");
+
+ String bucket = "redshift-copy-it";
+ s3.createBucket(bucket, "us-east-1");
+ s3.putObject(bucket, "people/p1.txt",
+ "1|alice\n2|bob\n".getBytes(java.nio.charset.StandardCharsets.UTF_8), "text/plain", java.util.Map.of());
+
+ try (Connection c = waitForConnection(cluster, "admin", "Secret123")) {
+ c.createStatement().execute("CREATE TABLE people (id int, name text)");
+ c.createStatement().execute("COPY people FROM 's3://redshift-copy-it/people/p1.txt'");
+ try (ResultSet rs = c.createStatement().executeQuery("SELECT count(*) FROM people")) {
+ assertTrue(rs.next());
+ assertEquals(2, rs.getInt(1));
+ }
+ }
+ }
+
+ @Test
+ void copyFromS3PrefixConcatenatesObjects() throws Exception {
+ clusterId = "it-copy-prefix";
+ Cluster cluster = service.createCluster(clusterId, "dc2.large", "admin", "Secret123");
+
+ String bucket = "redshift-copy-it-prefix";
+ s3.createBucket(bucket, "us-east-1");
+ s3.putObject(bucket, "d/a", "1|a\n".getBytes(java.nio.charset.StandardCharsets.UTF_8), "text/plain", java.util.Map.of());
+ s3.putObject(bucket, "d/b", "2|b\n".getBytes(java.nio.charset.StandardCharsets.UTF_8), "text/plain", java.util.Map.of());
+
+ try (Connection c = waitForConnection(cluster, "admin", "Secret123")) {
+ c.createStatement().execute("CREATE TABLE t (id int, v text)");
+ c.createStatement().execute("COPY t FROM 's3://redshift-copy-it-prefix/d/'");
+ try (ResultSet rs = c.createStatement().executeQuery("SELECT count(*) FROM t")) {
+ assertTrue(rs.next());
+ assertEquals(2, rs.getInt(1));
+ }
+ }
+ }
+
+ @Test
+ void copyFromS3GzipObjectLoadsRows() throws Exception {
+ clusterId = "it-copy-gzip";
+ Cluster cluster = service.createCluster(clusterId, "dc2.large", "admin", "Secret123");
+
+ String bucket = "redshift-copy-it-gzip";
+ s3.createBucket(bucket, "us-east-1");
+ java.io.ByteArrayOutputStream raw = new java.io.ByteArrayOutputStream();
+ try (java.util.zip.GZIPOutputStream gz = new java.util.zip.GZIPOutputStream(raw)) {
+ gz.write("10|x\n11|y\n".getBytes(java.nio.charset.StandardCharsets.UTF_8));
+ }
+ s3.putObject(bucket, "g/data.gz", raw.toByteArray(), "application/gzip", java.util.Map.of());
+
+ try (Connection c = waitForConnection(cluster, "admin", "Secret123")) {
+ c.createStatement().execute("CREATE TABLE g (id int, v text)");
+ c.createStatement().execute("COPY g FROM 's3://redshift-copy-it-gzip/g/data.gz' GZIP");
+ try (ResultSet rs = c.createStatement().executeQuery("SELECT count(*) FROM g")) {
+ assertTrue(rs.next());
+ assertEquals(2, rs.getInt(1));
+ }
+ }
+ }
+
+ @Test
+ void copyFromMissingObjectSurfacesASqlError() throws Exception {
+ clusterId = "it-copy-missing";
+ Cluster cluster = service.createCluster(clusterId, "dc2.large", "admin", "Secret123");
+
+ String bucket = "redshift-copy-it-missing";
+ s3.createBucket(bucket, "us-east-1");
+
+ try (Connection c = waitForConnection(cluster, "admin", "Secret123")) {
+ c.createStatement().execute("CREATE TABLE t2 (id int)");
+ SQLException ex = org.junit.jupiter.api.Assertions.assertThrows(
+ SQLException.class,
+ () -> c.createStatement().execute("COPY t2 FROM 's3://redshift-copy-it-missing/does/not/exist'"));
+ assertTrue(
+ ex.getMessage().toLowerCase().contains("not found"), ex.getMessage());
+ }
+ }
+
+ @Test
+ void copyFromMissingObjectInTransactionAbortsTransaction() throws Exception {
+ clusterId = "it-copy-tx-abort";
+ Cluster cluster = service.createCluster(clusterId, "dc2.large", "admin", "Secret123");
+
+ String bucket = "redshift-copy-it-tx";
+ s3.createBucket(bucket, "us-east-1");
+
+ try (Connection c = waitForConnection(cluster, "admin", "Secret123")) {
+ c.createStatement().execute("CREATE TABLE tx_test (id int)");
+ c.setAutoCommit(false);
+ Statement st = c.createStatement();
+ st.execute("INSERT INTO tx_test VALUES (1)");
+ org.junit.jupiter.api.Assertions.assertThrows(
+ SQLException.class,
+ () -> st.execute("COPY tx_test FROM 's3://redshift-copy-it-tx/does/not/exist'"));
+ // The backend must now be in the aborted transaction state ('E').
+ // Subsequent statements in this transaction block must fail.
+ org.junit.jupiter.api.Assertions.assertThrows(
+ SQLException.class,
+ () -> st.execute("INSERT INTO tx_test VALUES (2)"));
+ c.rollback();
+ c.setAutoCommit(true);
+ try (ResultSet rs = c.createStatement().executeQuery("SELECT count(*) FROM tx_test")) {
+ assertTrue(rs.next());
+ assertEquals(0, rs.getInt(1));
+ }
+ }
+ }
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.java
new file mode 100644
index 0000000000..e6a8cb3c72
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.java
@@ -0,0 +1,171 @@
+package io.github.hectorvent.floci.services.redshift.proxy;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+class CopyStatementParserTest {
+
+ @Test
+ void parsesMinimalCopyFromKey() {
+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse(
+ "COPY sales FROM 's3://warehouse/data/sales.txt'");
+ assertEquals("sales", c.targetTable());
+ assertEquals(List.of(), c.columns());
+ assertEquals("warehouse", c.bucket());
+ assertEquals("data/sales.txt", c.keyOrPrefix());
+ assertEquals("|", c.delimiter());
+ assertEquals(0, c.headerLines());
+ assertFalse(c.gzip());
+ assertFalse(c.csv());
+ assertNull(c.nullAs());
+ }
+
+ @Test
+ void parsesColumnsOptionsAndPrefix() {
+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse(
+ "COPY public.events (id, ts, note) FROM 's3://bkt/evt/' "
+ + "GZIP DELIMITER ',' IGNOREHEADER 2 NULL AS '\\\\N' FORMAT AS CSV");
+ assertEquals("public.events", c.targetTable());
+ assertEquals(List.of("id", "ts", "note"), c.columns());
+ assertEquals("bkt", c.bucket());
+ assertEquals("evt/", c.keyOrPrefix());
+ assertTrue(c.gzip());
+ assertTrue(c.csv());
+ assertEquals(",", c.delimiter());
+ assertEquals(2, c.headerLines());
+ assertEquals("\\N", c.nullAs());
+ }
+
+ @Test
+ void defaultsDelimiterToCommaForCsvAndTreatsHeaderKeywordAsOneLine() {
+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse(
+ "COPY t FROM 's3://b/k' CSV HEADER");
+ assertEquals(",", c.delimiter());
+ assertEquals(1, c.headerLines());
+ }
+
+ @Test
+ void decodesTabDelimiterToken() {
+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse(
+ "COPY t FROM 's3://b/k' DELIMITER '\\\\t'");
+ assertEquals("\t", c.delimiter());
+ }
+
+ @Test
+ void bucketOnlyPathGivesEmptyPrefix() {
+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse("COPY t FROM 's3://only-bucket'");
+ assertEquals("only-bucket", c.bucket());
+ assertEquals("", c.keyOrPrefix());
+ }
+
+ @Test
+ void stripsLeadingComments() {
+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse(
+ "-- load nightly\n/* batch */ COPY t FROM 's3://b/k'");
+ assertEquals("t", c.targetTable());
+ }
+
+ @Test
+ void rejectsInjectionInTableName() {
+ assertNull(CopyStatementParser.parse("COPY t; DROP TABLE u; FROM 's3://b/k'"));
+ assertNull(CopyStatementParser.parse("COPY (SELECT 1) FROM 's3://b/k'"));
+ }
+
+ @Test
+ void rejectsInjectionInColumnList() {
+ assertNull(CopyStatementParser.parse("COPY t (id, x) FROM STDIN) --) FROM 's3://b/k'"));
+ assertNull(CopyStatementParser.parse("COPY t (id, ts::text) FROM 's3://b/k'"));
+ }
+
+ @Test
+ void returnsNullForNonCopyAndForCopyWithoutS3() {
+ assertNull(CopyStatementParser.parse("SELECT 1"));
+ assertNull(CopyStatementParser.parse("CREATE TABLE t (id int) DISTKEY (id)"));
+ assertNull(CopyStatementParser.parse("COPY t FROM STDIN"));
+ assertNull(CopyStatementParser.parse("COPY t TO 's3://b/k'"));
+ assertNull(CopyStatementParser.parse(null));
+ assertNull(CopyStatementParser.parse(" "));
+ }
+
+ @Test
+ void quotedIdentifiersSurvive() {
+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse(
+ "COPY \"My Schema\".\"Tab\" (\"col one\") FROM 's3://b/k'");
+ assertEquals("\"My Schema\".\"Tab\"", c.targetTable());
+ assertEquals(List.of("\"col one\""), c.columns());
+ }
+
+ @Test
+ void explicitDelimiterOverridesCsvDefault() {
+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse(
+ "COPY t FROM 's3://b/k' CSV DELIMITER '\t'");
+ assertEquals("\t", c.delimiter());
+ assertTrue(c.csv());
+ }
+
+ @Test
+ void returnsNullForUnsupportedClauses() {
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' FORMAT AS PARQUET"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' JSON 'auto'"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' FIXEDWIDTH 'a:1,b:2'"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' GZIP MAXERROR 10"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' IAM_ROLE 'arn:aws:iam::0:role/r'"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' DATEFORMAT 'YYYY-MM-DD'"));
+ }
+
+ @Test
+ void returnsNullWhenAStatementFollowsTheCopy() {
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k'; DROP TABLE staging"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' GZIP; SELECT 1"));
+ }
+
+ @Test
+ void toleratesALoneTrailingSemicolon() {
+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse("COPY t FROM 's3://b/k' GZIP;");
+ assertTrue(c.gzip());
+ }
+
+ @Test
+ void aQuotedKeywordInAnOptionValueDoesNotChangeParsing() {
+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse(
+ "COPY t FROM 's3://b/k' NULL AS 'csv' DELIMITER ';'");
+ assertFalse(c.csv());
+ assertEquals(";", c.delimiter());
+ assertEquals("csv", c.nullAs());
+ }
+
+ @Test
+ void rejectsTyposInOptionKeywords() {
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' DELIMETER ','"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' GZIPP"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' HEADERR"));
+ }
+
+ @Test
+ void rejectsLeftoverOrUnknownTokens() {
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' GZIP EXTRA_TOKEN"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' SOME_UNKNOWN_OPTION"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' CSV FOO"));
+ }
+
+ @Test
+ void rejectsDuplicateOrConflictingOptions() {
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' GZIP GZIP"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' CSV FORMAT CSV"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' DELIMITER ',' DELIMITER '\t'"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' HEADER IGNOREHEADER 2"));
+ }
+
+ @Test
+ void parsesEscapedSingleQuoteInDelimiter() {
+ CopyStatementParser.S3CopyFrom c = CopyStatementParser.parse(
+ "COPY t FROM 's3://b/k' DELIMITER ''''");
+ assertEquals("'", c.delimiter());
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftAuthProxyTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftAuthProxyTest.java
index 804e5d3a08..a0f396f648 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftAuthProxyTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftAuthProxyTest.java
@@ -67,7 +67,8 @@ class RedshiftAuthProxyTest {
int proxyPort = freePort();
proxy = new RedshiftAuthProxy("111111111111:c1", "localhost", fakeBackend.getLocalPort(),
"admin", "Secret123", "dev",
- mock(RdsSigV4Validator.class), realTls(), (user, pw) -> true);
+ mock(RdsSigV4Validator.class), realTls(), (user, pw) -> true,
+ mock(io.github.hectorvent.floci.services.s3.S3Service.class));
proxy.start(proxyPort);
try (Socket client = new Socket("localhost", proxyPort)) {
@@ -120,7 +121,8 @@ class RedshiftAuthProxyTest {
int proxyPort = freePort();
proxy = new RedshiftAuthProxy("111111111111:c1", "localhost", fakeBackend.getLocalPort(),
"admin", "Secret123", "dev",
- mock(RdsSigV4Validator.class), realTls(), (user, pw) -> true);
+ mock(RdsSigV4Validator.class), realTls(), (user, pw) -> true,
+ mock(io.github.hectorvent.floci.services.s3.S3Service.class));
proxy.start(proxyPort);
// Connect, then drop without ever sending a startup packet.
@@ -150,7 +152,8 @@ class RedshiftAuthProxyTest {
proxy = new RedshiftAuthProxy("111111111111:c1", "localhost", fakeBackend.getLocalPort(),
"admin", "Secret123", "dev",
- mock(RdsSigV4Validator.class), realTls(), (user, pw) -> true);
+ mock(RdsSigV4Validator.class), realTls(), (user, pw) -> true,
+ mock(io.github.hectorvent.floci.services.s3.S3Service.class));
proxy.start(proxyPort); // must not throw despite the port being busy at first
assertTrue(portAccepts(proxyPort), "proxy never bound the port after the squatter released it");
@@ -162,7 +165,8 @@ class RedshiftAuthProxyTest {
int proxyPort = freePort();
proxy = new RedshiftAuthProxy("111111111111:c1", "localhost", fakeBackend.getLocalPort(),
"admin", "old", "dev",
- mock(RdsSigV4Validator.class), realTls(), (user, pw) -> true);
+ mock(RdsSigV4Validator.class), realTls(), (user, pw) -> true,
+ mock(io.github.hectorvent.floci.services.s3.S3Service.class));
proxy.start(proxyPort);
proxy.updateMasterPassword("rotated");
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftInterceptingBridgeTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftInterceptingBridgeTest.java
index b11b9fbe29..a7fdf5c879 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftInterceptingBridgeTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftInterceptingBridgeTest.java
@@ -1,13 +1,18 @@
package io.github.hectorvent.floci.services.redshift.proxy;
+import io.github.hectorvent.floci.services.s3.S3Service;
+import io.github.hectorvent.floci.services.s3.model.S3Object;
import org.jboss.logging.Logger;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
import java.io.IOException;
+import java.io.OutputStream;
import java.net.ServerSocket;
import java.net.Socket;
import java.nio.charset.StandardCharsets;
+import java.util.concurrent.atomic.AtomicReference;
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertEquals;
@@ -27,6 +32,8 @@ class RedshiftInterceptingBridgeTest {
private Socket testBackendEnd; // test reads what the bridge forwarded, writes backend replies
private Thread bridgeThread;
+ private S3Service s3Stub;
+
private void startBridge() throws IOException {
clientListener = new ServerSocket(0);
testClientEnd = new Socket("localhost", clientListener.getLocalPort());
@@ -36,7 +43,8 @@ class RedshiftInterceptingBridgeTest {
bridgeBackendEnd = new Socket("localhost", backendListener.getLocalPort());
testBackendEnd = backendListener.accept();
- RedshiftInterceptingBridge bridge = new RedshiftInterceptingBridge(bridgeClientEnd, bridgeBackendEnd);
+ s3Stub = Mockito.mock(S3Service.class);
+ RedshiftInterceptingBridge bridge = new RedshiftInterceptingBridge(bridgeClientEnd, bridgeBackendEnd, s3Stub);
bridgeThread = Thread.ofVirtual().name("bridge-under-test").start(bridge::run);
}
@@ -138,4 +146,70 @@ class RedshiftInterceptingBridgeTest {
assertFalse(bridgeThread.isAlive(), "bridge did not stop after Terminate");
assertEquals(-1, testClientEnd.getInputStream().read(), "bridge left the client socket open");
}
+
+ @Test
+ void interceptsCopyFromS3AndDoesNotForwardTheOriginalQuery() throws Exception {
+ startBridge();
+ Mockito.when(s3Stub.objectExists("wh", "k")).thenReturn(true);
+ Mockito.when(s3Stub.getObject("wh", "k")).thenReturn(
+ new S3Object("wh", "k", "1|a\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));
+
+ AtomicReference<Throwable> backendFailure = new AtomicReference<>();
+ // fake backend: expect a fabricated Query, answer 'G', then after CopyDone answer 'C'+'Z'
+ Thread backend = Thread.ofVirtual().start(() -> {
+ try {
+ PostgresWireDecoder in = new PostgresWireDecoder(testBackendEnd.getInputStream());
+ PostgresWireDecoder.FrontendMessage q = in.nextMessage();
+ assertEquals('Q', q.type());
+ assertTrue(q.getSql().contains("FROM STDIN"), q.getSql());
+ OutputStream out = testBackendEnd.getOutputStream();
+ out.write(new byte[]{'G', 0, 0, 0, 7, 0, 0, 0});
+ out.flush();
+ while (true) {
+ PostgresWireDecoder.FrontendMessage m = in.nextMessage();
+ if (m == null || m.type() == 'c') {
+ break;
+ }
+ }
+ byte[] tag = "COPY 1\0".getBytes(StandardCharsets.US_ASCII);
+ out.write('C');
+ out.write(new byte[]{0, 0, 0, (byte) (4 + tag.length)});
+ out.write(tag);
+ out.write(new byte[]{'Z', 0, 0, 0, 5, 'I'});
+ out.flush();
+ } catch (Throwable t) {
+ backendFailure.compareAndSet(null, t);
+ }
+ });
+
+ testClientEnd.getOutputStream().write(
+ PostgresWireDecoder.encodeQuery("COPY t FROM 's3://wh/k'"));
+ testClientEnd.getOutputStream().flush();
+
+ PostgresWireDecoder.FrontendMessage toClient =
+ new PostgresWireDecoder(testClientEnd.getInputStream()).nextMessage();
+ assertEquals('C', toClient.type());
+ backend.join();
+ if (backendFailure.get() != null) {
+ throw new AssertionError("fake backend failed", backendFailure.get());
+ }
+ }
+
+ @Test
+ void fallsOpenAndForwardsCopyWhenBackendHasAnOutstandingResponse() throws Exception {
+ startBridge();
+ // send a normal query first and never answer it, so outstandingResponses stays > 0
+ testClientEnd.getOutputStream().write(PostgresWireDecoder.encodeQuery("SELECT 1"));
+ testClientEnd.getOutputStream().flush();
+ PostgresWireDecoder backendIn = new PostgresWireDecoder(testBackendEnd.getInputStream());
+ assertEquals('Q', backendIn.nextMessage().type()); // SELECT 1 arrived, unanswered
+
+ testClientEnd.getOutputStream().write(PostgresWireDecoder.encodeQuery("COPY t FROM 's3://wh/k'"));
+ testClientEnd.getOutputStream().flush();
+
+ // within the ~2s deadline the bridge gives up and forwards the COPY verbatim
+ PostgresWireDecoder.FrontendMessage forwarded = backendIn.nextMessage();
+ assertEquals('Q', forwarded.type());
+ assertTrue(forwarded.getSql().contains("s3://wh/k"), forwarded.getSql());
+ }
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManagerTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManagerTest.java
index 0c07c91cb7..d62c6658f8 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManagerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/RedshiftProxyManagerTest.java
@@ -27,7 +27,8 @@ class RedshiftProxyManagerTest {
private RedshiftProxyManager newManager() {
return new RedshiftProxyManager(
- mock(RdsSigV4Validator.class), mock(RdsProxyTlsCertificates.class));
+ mock(RdsSigV4Validator.class), mock(RdsProxyTlsCertificates.class),
+ mock(io.github.hectorvent.floci.services.s3.S3Service.class));
}
private static void start(RedshiftProxyManager manager, String key, int proxyPort) {
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java
new file mode 100644
index 0000000000..28b9a8aeaa
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java
@@ -0,0 +1,549 @@
+package io.github.hectorvent.floci.services.redshift.proxy;
+
+import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.services.s3.S3Service;
+import io.github.hectorvent.floci.services.s3.model.S3Object;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.ServerSocket;
+import java.net.Socket;
+import java.net.SocketTimeoutException;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.zip.GZIPOutputStream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyInt;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.ArgumentMatchers.isNull;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+class S3CopySimulatorTest {
+
+ private ServerSocket listener;
+ private Socket simClient; // simulator's "client" end
+ private Socket simBackend; // simulator's "backend" end
+ private Socket testClient; // test reads ErrorResponse / CommandComplete here
+ private Socket testBackend; // test plays PostgreSQL here
+ private S3Service s3;
+
+ private final AtomicReference<Throwable> backendFailure = new AtomicReference<>();
+
+ @BeforeEach
+ void setUp() throws IOException {
+ listener = new ServerSocket(0);
+ simClient = new Socket("localhost", listener.getLocalPort());
+ testClient = listener.accept();
+
+ ServerSocket backendListener = new ServerSocket(0);
+ simBackend = new Socket("localhost", backendListener.getLocalPort());
+ testBackend = backendListener.accept();
+ backendListener.close();
+
+ s3 = mock(S3Service.class);
+ }
+
+ @AfterEach
+ void tearDown() throws IOException {
+ for (Socket s : new Socket[]{simClient, simBackend, testClient, testBackend}) {
+ if (s != null && !s.isClosed()) {
+ s.close();
+ }
+ }
+ if (listener != null && !listener.isClosed()) {
+ listener.close();
+ }
+ }
+
+ @FunctionalInterface
+ private interface BackendScript {
+ void run() throws IOException;
+ }
+
+ /** Run a fake-backend script on a virtual thread, capturing any failure for the test thread. */
+ private Thread backendThread(BackendScript script) {
+ return Thread.ofVirtual().start(() -> {
+ try {
+ script.run();
+ } catch (Throwable t) {
+ backendFailure.compareAndSet(null, t);
+ }
+ });
+ }
+
+ private void joinBackend(Thread t) throws InterruptedException {
+ t.join();
+ Throwable failure = backendFailure.get();
+ if (failure != null) {
+ throw new AssertionError("fake backend failed: " + failure, failure);
+ }
+ }
+
+ private static byte[] gzip(String s) throws IOException {
+ ByteArrayOutputStream bytes = new ByteArrayOutputStream();
+ try (GZIPOutputStream gz = new GZIPOutputStream(bytes)) {
+ gz.write(s.getBytes(StandardCharsets.US_ASCII));
+ }
+ return bytes.toByteArray();
+ }
+
+ private static byte[] intBytes(int v) {
+ return new byte[]{(byte) (v >>> 24), (byte) (v >>> 16), (byte) (v >>> 8), (byte) v};
+ }
+
+ private PostgresWireDecoder.FrontendMessage nextFromSimulatorToBackend() throws IOException {
+ return new PostgresWireDecoder(testBackend.getInputStream()).nextMessage();
+ }
+
+ /** Play a minimal happy-path PostgreSQL backend: 'G' then, after CopyDone, 'C' and 'Z'. */
+ private void playHappyBackend(ByteArrayOutputStream capturedCopyData) throws IOException {
+ OutputStream out = testBackend.getOutputStream();
+ PostgresWireDecoder in = new PostgresWireDecoder(testBackend.getInputStream());
+ PostgresWireDecoder.FrontendMessage query = in.nextMessage();
+ assertEquals('Q', query.type());
+ // CopyInResponse 'G': body = format(1) + columnCount(2) = 3 bytes, all zero => length 7
+ out.write(new byte[]{'G', 0, 0, 0, 7, 0, 0, 0});
+ out.flush();
+ while (true) {
+ PostgresWireDecoder.FrontendMessage m = in.nextMessage();
+ if (m == null || m.type() == 'c') {
+ break;
+ }
+ if (m.type() == 'd') {
+ capturedCopyData.write(m.body());
+ }
+ }
+ byte[] tag = "COPY 1\0".getBytes(StandardCharsets.US_ASCII);
+ out.write('C');
+ out.write(intBytes(4 + tag.length));
+ out.write(tag);
+ out.write(new byte[]{'Z', 0, 0, 0, 5, 'I'});
+ out.flush();
+ }
+
+ private String captureFabricatedThenComplete() throws IOException {
+ OutputStream out = testBackend.getOutputStream();
+ PostgresWireDecoder in = new PostgresWireDecoder(testBackend.getInputStream());
+ PostgresWireDecoder.FrontendMessage q = in.nextMessage();
+ assertEquals('Q', q.type());
+ out.write(new byte[]{'G', 0, 0, 0, 7, 0, 0, 0});
+ out.flush();
+ while (true) {
+ PostgresWireDecoder.FrontendMessage m = in.nextMessage();
+ if (m == null || m.type() == 'c') {
+ break;
+ }
+ }
+ out.write(new byte[]{'Z', 0, 0, 0, 5, 'I'});
+ out.flush();
+ return q.getSql();
+ }
+
+ @Test
+ void loadsSingleKeyAsPipeTextAndForwardsCommandComplete() throws Exception {
+ when(s3.objectExists("wh", "d/a.txt")).thenReturn(true);
+ when(s3.getObject("wh", "d/a.txt")).thenReturn(
+ new S3Object("wh", "d/a.txt", "1|alice\n2|bob\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));
+
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "people", List.of(), "wh", "d/a.txt", "|", 0, false, false, null);
+
+ ByteArrayOutputStream copyData = new ByteArrayOutputStream();
+ Thread backend = backendThread(() -> playHappyBackend(copyData));
+
+ boolean handled = S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');
+ joinBackend(backend);
+
+ assertTrue(handled);
+ assertEquals("1|alice\n2|bob\n", copyData.toString(StandardCharsets.US_ASCII));
+
+ PostgresWireDecoder.FrontendMessage toClient = new PostgresWireDecoder(testClient.getInputStream()).nextMessage();
+ assertEquals('C', toClient.type());
+ }
+
+ @Test
+ void fabricatedQueryUsesTextFormatAndPipeDelimiterByDefault() throws Exception {
+ when(s3.objectExists("wh", "k")).thenReturn(true);
+ when(s3.getObject("wh", "k")).thenReturn(
+ new S3Object("wh", "k", "x\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "t", List.of("a", "b"), "wh", "k", "|", 0, false, false, null);
+
+ AtomicReference<String> fabricated = new AtomicReference<>();
+ Thread backend = backendThread(() -> fabricated.set(captureFabricatedThenComplete()));
+
+ S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');
+ joinBackend(backend);
+
+ String sql = fabricated.get();
+ assertTrue(sql.contains("COPY t (a, b) FROM STDIN"), sql);
+ assertTrue(sql.toUpperCase().contains("FORMAT TEXT"), sql);
+ assertTrue(sql.contains("DELIMITER '|'"), sql);
+ assertFalse(sql.toUpperCase().contains("HEADER"), sql);
+ }
+
+ @Test
+ void fabricatedQueryUsesCsvFormatAndCommaDelimiterByDefault() throws Exception {
+ when(s3.objectExists("wh", "k")).thenReturn(true);
+ when(s3.getObject("wh", "k")).thenReturn(
+ new S3Object("wh", "k", "x\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "t", List.of("a", "b"), "wh", "k", null, 0, false, true, null);
+
+ AtomicReference<String> fabricated = new AtomicReference<>();
+ Thread backend = backendThread(() -> fabricated.set(captureFabricatedThenComplete()));
+
+ S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');
+ joinBackend(backend);
+
+ String sql = fabricated.get();
+ assertTrue(sql.contains("COPY t (a, b) FROM STDIN"), sql);
+ assertTrue(sql.toUpperCase().contains("FORMAT CSV"), sql);
+ assertTrue(sql.contains("DELIMITER ','"), sql);
+ }
+
+ @Test
+ void nullAsBackslashNIsPassedThroughLiterally() throws Exception {
+ when(s3.objectExists("wh", "k")).thenReturn(true);
+ when(s3.getObject("wh", "k")).thenReturn(
+ new S3Object("wh", "k", "x\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "t", List.of(), "wh", "k", "|", 0, false, false, "\\N");
+
+ AtomicReference<String> fabricated = new AtomicReference<>();
+ Thread backend = backendThread(() -> fabricated.set(captureFabricatedThenComplete()));
+
+ S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');
+ joinBackend(backend);
+
+ String sql = fabricated.get();
+ assertTrue(sql.contains("NULL '\\N'"), sql);
+ assertFalse(sql.contains("NULL '\\\\N'"), sql);
+ }
+
+ @Test
+ void concatenatesPrefixObjectsInKeyOrderAndSkipsHeaderOnFirstOnly() throws Exception {
+ when(s3.objectExists("wh", "p/")).thenReturn(false);
+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("p/"), isNull(), anyInt(), any(), any())).thenReturn(
+ new S3Service.ListObjectsResult(List.of(
+ new S3Object("wh", "p/1", "h1|h2\n1|a\n".getBytes(StandardCharsets.US_ASCII), "text/plain"),
+ new S3Object("wh", "p/2", "2|b\n".getBytes(StandardCharsets.US_ASCII), "text/plain")),
+ List.of(), false, null));
+ when(s3.getObject("wh", "p/1")).thenReturn(
+ new S3Object("wh", "p/1", "h1|h2\n1|a\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));
+ when(s3.getObject("wh", "p/2")).thenReturn(
+ new S3Object("wh", "p/2", "2|b\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));
+
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "t", List.of(), "wh", "p/", "|", 1, false, false, null);
+
+ ByteArrayOutputStream copyData = new ByteArrayOutputStream();
+ Thread backend = backendThread(() -> playHappyBackend(copyData));
+ S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');
+ joinBackend(backend);
+
+ assertEquals("1|a\n2|b\n", copyData.toString(StandardCharsets.US_ASCII));
+ }
+
+ @Test
+ void decompressesGzipObjects() throws Exception {
+ when(s3.objectExists("wh", "k.gz")).thenReturn(true);
+ when(s3.getObject("wh", "k.gz")).thenReturn(
+ new S3Object("wh", "k.gz", gzip("1|a\n2|b\n"), "application/gzip"));
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "t", List.of(), "wh", "k.gz", "|", 0, true, false, null);
+
+ ByteArrayOutputStream copyData = new ByteArrayOutputStream();
+ Thread backend = backendThread(() -> playHappyBackend(copyData));
+ S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');
+ joinBackend(backend);
+
+ assertEquals("1|a\n2|b\n", copyData.toString(StandardCharsets.US_ASCII));
+ }
+
+ @Test
+ void deniedBucketSendsInsufficientPrivilegeErrorResponse() throws Exception {
+ doThrow(new AwsException("AccessDenied", "no", 403))
+ .when(s3).authorizeAnonymousListBucket("wh");
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "t", List.of(), "wh", "k", "|", 0, false, false, null);
+
+ boolean handled = S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');
+ assertTrue(handled);
+
+ PostgresWireDecoder in = new PostgresWireDecoder(testClient.getInputStream());
+ PostgresWireDecoder.FrontendMessage err = in.nextMessage();
+ assertEquals('E', err.type());
+ assertTrue(new String(err.body(), StandardCharsets.US_ASCII).contains("42501"));
+ assertEquals('Z', in.nextMessage().type());
+ }
+
+ @Test
+ void missingKeySendsNotFoundErrorResponse() throws Exception {
+ when(s3.objectExists("wh", "missing")).thenReturn(false);
+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("missing"), isNull(), anyInt(), any(), any())).thenReturn(
+ new S3Service.ListObjectsResult(List.of(), List.of(), false, null));
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "t", List.of(), "wh", "missing", "|", 0, false, false, null);
+
+ boolean handled = S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');
+ assertTrue(handled);
+
+ PostgresWireDecoder in = new PostgresWireDecoder(testClient.getInputStream());
+ PostgresWireDecoder.FrontendMessage err = in.nextMessage();
+ assertEquals('E', err.type());
+ assertTrue(new String(err.body(), StandardCharsets.US_ASCII).contains("not found"));
+ assertEquals('Z', in.nextMessage().type());
+ }
+
+ @Test
+ void backendErrorInsteadOfCopyInIsForwardedToClient() throws Exception {
+ when(s3.objectExists("wh", "k")).thenReturn(true);
+ when(s3.getObject("wh", "k")).thenReturn(
+ new S3Object("wh", "k", "1\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "nosuch", List.of(), "wh", "k", "|", 0, false, false, null);
+
+ Thread backend = backendThread(() -> {
+ new PostgresWireDecoder(testBackend.getInputStream()).nextMessage();
+ byte[] msg = "SERROR\0C42P01\0Mrelation \"nosuch\" does not exist\0\0".getBytes(StandardCharsets.US_ASCII);
+ OutputStream out = testBackend.getOutputStream();
+ out.write('E');
+ out.write(intBytes(4 + msg.length));
+ out.write(msg);
+ out.write(new byte[]{'Z', 0, 0, 0, 5, 'I'});
+ out.flush();
+ });
+
+ boolean handled = S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');
+ joinBackend(backend);
+ assertTrue(handled);
+
+ PostgresWireDecoder in = new PostgresWireDecoder(testClient.getInputStream());
+ assertEquals('E', in.nextMessage().type());
+ assertEquals('Z', in.nextMessage().type());
+ }
+
+ @Test
+ void midCopyFailureSendsExactlyOneErrorAndReadyForQuery() throws Exception {
+ when(s3.objectExists("wh", "k")).thenReturn(true);
+ when(s3.getObject("wh", "k")).thenThrow(new AwsException("InternalError", "boom", 500));
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "t", List.of(), "wh", "k", "|", 0, false, false, null);
+
+ Thread backend = backendThread(() -> {
+ OutputStream out = testBackend.getOutputStream();
+ PostgresWireDecoder in = new PostgresWireDecoder(testBackend.getInputStream());
+ assertEquals('Q', in.nextMessage().type());
+ out.write(new byte[]{'G', 0, 0, 0, 7, 0, 0, 0});
+ out.flush();
+ // wait for the simulator's CopyFail 'f'
+ PostgresWireDecoder.FrontendMessage m;
+ while ((m = in.nextMessage()) != null && m.type() != 'f') {
+ // discard any CopyData already buffered
+ }
+ assertEquals('f', m == null ? 0 : m.type());
+ byte[] err = "SERROR\0C57014\0MCOPY aborted\0\0".getBytes(StandardCharsets.US_ASCII);
+ out.write('E');
+ out.write(intBytes(4 + err.length));
+ out.write(err);
+ out.write(new byte[]{'Z', 0, 0, 0, 5, 'I'});
+ out.flush();
+ });
+
+ S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');
+ joinBackend(backend);
+
+ PostgresWireDecoder in = new PostgresWireDecoder(testClient.getInputStream());
+ assertEquals('E', in.nextMessage().type());
+ assertEquals('Z', in.nextMessage().type());
+ testClient.setSoTimeout(400);
+ assertThrows(SocketTimeoutException.class, () -> testClient.getInputStream().read(),
+ "no second response should follow");
+ }
+
+ @Test
+ void backendEofBeforeCopyInResponseSendsErrorToClient() throws Exception {
+ when(s3.objectExists("wh", "k")).thenReturn(true);
+ when(s3.getObject("wh", "k")).thenReturn(
+ new S3Object("wh", "k", "1\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "t", List.of(), "wh", "k", "|", 0, false, false, null);
+
+ Thread backend = backendThread(() -> {
+ new PostgresWireDecoder(testBackend.getInputStream()).nextMessage();
+ testBackend.close();
+ });
+
+ boolean handled = S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');
+ joinBackend(backend);
+ assertTrue(handled);
+
+ PostgresWireDecoder in = new PostgresWireDecoder(testClient.getInputStream());
+ assertEquals('E', in.nextMessage().type());
+ assertEquals('Z', in.nextMessage().type());
+ }
+
+ @Test
+ void synthesizedErrorReportsFailedTransactionStatusWhenInABlock() throws Exception {
+ when(s3.objectExists("wh", "missing")).thenReturn(false);
+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("missing"), isNull(), anyInt(), any(), any())).thenReturn(
+ new S3Service.ListObjectsResult(List.of(), List.of(), false, null));
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "t", List.of(), "wh", "missing", "|", 0, false, false, null);
+
+ Thread backend = backendThread(() -> {
+ PostgresWireDecoder in = new PostgresWireDecoder(testBackend.getInputStream());
+ PostgresWireDecoder.FrontendMessage q = in.nextMessage();
+ assertEquals('Q', q.type());
+ assertTrue(q.getSql().contains("FLOCI_ABORT_TX"));
+ OutputStream out = testBackend.getOutputStream();
+ byte[] err = "SERROR\0C42601\0Msyntax error\0\0".getBytes(StandardCharsets.US_ASCII);
+ out.write('E');
+ out.write(intBytes(4 + err.length));
+ out.write(err);
+ out.write(new byte[]{'Z', 0, 0, 0, 5, 'E'});
+ out.flush();
+ });
+
+ S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'T');
+ joinBackend(backend);
+
+ PostgresWireDecoder in = new PostgresWireDecoder(testClient.getInputStream());
+ assertEquals('E', in.nextMessage().type());
+ PostgresWireDecoder.FrontendMessage ready = in.nextMessage();
+ assertEquals('Z', ready.type());
+ assertEquals('E', (char) ready.body()[0]);
+ }
+
+ @Test
+ void paginatesPrefixListingAcrossMultiplePages() throws Exception {
+ when(s3.objectExists("wh", "multi/")).thenReturn(false);
+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("multi/"), isNull(), anyInt(), isNull(), isNull())).thenReturn(
+ new S3Service.ListObjectsResult(List.of(
+ new S3Object("wh", "multi/1", "1|a\n".getBytes(StandardCharsets.US_ASCII), "text/plain")),
+ List.of(), true, "tok-1"));
+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("multi/"), isNull(), anyInt(), eq("tok-1"), isNull())).thenReturn(
+ new S3Service.ListObjectsResult(List.of(
+ new S3Object("wh", "multi/2", "2|b\n".getBytes(StandardCharsets.US_ASCII), "text/plain")),
+ List.of(), false, null));
+
+ when(s3.getObject("wh", "multi/1")).thenReturn(
+ new S3Object("wh", "multi/1", "1|a\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));
+ when(s3.getObject("wh", "multi/2")).thenReturn(
+ new S3Object("wh", "multi/2", "2|b\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));
+
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "t", List.of(), "wh", "multi/", "|", 0, false, false, null);
+
+ ByteArrayOutputStream copyData = new ByteArrayOutputStream();
+ Thread backend = backendThread(() -> playHappyBackend(copyData));
+ S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');
+ joinBackend(backend);
+
+ assertEquals("1|a\n2|b\n", copyData.toString(StandardCharsets.US_ASCII));
+ }
+
+ @Test
+ void backendTimeoutAwaitingCopyInResponseClosesBackend() throws Exception {
+ when(s3.objectExists("wh", "k")).thenReturn(true);
+ when(s3.getObject("wh", "k")).thenReturn(
+ new S3Object("wh", "k", "1\n".getBytes(StandardCharsets.US_ASCII), "text/plain"));
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "t", List.of(), "wh", "k", "|", 0, false, false, null);
+
+ simBackend.setSoTimeout(100);
+
+ Thread backend = backendThread(() -> {
+ new PostgresWireDecoder(testBackend.getInputStream()).nextMessage();
+ try {
+ Thread.sleep(300);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ });
+
+ boolean handled = S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'I');
+ joinBackend(backend);
+ assertTrue(handled);
+
+ assertTrue(simBackend.isClosed(), "simBackend must be closed upon timeout");
+
+ PostgresWireDecoder in = new PostgresWireDecoder(testClient.getInputStream());
+ assertEquals('E', in.nextMessage().type());
+ assertEquals('Z', in.nextMessage().type());
+ }
+
+ @Test
+ void failedBackendTransactionAbortClosesBackendAndClientWithoutReportingFailedTransaction() throws Exception {
+ when(s3.objectExists("wh", "missing")).thenReturn(false);
+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("missing"), isNull(), anyInt(), any(), any())).thenReturn(
+ new S3Service.ListObjectsResult(List.of(), List.of(), false, null));
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "t", List.of(), "wh", "missing", "|", 0, false, false, null);
+
+ // Backend closes socket when abort query arrives, simulating dropped connection
+ Thread backend = backendThread(() -> {
+ PostgresWireDecoder in = new PostgresWireDecoder(testBackend.getInputStream());
+ PostgresWireDecoder.FrontendMessage q = in.nextMessage();
+ assertEquals('Q', q.type());
+ assertTrue(q.getSql().contains("FLOCI_ABORT_TX"));
+ testBackend.close();
+ });
+
+ S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'T');
+ joinBackend(backend);
+
+ assertTrue(simBackend.isClosed(), "simBackend must be closed upon synchronization failure");
+ assertTrue(simClient.isClosed(), "simClient must be closed upon synchronization failure");
+
+ PostgresWireDecoder in = new PostgresWireDecoder(testClient.getInputStream());
+ PostgresWireDecoder.FrontendMessage err = in.nextMessage();
+ assertEquals('E', err.type());
+ assertNull(in.nextMessage(), "Client must not receive an unconfirmed ReadyForQuery");
+ }
+
+ @Test
+ void backendReturnsNonETransactionStatusAfterAbortClosesBackendAndClient() throws Exception {
+ when(s3.objectExists("wh", "missing")).thenReturn(false);
+ when(s3.listObjectsWithPrefixes(eq("wh"), eq("missing"), isNull(), anyInt(), any(), any())).thenReturn(
+ new S3Service.ListObjectsResult(List.of(), List.of(), false, null));
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "t", List.of(), "wh", "missing", "|", 0, false, false, null);
+
+ // Fake backend returns ReadyForQuery with 'T' (e.g. abort query unexpectedly succeeded)
+ Thread backend = backendThread(() -> {
+ PostgresWireDecoder in = new PostgresWireDecoder(testBackend.getInputStream());
+ PostgresWireDecoder.FrontendMessage q = in.nextMessage();
+ assertEquals('Q', q.type());
+ assertTrue(q.getSql().contains("FLOCI_ABORT_TX"));
+ OutputStream out = testBackend.getOutputStream();
+ out.write(new byte[]{'Z', 0, 0, 0, 5, 'T'});
+ out.flush();
+ });
+
+ S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, 'T');
+ joinBackend(backend);
+
+ assertTrue(simBackend.isClosed(), "simBackend must be closed when abort returns non-E status");
+ assertTrue(simClient.isClosed(), "simClient must be closed when abort returns non-E status");
+
+ PostgresWireDecoder in = new PostgresWireDecoder(testClient.getInputStream());
+ PostgresWireDecoder.FrontendMessage err = in.nextMessage();
+ assertEquals('E', err.type());
+ assertNull(in.nextMessage(), "Client must not receive an unconfirmed ReadyForQuery");
+ }
+}