Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(redshift): support S3 COPY JSON and manifest, and DynamoDB zero-ETL backfill (#3889)

Vũ Thế Huy9379fa117974
diff --git a/docs/services/redshift.md b/docs/services/redshift.md
index 6dac6438dd..d996043b4f 100644
--- a/docs/services/redshift.md
+++ b/docs/services/redshift.md
@@ -57,8 +57,16 @@ and DynamoDB keys and images as JSON text. Record event ids are unique, so retri
The integration consumer has its own stream checkpoint and does not share Lambda event source
mapping state. Deleting an integration stops its consumer while retaining the landing table.
-Serverless Redshift targets, backfill, schema inference, and relational projection are not supported
-in this first implementation.
+
+Items already present in the source table when `CreateIntegration` runs are backfilled into the
+landing table with a paginated `Scan`, one page per poll tick, and the scan resumes after a Floci
+restart. While the backfill runs, `DescribeIntegrations` reports `Status` as `syncing` (as a Floci
+approximation to indicate that initial backfill is in progress); it becomes `active` once the scan is
+exhausted. The landing table is an append-only log, so an item changed while its table is still being
+backfilled can appear twice: once from the scan and once from the stream.
+
+Serverless Redshift targets, schema inference, and relational projection are not supported in this
+first implementation.
## CloudFormation
@@ -209,9 +217,25 @@ order) through its own S3 service and streams the rows into the backing PostgreS
`COPY ... FROM STDIN`.
- Supported options: `DELIMITER`, `FORMAT CSV` (or a bare `CSV`), `GZIP`, `IGNOREHEADER <n>` and
- `HEADER`, `NULL AS`, and an explicit column list.
+ `HEADER`, `NULL AS`, `FORMAT AS JSON 'auto'`, `FORMAT AS JSON 'auto ignorecase'` (or bare `JSON 'auto'`),
+ `MANIFEST`, and an explicit column list.
- The default framing is pipe-delimited text, matching Redshift. `FORMAT CSV` switches to CSV with
a comma default delimiter.
+- `FORMAT AS JSON 'auto'` loads JSON objects, mapping JSON keys to table columns with exact case
+ sensitivity. `FORMAT AS JSON 'auto ignorecase'` performs case-insensitive key matching. Input
+ objects can be separated by any whitespace, including multiline pretty-printed JSON. Non-object root
+ values abort the load. When columns are not specified in the COPY statement, table column names and
+ order are automatically discovered from the database catalog, but only over the **Simple Query
+ protocol**. Over Extended Query (the default for a JDBC `PreparedStatement`, and for a plain
+ `Statement` under recent pgjdbc versions) the column list is fixed by the time `Parse` is sent,
+ before any backend round trip is possible, so catalog discovery cannot run there: omitting the column
+ list fails the COPY with a clear error instead of silently guessing. Specify the column list explicitly
+ for Extended Query, or connect with `preferQueryMode=simple` to use discovery. Nested objects and
+ arrays are serialized as JSON strings.
+- `MANIFEST` resolves file keys from a JSON manifest file (`{"entries": [{"url": "s3://...", "mandatory": boolean}]}`),
+ compatible with output from `UNLOAD ... MANIFEST`. Missing files marked `mandatory: true` abort the load.
+ When `mandatory` is omitted, it defaults to `false`. All entries in the manifest must reside in the same S3
+ bucket as the manifest file itself; cross-bucket manifest entries are rejected as an intentional deviation.
- `IGNOREHEADER` and `HEADER` skip lines from the first resolved object only.
- `GZIP` is the only input compression recognized; `BZIP2`, `LZOP` and `ZSTD` are not.
- `IAM_ROLE '<role-arn>'` is supported. The role must be associated with the cluster, exist in
@@ -219,7 +243,7 @@ order) through its own S3 service and streams the rows into the backing PostgreS
`FLOCI_SERVICES_S3_ENFORCE_AUTH` off, S3 policy checks are skipped. With it on, the role's
identity policy must allow the required S3 actions, and any bucket policy must not deny the
request. `IAM_ROLE default` is not supported.
-- Any other clause (`FIXEDWIDTH`, `JSON`, `PARQUET`, `AVRO`, `ORC`, `MANIFEST`, `MAXERROR`,
+- Any other clause (`FIXEDWIDTH`, `PARQUET`, `AVRO`, `ORC`, `MAXERROR`,
`DATEFORMAT`, `TIMEFORMAT`, `REGION`, `ENCODING`, `ESCAPE`, `REMOVEQUOTES`, `BLANKSASNULL`,
`EMPTYASNULL`, `TRUNCATECOLUMNS`, `ACCEPTINVCHARS`, `CREDENTIALS`, and so on) is not
recognized: the statement is forwarded unchanged and PostgreSQL returns its own error.
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftDynamoDbZeroEtlConsumer.java b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftDynamoDbZeroEtlConsumer.java
index 98d9bbd0ca..fe80a5f380 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftDynamoDbZeroEtlConsumer.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftDynamoDbZeroEtlConsumer.java
@@ -1,9 +1,18 @@
package io.github.hectorvent.floci.services.redshift;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.core.common.AwsArnUtils;
import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.core.common.RequestScopes;
+import io.github.hectorvent.floci.services.dynamodb.DynamoDbService;
import io.github.hectorvent.floci.services.dynamodb.DynamoDbStreamService;
import io.github.hectorvent.floci.services.dynamodb.model.DynamoDbStreamRecord;
+import io.github.hectorvent.floci.services.dynamodb.model.KeySchemaElement;
+import io.github.hectorvent.floci.services.dynamodb.model.TableDefinition;
import io.github.hectorvent.floci.services.redshift.model.Integration;
import io.github.hectorvent.floci.services.redshiftdata.RedshiftZeroEtlWriter;
import io.quarkus.runtime.StartupEvent;
@@ -14,6 +23,10 @@ import jakarta.enterprise.event.Observes;
import jakarta.inject.Inject;
import org.jboss.logging.Logger;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
@@ -26,8 +39,10 @@ public class RedshiftDynamoDbZeroEtlConsumer {
private final Vertx vertx;
private final DynamoDbStreamService streamService;
+ private final DynamoDbService dynamoDbService;
private final RedshiftService redshiftService;
private final RedshiftZeroEtlWriter writer;
+ private final ObjectMapper objectMapper;
private final long pollIntervalMs;
private final ConcurrentHashMap<String, Long> timerIds = new ConcurrentHashMap<>();
private final ConcurrentHashMap<String, Boolean> activePolls = new ConcurrentHashMap<>();
@@ -40,27 +55,35 @@ public class RedshiftDynamoDbZeroEtlConsumer {
@Inject
public RedshiftDynamoDbZeroEtlConsumer(Vertx vertx,
DynamoDbStreamService streamService,
+ DynamoDbService dynamoDbService,
RedshiftService redshiftService,
RedshiftZeroEtlWriter writer,
+ ObjectMapper objectMapper,
EmulatorConfig config) {
- this(vertx, streamService, redshiftService, writer, config.services().redshift().pollIntervalMs());
+ this(vertx, streamService, dynamoDbService, redshiftService, writer, objectMapper,
+ config.services().redshift().pollIntervalMs());
}
RedshiftDynamoDbZeroEtlConsumer(DynamoDbStreamService streamService,
+ DynamoDbService dynamoDbService,
RedshiftService redshiftService,
RedshiftZeroEtlWriter writer) {
- this(null, streamService, redshiftService, writer, 1000);
+ this(null, streamService, dynamoDbService, redshiftService, writer, new ObjectMapper(), 1000);
}
private RedshiftDynamoDbZeroEtlConsumer(Vertx vertx,
DynamoDbStreamService streamService,
+ DynamoDbService dynamoDbService,
RedshiftService redshiftService,
RedshiftZeroEtlWriter writer,
+ ObjectMapper objectMapper,
long pollIntervalMs) {
this.vertx = vertx;
this.streamService = streamService;
+ this.dynamoDbService = dynamoDbService;
this.redshiftService = redshiftService;
this.writer = writer;
+ this.objectMapper = objectMapper;
this.pollIntervalMs = pollIntervalMs;
}
@@ -95,6 +118,10 @@ public class RedshiftDynamoDbZeroEtlConsumer {
void pollOnce(Integration integration) {
writer.createLandingTable(integration.getAccountId(), integration.getTargetClusterIdentifier(),
integration.getLandingTableName());
+ if (!integration.isBackfillCompleted()) {
+ pollBackfillPage(integration);
+ return;
+ }
String iteratorType = integration.getCheckpointSequenceNumber() == null
? "TRIM_HORIZON" : "AFTER_SEQUENCE_NUMBER";
String iterator = streamService.getShardIterator(integration.getSourceStreamArn(),
@@ -110,6 +137,88 @@ public class RedshiftDynamoDbZeroEtlConsumer {
integration.setCheckpointSequenceNumber(sequence);
}
+ private void pollBackfillPage(Integration integration) {
+ RequestScopes.runAs(integration.getAccountId(), () -> {
+ String tableName = extractTableName(integration.getSourceStreamArn());
+ String region = AwsArnUtils.parse(integration.getSourceStreamArn()).region();
+ TableDefinition table = dynamoDbService.describeTable(tableName, region);
+ JsonNode exclusiveStartKey = parseBackfillKey(integration.getBackfillLastEvaluatedKey());
+
+ DynamoDbService.ScanResult result = dynamoDbService.scan(tableName, null, null, null, null,
+ BATCH_SIZE, exclusiveStartKey, region);
+ if (!result.items().isEmpty()) {
+ List<DynamoDbStreamRecord> records = result.items().stream()
+ .map(item -> toBackfillRecord(integration, item, table))
+ .toList();
+ writer.writeBatch(integration.getAccountId(), integration.getTargetClusterIdentifier(),
+ integration.getLandingTableName(), records);
+ }
+
+ boolean completed = result.lastEvaluatedKey() == null;
+ String nextKey = completed ? null : writeAsString(result.lastEvaluatedKey());
+ redshiftService.updateIntegrationBackfillProgress(integration.getAccountId(), integration.getIntegrationArn(),
+ nextKey, completed);
+ integration.setBackfillLastEvaluatedKey(nextKey);
+ integration.setBackfillCompleted(completed);
+ });
+ }
+
+ private DynamoDbStreamRecord toBackfillRecord(Integration integration, JsonNode item, TableDefinition table) {
+ ObjectNode keys = objectMapper.createObjectNode();
+ for (KeySchemaElement keySchemaElement : table.getKeySchema()) {
+ String attributeName = keySchemaElement.getAttributeName();
+ if (item.has(attributeName)) {
+ keys.set(attributeName, item.get(attributeName));
+ }
+ }
+ DynamoDbStreamRecord record = new DynamoDbStreamRecord();
+ record.setEventId("backfill#" + integration.getIntegrationArn() + "#" + sha256Hex(keys.toString()));
+ record.setEventName("INSERT");
+ record.setSequenceNumber("backfill");
+ record.setKeys(keys);
+ record.setNewImage(item);
+ return record;
+ }
+
+ private static String extractTableName(String sourceStreamArn) {
+ String resource = AwsArnUtils.parse(sourceStreamArn).resource();
+ return resource.substring("table/".length(), resource.indexOf("/stream/"));
+ }
+
+ private JsonNode parseBackfillKey(String backfillLastEvaluatedKey) {
+ if (backfillLastEvaluatedKey == null) {
+ return null;
+ }
+ try {
+ return objectMapper.readTree(backfillLastEvaluatedKey);
+ } catch (JsonProcessingException e) {
+ throw new AwsException("InternalFailure",
+ "Could not parse a persisted zero-ETL backfill checkpoint.", 500);
+ }
+ }
+
+ private String writeAsString(JsonNode node) {
+ try {
+ return objectMapper.writeValueAsString(node);
+ } catch (JsonProcessingException e) {
+ throw new AwsException("InternalFailure",
+ "Could not persist a zero-ETL backfill checkpoint.", 500);
+ }
+ }
+
+ private static String sha256Hex(String value) {
+ try {
+ byte[] digest = MessageDigest.getInstance("SHA-256").digest(value.getBytes(StandardCharsets.UTF_8));
+ StringBuilder sb = new StringBuilder(digest.length * 2);
+ for (byte b : digest) {
+ sb.append(String.format("%02x", b));
+ }
+ return sb.toString();
+ } catch (NoSuchAlgorithmException e) {
+ throw new IllegalStateException("SHA-256 is not available", e);
+ }
+ }
+
public void reset() {
for (String integrationArn : timerIds.keySet()) {
stopPolling(integrationArn);
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftService.java b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftService.java
index 546ef96f14..516b9d543e 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftService.java
@@ -371,8 +371,8 @@ public class RedshiftService {
integration.setRetryCount(0);
integration.setLastError(null);
integration.setPollingEnabled(true);
- // Real integrations pass through creating before settling; nothing here has work to do.
- integration.setStatus("active");
+ // Floci approximation: report `syncing` while the backfill scan runs, then `active`.
+ integration.setStatus("syncing");
integration.setKmsKeyId(kmsKeyId);
integration.setCreateTime(DateTimeFormatter.ISO_INSTANT.format(Instant.now()));
integration.setDescription(description);
@@ -480,6 +480,26 @@ public class RedshiftService {
throw new AwsException("IntegrationNotFoundFault", "The requested integration doesn't exist.", 404);
}
+ public synchronized void updateIntegrationBackfillProgress(String accountId, String integrationArn,
+ String backfillLastEvaluatedKey,
+ boolean backfillCompleted) {
+ for (String key : integrations.keysForAccount(accountId)) {
+ Optional<Integration> stored = integrations.getForAccount(accountId, key);
+ if (stored.isEmpty() || !integrationArn.equals(stored.get().getIntegrationArn())) {
+ continue;
+ }
+ Integration integration = stored.get();
+ integration.setBackfillLastEvaluatedKey(backfillLastEvaluatedKey);
+ integration.setBackfillCompleted(backfillCompleted);
+ integration.setRetryCount(0);
+ integration.setLastError(null);
+ integration.setStatus(backfillCompleted ? "active" : "syncing");
+ integrations.putForAccount(accountId, key, integration);
+ return;
+ }
+ throw new AwsException("IntegrationNotFoundFault", "The requested integration doesn't exist.", 404);
+ }
+
/** One page of integrations plus the marker to continue with, or {@code null} at the end. */
public record IntegrationPage(List<Integration> integrations, String marker) {}
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/model/Integration.java b/src/main/java/io/github/hectorvent/floci/services/redshift/model/Integration.java
index 085b35b75e..4b26c99003 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/model/Integration.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/model/Integration.java
@@ -24,6 +24,8 @@ public class Integration {
private String targetClusterIdentifier;
private String landingTableName;
private String checkpointSequenceNumber;
+ private String backfillLastEvaluatedKey;
+ private boolean backfillCompleted;
private int retryCount;
private String lastError;
private boolean pollingEnabled;
@@ -55,6 +57,10 @@ public class Integration {
public void setLandingTableName(String landingTableName) { this.landingTableName = landingTableName; }
public String getCheckpointSequenceNumber() { return checkpointSequenceNumber; }
public void setCheckpointSequenceNumber(String checkpointSequenceNumber) { this.checkpointSequenceNumber = checkpointSequenceNumber; }
+ public String getBackfillLastEvaluatedKey() { return backfillLastEvaluatedKey; }
+ public void setBackfillLastEvaluatedKey(String backfillLastEvaluatedKey) { this.backfillLastEvaluatedKey = backfillLastEvaluatedKey; }
+ public boolean isBackfillCompleted() { return backfillCompleted; }
+ public void setBackfillCompleted(boolean backfillCompleted) { this.backfillCompleted = backfillCompleted; }
public int getRetryCount() { return retryCount; }
public void setRetryCount(int retryCount) { this.retryCount = retryCount; }
public String getLastError() { return lastError; }
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.java
index bb5d52c626..22d0e82bb1 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParser.java
@@ -26,7 +26,42 @@ public final class CopyStatementParser {
boolean gzip,
boolean csv,
String nullAs,
- String iamRoleArn) implements S3Statement {
+ String iamRoleArn,
+ boolean jsonAuto,
+ boolean jsonAutoIgnoreCase,
+ boolean manifest) implements S3Statement {
+
+ public S3CopyFrom(
+ String targetTable,
+ List<String> columns,
+ String bucket,
+ String keyOrPrefix,
+ String delimiter,
+ int headerLines,
+ boolean gzip,
+ boolean csv,
+ String nullAs,
+ String iamRoleArn,
+ boolean jsonAuto,
+ boolean manifest) {
+ this(targetTable, columns, bucket, keyOrPrefix, delimiter, headerLines, gzip, csv,
+ nullAs, iamRoleArn, jsonAuto, false, manifest);
+ }
+
+ public S3CopyFrom(
+ String targetTable,
+ List<String> columns,
+ String bucket,
+ String keyOrPrefix,
+ String delimiter,
+ int headerLines,
+ boolean gzip,
+ boolean csv,
+ String nullAs,
+ String iamRoleArn) {
+ this(targetTable, columns, bucket, keyOrPrefix, delimiter, headerLines, gzip, csv,
+ nullAs, iamRoleArn, false, false, false);
+ }
}
public record S3Unload(
@@ -88,6 +123,8 @@ public final class CopyStatementParser {
private static final Pattern HEADER_PATTERN = Pattern.compile("(?i)\\bHEADER\\b");
private static final Pattern GZIP_PATTERN = Pattern.compile("(?i)\\bGZIP\\b");
private static final Pattern CSV_PATTERN = Pattern.compile("(?i)\\b(?:FORMAT\\s+(?:AS\\s+)?)?CSV\\b");
+ private static final Pattern JSON_AUTO_PATTERN = Pattern.compile(
+ "(?i)\\b(?:FORMAT\\s+(?:AS\\s+)?)?JSON(?:\\s+AS)?\\s+['\"]auto(?:\\s+(ignorecase))?['\"]");
/**
* Options this simulator does not implement. A COPY carrying any of these is not intercepted:
@@ -95,7 +132,7 @@ public final class CopyStatementParser {
* silently loading the data with the wrong framing.
*/
private static final Pattern UNSUPPORTED_CLAUSE = Pattern.compile(
- "(?i)\\b(FIXEDWIDTH|PARQUET|AVRO|ORC|JSON|SHAPEFILE|BZIP2|LZOP|ZSTD|MANIFEST|MAXERROR"
+ "(?i)\\b(FIXEDWIDTH|PARQUET|AVRO|ORC|SHAPEFILE|BZIP2|LZOP|ZSTD|MAXERROR"
+ "|DATEFORMAT|TIMEFORMAT|ENCRYPTED|ENCODING|REGION|CREDENTIALS"
+ "|ACCESS_KEY_ID|SECRET_ACCESS_KEY|SESSION_TOKEN|MASTER_SYMMETRIC_KEY|KMS_KEY_ID"
+ "|ACCEPTINVCHARS|ACCEPTANYDATE|BLANKSASNULL|EMPTYASNULL|FILLRECORD|TRIMBLANKS"
@@ -329,14 +366,18 @@ public final class CopyStatementParser {
boolean csv = false;
boolean gzip = false;
+ boolean jsonAuto = false;
+ boolean jsonAutoIgnoreCase = false;
+ boolean manifest = false;
String nullAs = null;
String delimiter = null;
int headerLines = 0;
-
String iamRoleArn = null;
boolean seenCsv = false;
boolean seenGzip = false;
+ boolean seenJson = false;
+ boolean seenManifest = false;
boolean seenNull = false;
boolean seenDelimiter = false;
boolean seenHeader = false;
@@ -344,6 +385,8 @@ public final class CopyStatementParser {
Matcher csvMatcher = CSV_PATTERN.matcher(options);
Matcher gzipMatcher = GZIP_PATTERN.matcher(options);
+ Matcher jsonAutoMatcher = JSON_AUTO_PATTERN.matcher(options);
+ Matcher manifestMatcher = MANIFEST_PATTERN.matcher(options);
Matcher ignoreHeaderMatcher = IGNOREHEADER_PATTERN.matcher(options);
Matcher headerMatcher = HEADER_PATTERN.matcher(options);
Matcher delimiterMatcher = DELIMITER_PATTERN.matcher(options);
@@ -361,12 +404,27 @@ public final class CopyStatementParser {
}
if (matchClause(csvMatcher, offset, len)) {
- if (seenCsv) {
+ if (seenCsv || seenJson) {
return null;
}
seenCsv = true;
csv = true;
offset = csvMatcher.end();
+ } else if (matchClause(jsonAutoMatcher, offset, len)) {
+ if (seenJson || seenCsv) {
+ return null;
+ }
+ seenJson = true;
+ jsonAuto = true;
+ jsonAutoIgnoreCase = jsonAutoMatcher.group(1) != null;
+ offset = jsonAutoMatcher.end();
+ } else if (matchClause(manifestMatcher, offset, len)) {
+ if (seenManifest) {
+ return null;
+ }
+ seenManifest = true;
+ manifest = true;
+ offset = manifestMatcher.end();
} else if (matchClause(gzipMatcher, offset, len)) {
if (seenGzip) {
return null;
@@ -418,7 +476,8 @@ public final class CopyStatementParser {
delimiter = csv ? "," : "|";
}
- return new S3CopyFrom(table, columns, bucket, keyOrPrefix, delimiter, headerLines, gzip, csv, nullAs, iamRoleArn);
+ return new S3CopyFrom(table, columns, bucket, keyOrPrefix, delimiter, headerLines, gzip, csv,
+ nullAs, iamRoleArn, jsonAuto, jsonAutoIgnoreCase, manifest);
}
private static boolean matchClause(Matcher m, int start, int end) {
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/JsonLinesToCsvConverter.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/JsonLinesToCsvConverter.java
new file mode 100644
index 0000000000..8978313f54
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/JsonLinesToCsvConverter.java
@@ -0,0 +1,122 @@
+package io.github.hectorvent.floci.services.redshift.proxy;
+
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+final class JsonLinesToCsvConverter {
+ private static final ObjectMapper MAPPER = new ObjectMapper();
+ private static final String SQLSTATE_INTERNAL = "XX000";
+
+ private JsonLinesToCsvConverter() {
+ }
+
+ static void convert(InputStream in, List<String> targetColumns, OutputStream out) throws IOException {
+ convert(in, targetColumns, out, false);
+ }
+
+ static void convert(InputStream in, List<String> targetColumns, OutputStream out, boolean ignoreCase) throws IOException {
+ if (targetColumns == null || targetColumns.isEmpty()) {
+ // Catalog-based column discovery for FORMAT AS JSON 'auto' only runs over the Simple
+ // Query protocol; over Extended Query the column list is fixed at Parse time, before
+ // any backend round trip is possible. This is the only way targetColumns ever arrives
+ // empty here, so the message names Extended Query directly instead of guessing.
+ throw new S3CopySimulator.S3TransferException(SQLSTATE_INTERNAL,
+ "COPY ... FORMAT AS JSON 'auto' requires an explicit column list over the "
+ + "Extended Query protocol; specify columns, or connect with "
+ + "preferQueryMode=simple to use catalog-based column discovery", null);
+ }
+
+ List<String> lookupColumns = ignoreCase
+ ? targetColumns.stream().map(String::toLowerCase).toList()
+ : targetColumns;
+
+ try (JsonParser parser = MAPPER.getFactory().createParser(in)) {
+ while (parser.nextToken() != null) {
+ JsonNode rootNode;
+ try {
+ rootNode = MAPPER.readTree(parser);
+ } catch (IOException e) {
+ throw new S3CopySimulator.S3TransferException(SQLSTATE_INTERNAL,
+ "JSON parse error in S3 data: " + e.getMessage(), e);
+ }
+ if (rootNode == null) {
+ continue;
+ }
+ if (!rootNode.isObject()) {
+ String preview = rootNode.toString();
+ if (preview.length() > 60) {
+ preview = preview.substring(0, 60) + "...";
+ }
+ throw new S3CopySimulator.S3TransferException(SQLSTATE_INTERNAL,
+ "JSON value is not an object: " + preview, null);
+ }
+
+ Map<String, JsonNode> fieldMap = null;
+ if (ignoreCase) {
+ fieldMap = new HashMap<>();
+ Iterator<Map.Entry<String, JsonNode>> fields = rootNode.fields();
+ while (fields.hasNext()) {
+ Map.Entry<String, JsonNode> entry = fields.next();
+ fieldMap.put(entry.getKey().toLowerCase(), entry.getValue());
+ }
+ }
+
+ StringBuilder csvLine = new StringBuilder();
+ for (int i = 0; i < lookupColumns.size(); i++) {
+ if (i > 0) {
+ csvLine.append(',');
+ }
+ String col = lookupColumns.get(i);
+ JsonNode val = ignoreCase ? fieldMap.get(col) : rootNode.get(col);
+ if (val == null || val.isNull()) {
+ // NULL in Postgres CSV is unquoted empty string
+ continue;
+ }
+ if (val.isNumber() || val.isBoolean()) {
+ csvLine.append(val.asText());
+ } else if (val.isTextual()) {
+ String text = val.asText();
+ if (text.isEmpty()) {
+ csvLine.append("\"\"");
+ } else {
+ csvLine.append(escapeCsv(text));
+ }
+ } else {
+ // Nested Object or Array: serialize as JSON text
+ csvLine.append(escapeCsv(MAPPER.writeValueAsString(val)));
+ }
+ }
+ csvLine.append('\n');
+ byte[] bytes = csvLine.toString().getBytes(StandardCharsets.UTF_8);
+ out.write(bytes);
+ }
+ } catch (IOException e) {
+ Throwable cause = e.getCause() != null ? e.getCause() : e;
+ throw new S3CopySimulator.S3TransferException(SQLSTATE_INTERNAL,
+ "JSON parse error in S3 data: " + cause.getMessage(), cause);
+ }
+ out.flush();
+ }
+
+ static String escapeCsv(String text) {
+ if (text == null) {
+ return "";
+ }
+ boolean needsQuotes = text.contains(",") || text.contains("\"") || text.contains("\n") || text.contains("\r");
+ String escaped = text.replace("\"", "\"\"");
+ if (needsQuotes) {
+ return "\"" + escaped + "\"";
+ }
+ return escaped;
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/ManifestReader.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/ManifestReader.java
new file mode 100644
index 0000000000..26915bad91
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/ManifestReader.java
@@ -0,0 +1,74 @@
+package io.github.hectorvent.floci.services.redshift.proxy;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import io.github.hectorvent.floci.services.s3.S3Service;
+import io.github.hectorvent.floci.services.s3.model.S3Object;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+final class ManifestReader {
+ private static final ObjectMapper MAPPER = new ObjectMapper();
+ private static final String SQLSTATE_INTERNAL = "XX000";
+
+ private ManifestReader() {
+ }
+
+ static List<String> resolveManifestKeys(String bucket, String manifestKey, S3Service s3) {
+ S3Object object = s3.getObject(bucket, manifestKey);
+ if (object == null || object.getData() == null) {
+ throw new S3CopySimulator.S3TransferException(SQLSTATE_INTERNAL,
+ "S3 manifest file not found: s3://" + bucket + "/" + manifestKey, null);
+ }
+ JsonNode root;
+ try {
+ root = MAPPER.readTree(new String(object.getData(), StandardCharsets.UTF_8));
+ } catch (IOException e) {
+ throw new S3CopySimulator.S3TransferException(SQLSTATE_INTERNAL,
+ "Invalid manifest file format in s3://" + bucket + "/" + manifestKey, e);
+ }
+ JsonNode entries = root.get("entries");
+ if (entries == null || !entries.isArray()) {
+ throw new S3CopySimulator.S3TransferException(SQLSTATE_INTERNAL,
+ "Manifest file missing 'entries' array", null);
+ }
+
+ List<String> keys = new ArrayList<>();
+ for (JsonNode entry : entries) {
+ JsonNode urlNode = entry.get("url");
+ if (urlNode == null || !urlNode.isTextual()) {
+ continue;
+ }
+ String url = urlNode.asText();
+ boolean mandatory = entry.has("mandatory") && entry.get("mandatory").asBoolean(false);
+ if (!url.startsWith("s3://")) {
+ throw new S3CopySimulator.S3TransferException(SQLSTATE_INTERNAL,
+ "Invalid S3 URL in manifest: " + url, null);
+ }
+ String path = url.substring(5);
+ int slash = path.indexOf('/');
+ if (slash < 0) {
+ throw new S3CopySimulator.S3TransferException(SQLSTATE_INTERNAL,
+ "Malformed S3 URL in manifest: " + url, null);
+ }
+ String entryBucket = path.substring(0, slash);
+ String entryKey = path.substring(slash + 1);
+
+ if (!bucket.equals(entryBucket)) {
+ throw new S3CopySimulator.S3TransferException(SQLSTATE_INTERNAL,
+ "Cross-bucket manifest entry not supported: " + url, null);
+ }
+
+ if (s3.objectExists(entryBucket, entryKey)) {
+ keys.add(entryKey);
+ } else if (mandatory) {
+ throw new S3CopySimulator.S3TransferException(SQLSTATE_INTERNAL,
+ "The specified key does not exist: s3://" + entryBucket + "/" + entryKey, null);
+ }
+ }
+ return keys;
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.java b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.java
index 12e4fdf01e..d72944b587 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulator.java
@@ -28,6 +28,7 @@ import java.util.Map;
import java.util.Optional;
import java.util.concurrent.Semaphore;
import java.util.function.IntConsumer;
+import java.util.stream.Collectors;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;
@@ -231,11 +232,20 @@ public final class S3CopySimulator {
: null;
try {
try {
- if (roleSession != null) {
- authorizeRoleAction(s3, iamService, spec.iamRoleArn(), "s3:ListBucket", bucketArn(spec.bucket()));
- s3.authorizeSignedListBucket(roleSession.accessKeyId(), roleSession.sessionToken(), spec.bucket());
+ if (spec.manifest()) {
+ if (roleSession != null) {
+ authorizeRoleAction(s3, iamService, spec.iamRoleArn(), "s3:GetObject", objectArn(spec.bucket(), spec.keyOrPrefix()));
+ s3.authorizeSignedGetObject(roleSession.accessKeyId(), roleSession.sessionToken(), spec.bucket(), spec.keyOrPrefix());
+ } else {
+ s3.authorizeAnonymousGetObject(spec.bucket(), spec.keyOrPrefix());
+ }
} else {
- s3.authorizeAnonymousListBucket(spec.bucket());
+ if (roleSession != null) {
+ authorizeRoleAction(s3, iamService, spec.iamRoleArn(), "s3:ListBucket", bucketArn(spec.bucket()));
+ s3.authorizeSignedListBucket(roleSession.accessKeyId(), roleSession.sessionToken(), spec.bucket());
+ } else {
+ s3.authorizeAnonymousListBucket(spec.bucket());
+ }
}
} catch (AwsException e) {
throw new S3TransferException(SQLSTATE_INSUFFICIENT_PRIVILEGE,
@@ -275,11 +285,19 @@ public final class S3CopySimulator {
}
static String copyBackendSql(CopyStatementParser.S3CopyFrom spec) {
- return fabricateCopy(spec);
+ return fabricateCopy(spec, null);
+ }
+
+ static String copyBackendSql(CopyStatementParser.S3CopyFrom spec, List<String> discoveredColumns) {
+ return fabricateCopy(spec, discoveredColumns);
}
static void streamCopyInput(CopyInput input, OutputStream backendOut) throws IOException {
- streamObjects(input.spec(), input.s3(), input.iamService(), input.roleSession(), input.keys(), backendOut);
+ streamCopyInput(input, null, backendOut);
+ }
+
+ static void streamCopyInput(CopyInput input, List<String> discoveredColumns, OutputStream backendOut) throws IOException {
+ streamObjects(input.spec(), discoveredColumns, input.s3(), input.iamService(), input.roleSession(), input.keys(), backendOut);
}
static void releaseCopySession(CopyInput input) {
@@ -398,8 +416,16 @@ public final class S3CopySimulator {
}
try {
+ List<String> discoveredColumns = null;
+ if (spec.jsonAuto() && (spec.columns() == null || spec.columns().isEmpty())) {
+ discoveredColumns = discoverTableColumns(client, backend, spec, txStatus, onStatusChange);
+ if (discoveredColumns == null) {
+ return true;
+ }
+ }
+
OutputStream backendOut = backend.getOutputStream();
- backendOut.write(PostgresWireDecoder.encodeQuery(copyBackendSql(spec)));
+ backendOut.write(PostgresWireDecoder.encodeQuery(copyBackendSql(spec, discoveredColumns)));
backendOut.flush();
PostgresWireDecoder backendDecoder = new PostgresWireDecoder(backend.getInputStream());
@@ -432,7 +458,7 @@ public final class S3CopySimulator {
// a CopyFail to the backend, whose ErrorResponse/ReadyForQuery is relayed to the client;
// or, if the backend is unreachable, one synthesized ErrorResponse/ReadyForQuery.
try {
- streamCopyInput(input, backendOut);
+ streamCopyInput(input, discoveredColumns, backendOut);
writeCopyDone(backendOut);
drainToReadyForQuery(backendDecoder, client, onStatusChange);
} catch (RuntimeException | IOException e) {
@@ -461,6 +487,9 @@ public final class S3CopySimulator {
}
private static List<String> resolveKeys(CopyStatementParser.S3CopyFrom spec, S3Service s3) {
+ if (spec.manifest()) {
+ return ManifestReader.resolveManifestKeys(spec.bucket(), spec.keyOrPrefix(), s3);
+ }
List<String> keys = new ArrayList<>();
if (s3.objectExists(spec.bucket(), spec.keyOrPrefix())) {
keys.add(spec.keyOrPrefix());
@@ -482,13 +511,22 @@ public final class S3CopySimulator {
return keys;
}
- private static String fabricateCopy(CopyStatementParser.S3CopyFrom spec) {
+ private static String fabricateCopy(CopyStatementParser.S3CopyFrom spec, List<String> discoveredColumns) {
StringBuilder sql = new StringBuilder("COPY ").append(spec.targetTable());
- if (spec.columns() != null && !spec.columns().isEmpty()) {
- sql.append(" (").append(String.join(", ", spec.columns())).append(")");
+ if (discoveredColumns != null && !discoveredColumns.isEmpty()) {
+ sql.append(" (")
+ .append(discoveredColumns.stream()
+ .map(c -> "\"" + c.replace("\"", "\"\"") + "\"")
+ .collect(Collectors.joining(", ")))
+ .append(")");
+ } else if (spec.columns() != null && !spec.columns().isEmpty()) {
+ sql.append(" (")
+ .append(String.join(", ", spec.columns()))
+ .append(")");
}
- sql.append(" FROM STDIN WITH (FORMAT ").append(spec.csv() ? "csv" : "text");
- String delimiter = spec.delimiter() != null ? spec.delimiter() : (spec.csv() ? "," : "|");
+ boolean csv = spec.csv() || spec.jsonAuto();
+ sql.append(" FROM STDIN WITH (FORMAT ").append(csv ? "csv" : "text");
+ String delimiter = spec.jsonAuto() ? "," : (spec.delimiter() != null ? spec.delimiter() : (csv ? "," : "|"));
sql.append(", DELIMITER '").append(quoteLiteral(delimiter)).append("'");
if (spec.nullAs() != null) {
sql.append(", NULL '").append(quoteLiteral(spec.nullAs())).append("'");
@@ -497,6 +535,14 @@ public final class S3CopySimulator {
return sql.toString();
}
+ private static String unquoteIdentifier(String identifier) {
+ if (identifier != null && identifier.length() >= 2
+ && identifier.startsWith("\"") && identifier.endsWith("\"")) {
+ return identifier.substring(1, identifier.length() - 1).replace("\"\"", "\"");
+ }
+ return identifier;
+ }
+
/**
* Quote a value for a single-quoted SQL string literal. PostgreSQL defaults to
* {@code standard_conforming_strings = on}, where a backslash is an ordinary character, so only
@@ -506,40 +552,150 @@ public final class S3CopySimulator {
return value.replace("'", "''");
}
- private static void streamObjects(CopyStatementParser.S3CopyFrom spec, S3Service s3,
- IamService iamService, RoleSession roleSession,
+ private static void streamObjects(CopyStatementParser.S3CopyFrom spec, List<String> discoveredColumns,
+ S3Service s3, IamService iamService, RoleSession roleSession,
List<String> keys, OutputStream backendOut) throws IOException {
byte[] buffer = new byte[CHUNK];
for (int i = 0; i < keys.size(); i++) {
if (roleSession != null) {
authorizeRoleAction(s3, iamService, spec.iamRoleArn(), "s3:GetObject", objectArn(spec.bucket(), keys.get(i)));
s3.authorizeSignedGetObject(roleSession.accessKeyId(), roleSession.sessionToken(), spec.bucket(), keys.get(i));
+ } else {
+ s3.authorizeAnonymousGetObject(spec.bucket(), keys.get(i));
}
S3Object object = s3.getObject(spec.bucket(), keys.get(i));
byte[] data = object != null && object.getData() != null ? object.getData() : new byte[0];
- InputStream in = new ByteArrayInputStream(data);
- if (spec.gzip()) {
- in = new GZIPInputStream(in);
+ InputStream rawIn = new ByteArrayInputStream(data);
+ try (InputStream in = spec.gzip() ? new GZIPInputStream(rawIn) : rawIn) {
+ if (spec.jsonAuto()) {
+ List<String> targetCols = (discoveredColumns != null && !discoveredColumns.isEmpty())
+ ? discoveredColumns
+ : (spec.columns() != null
+ ? spec.columns().stream().map(S3CopySimulator::unquoteIdentifier).toList()
+ : List.of());
+ CopyDataOutputStream copyDataOut = new CopyDataOutputStream(backendOut);
+ JsonLinesToCsvConverter.convert(in, targetCols, copyDataOut, spec.jsonAutoIgnoreCase());
+ copyDataOut.flush();
+ } else {
+ if (i == 0 && spec.headerLines() > 0) {
+ skipLines(in, spec.headerLines());
+ }
+ int read;
+ boolean endsWithNewline = false;
+ boolean hasData = false;
+ while ((read = in.read(buffer)) != -1) {
+ if (read > 0) {
+ hasData = true;
+ endsWithNewline = (buffer[read - 1] == '\n');
+ writeCopyData(backendOut, buffer, read);
+ }
+ }
+ if (hasData && !endsWithNewline) {
+ writeCopyData(backendOut, new byte[]{'\n'}, 1);
+ }
+ }
+ }
+ }
+ backendOut.flush();
+ }
+
+ private static List<String> discoverTableColumns(Socket client, Socket backend,
+ CopyStatementParser.S3CopyFrom spec,
+ char txStatus, IntConsumer onStatusChange) throws IOException {
+ String query = "SELECT a.attname FROM pg_catalog.pg_attribute a "
+ + "WHERE a.attrelid = to_regclass('" + quoteLiteral(spec.targetTable()) + "') "
+ + "AND a.attnum > 0 AND NOT a.attisdropped "
+ + "ORDER BY a.attnum";
+
+ OutputStream backendOut = backend.getOutputStream();
+ backendOut.write(PostgresWireDecoder.encodeQuery(query));
+ backendOut.flush();
+
+ PostgresWireDecoder backendDecoder = new PostgresWireDecoder(backend.getInputStream());
+ List<String> cols = new ArrayList<>();
+ PostgresWireDecoder.FrontendMessage msg;
+ while ((msg = backendDecoder.nextMessage()) != null) {
+ char type = msg.type();
+ if (type == 'D') {
+ byte[] body = msg.body();
+ if (body.length >= 6) {
+ int colCount = ((body[0] & 0xFF) << 8) | (body[1] & 0xFF);
+ if (colCount >= 1) {
+ int colLen = ((body[2] & 0xFF) << 24) | ((body[3] & 0xFF) << 16)
+ | ((body[4] & 0xFF) << 8) | (body[5] & 0xFF);
+ if (colLen > 0 && 6 + colLen <= body.length) {
+ cols.add(new String(body, 6, colLen, StandardCharsets.UTF_8));
+ }
+ }
+ }
+ } else if (type == 'E') {
+ forward(client, msg);
+ drainToReadyForQuery(backendDecoder, client, onStatusChange);
+ return null;
+ } else if (type == 'N' || type == 'A' || type == 'S') {
+ forward(client, msg);
+ } else if (type == 'Z') {
+ if (onStatusChange != null && msg.body().length > 0) {
+ onStatusChange.accept(msg.body()[0]);
+ }
+ break;
}
- if (i == 0 && spec.headerLines() > 0) {
- skipLines(in, spec.headerLines());
+ }
+ if (cols.isEmpty()) {
+ sendError(client, backend, "42P01",
+ "relation \"" + spec.targetTable() + "\" does not exist or has no user columns",
+ txStatus, onStatusChange);
+ return null;
+ }
+ return cols;
+ }
+
+ private static final class CopyDataOutputStream extends OutputStream {
+ private final OutputStream backendOut;
+ private final byte[] chunkBuf = new byte[CHUNK];
+ private int count = 0;
+
+ CopyDataOutputStream(OutputStream backendOut) {
+ this.backendOut = backendOut;
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+ chunkBuf[count++] = (byte) b;
+ if (count == chunkBuf.length) {
+ flushChunk();
}
- int read;
- boolean endsWithNewline = false;
- boolean hasData = false;
- while ((read = in.read(buffer)) != -1) {
- if (read > 0) {
- hasData = true;
- endsWithNewline = (buffer[read - 1] == '\n');
- writeCopyData(backendOut, buffer, read);
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ int remaining = len;
+ int offset = off;
+ while (remaining > 0) {
+ int space = chunkBuf.length - count;
+ int toCopy = Math.min(remaining, space);
+ System.arraycopy(b, offset, chunkBuf, count, toCopy);
+ count += toCopy;
+ offset += toCopy;
+ remaining -= toCopy;
+ if (count == chunkBuf.length) {
+ flushChunk();
}
}
- in.close();
- if (hasData && !endsWithNewline) {
- writeCopyData(backendOut, new byte[]{'\n'}, 1);
+ }
+
+ private void flushChunk() throws IOException {
+ if (count > 0) {
+ writeCopyData(backendOut, chunkBuf, count);
+ count = 0;
}
}
- backendOut.flush();
+
+ @Override
+ public void flush() throws IOException {
+ flushChunk();
+ backendOut.flush();
+ }
}
private static void skipLines(InputStream in, int lines) throws IOException {
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftDynamoDbZeroEtlConsumerTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftDynamoDbZeroEtlConsumerTest.java
index e59b89545b..7610d7a966 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftDynamoDbZeroEtlConsumerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftDynamoDbZeroEtlConsumerTest.java
@@ -1,25 +1,51 @@
package io.github.hectorvent.floci.services.redshift;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.core.common.RequestScopes;
+import io.github.hectorvent.floci.services.dynamodb.DynamoDbService;
import io.github.hectorvent.floci.services.dynamodb.DynamoDbStreamService;
+import io.github.hectorvent.floci.services.dynamodb.model.AttributeDefinition;
import io.github.hectorvent.floci.services.dynamodb.model.DynamoDbStreamRecord;
+import io.github.hectorvent.floci.services.dynamodb.model.KeySchemaElement;
import io.github.hectorvent.floci.services.redshift.model.Integration;
import io.github.hectorvent.floci.services.redshiftdata.RedshiftZeroEtlWriter;
+import io.quarkus.test.junit.QuarkusTest;
+import jakarta.inject.Inject;
import org.junit.jupiter.api.Test;
+import org.mockito.ArgumentCaptor;
import java.util.List;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.eq;
-import static org.mockito.Mockito.*;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+@QuarkusTest
class RedshiftDynamoDbZeroEtlConsumerTest {
+ private static final ObjectMapper MAPPER = new ObjectMapper();
+
+ @Inject
+ DynamoDbService dynamoDbService;
+
@Test
void writesRecordsAndAdvancesCheckpointOnlyAfterSuccessfulBatch() {
DynamoDbStreamService streamService = mock(DynamoDbStreamService.class);
RedshiftService redshiftService = mock(RedshiftService.class);
RedshiftZeroEtlWriter writer = mock(RedshiftZeroEtlWriter.class);
- Integration integration = integration();
+ Integration integration = integration("orders");
+ integration.setBackfillCompleted(true);
DynamoDbStreamRecord record = new DynamoDbStreamRecord();
record.setEventId("event-1");
record.setSequenceNumber("000000000000000000001");
@@ -32,7 +58,7 @@ class RedshiftDynamoDbZeroEtlConsumerTest {
.thenReturn(record.getSequenceNumber());
RedshiftDynamoDbZeroEtlConsumer consumer =
- new RedshiftDynamoDbZeroEtlConsumer(streamService, redshiftService, writer);
+ new RedshiftDynamoDbZeroEtlConsumer(streamService, dynamoDbService, redshiftService, writer);
consumer.pollOnce(integration);
verify(writer).createLandingTable(integration.getAccountId(), "warehouse", "floci_zetl_orders");
@@ -41,11 +67,120 @@ class RedshiftDynamoDbZeroEtlConsumerTest {
record.getSequenceNumber(), true, null);
}
- private static Integration integration() {
+ @Test
+ void backfillsOnePageAtATimeBeforeSwitchingToStreamPolling() {
+ DynamoDbStreamService streamService = mock(DynamoDbStreamService.class);
+ RedshiftService redshiftService = mock(RedshiftService.class);
+ RedshiftZeroEtlWriter writer = mock(RedshiftZeroEtlWriter.class);
+ String tableName = "orders-page-" + System.nanoTime();
+ Integration integration = integration(tableName);
+ assertFalse(integration.isBackfillCompleted());
+
+ // Create table and items in the integration's account scope (111111111111).
+ // Populate > BATCH_SIZE (100) items so the scan produces a LastEvaluatedKey.
+ RequestScopes.runAs(integration.getAccountId(), () -> {
+ dynamoDbService.createTable(tableName,
+ List.of(new KeySchemaElement("id", "HASH")),
+ List.of(new AttributeDefinition("id", "S")),
+ 5L, 5L, "us-east-1");
+ for (int i = 1; i <= 101; i++) {
+ dynamoDbService.putItem(tableName, itemWithId(String.valueOf(i)), "us-east-1");
+ }
+ });
+
+ // The table must NOT exist in the default account: this proves the account scope is applied.
+ assertThrows(AwsException.class, () -> dynamoDbService.describeTable(tableName, "us-east-1"));
+
+ RedshiftDynamoDbZeroEtlConsumer consumer =
+ new RedshiftDynamoDbZeroEtlConsumer(streamService, dynamoDbService, redshiftService, writer);
+ consumer.pollOnce(integration);
+
+ assertFalse(integration.isBackfillCompleted());
+ assertNotNull(integration.getBackfillLastEvaluatedKey());
+ verify(redshiftService).updateIntegrationBackfillProgress(eq(integration.getAccountId()),
+ eq(integration.getIntegrationArn()), eq(integration.getBackfillLastEvaluatedKey()), eq(false));
+ verify(streamService, never()).getShardIterator(any(), any(), any(), any());
+
+ @SuppressWarnings("unchecked")
+ ArgumentCaptor<List<DynamoDbStreamRecord>> captor = ArgumentCaptor.forClass(List.class);
+ verify(writer).writeBatch(eq(integration.getAccountId()), eq("warehouse"), eq("floci_zetl_orders"),
+ captor.capture());
+ assertEquals(100, captor.getValue().size());
+ DynamoDbStreamRecord written = captor.getValue().get(0);
+ assertEquals("INSERT", written.getEventName());
+ assertTrue(written.getEventId().startsWith("backfill#" + integration.getIntegrationArn() + "#"));
+ }
+
+ @Test
+ void backfillCompletesWhenScanReturnsNoLastEvaluatedKey() {
+ DynamoDbStreamService streamService = mock(DynamoDbStreamService.class);
+ RedshiftService redshiftService = mock(RedshiftService.class);
+ RedshiftZeroEtlWriter writer = mock(RedshiftZeroEtlWriter.class);
+ String tableName = "orders-empty-" + System.nanoTime();
+ Integration integration = integration(tableName);
+
+ RequestScopes.runAs(integration.getAccountId(), () -> {
+ dynamoDbService.createTable(tableName,
+ List.of(new KeySchemaElement("id", "HASH")),
+ List.of(new AttributeDefinition("id", "S")),
+ 5L, 5L, "us-east-1");
+ });
+
+ assertThrows(AwsException.class, () -> dynamoDbService.describeTable(tableName, "us-east-1"));
+
+ RedshiftDynamoDbZeroEtlConsumer consumer =
+ new RedshiftDynamoDbZeroEtlConsumer(streamService, dynamoDbService, redshiftService, writer);
+ consumer.pollOnce(integration);
+
+ assertTrue(integration.isBackfillCompleted());
+ verify(redshiftService).updateIntegrationBackfillProgress(integration.getAccountId(),
+ integration.getIntegrationArn(), null, true);
+ verify(writer, never()).writeBatch(any(), any(), any(), any());
+ }
+
+ @Test
+ void sameItemProducesTheSameEventIdAcrossScans() {
+ DynamoDbStreamService streamService = mock(DynamoDbStreamService.class);
+ RedshiftService redshiftService = mock(RedshiftService.class);
+ RedshiftZeroEtlWriter writer = mock(RedshiftZeroEtlWriter.class);
+ String tableName = "orders-same-" + System.nanoTime();
+ Integration integration = integration(tableName);
+
+ ObjectNode item = itemWithId("1");
+ RequestScopes.runAs(integration.getAccountId(), () -> {
+ dynamoDbService.createTable(tableName,
+ List.of(new KeySchemaElement("id", "HASH")),
+ List.of(new AttributeDefinition("id", "S")),
+ 5L, 5L, "us-east-1");
+ dynamoDbService.putItem(tableName, item, "us-east-1");
+ });
+
+ RedshiftDynamoDbZeroEtlConsumer consumer =
+ new RedshiftDynamoDbZeroEtlConsumer(streamService, dynamoDbService, redshiftService, writer);
+ consumer.pollOnce(integration);
+ integration.setBackfillCompleted(false);
+ consumer.pollOnce(integration);
+
+ @SuppressWarnings("unchecked")
+ ArgumentCaptor<List<DynamoDbStreamRecord>> captor = ArgumentCaptor.forClass(List.class);
+ verify(writer, times(2)).writeBatch(any(), any(), any(), captor.capture());
+ assertEquals(captor.getAllValues().get(0).get(0).getEventId(),
+ captor.getAllValues().get(1).get(0).getEventId());
+ }
+
+ private static ObjectNode itemWithId(String id) {
+ ObjectNode item = MAPPER.createObjectNode();
+ ObjectNode idAttr = MAPPER.createObjectNode();
+ idAttr.put("S", id);
+ item.set("id", idAttr);
+ return item;
+ }
+
+ private static Integration integration(String tableName) {
Integration integration = new Integration();
integration.setIntegrationArn("arn:aws:redshift:us-east-1:111111111111:integration:one");
integration.setAccountId("111111111111");
- integration.setSourceStreamArn("arn:aws:dynamodb:us-east-1:111111111111:table/orders/stream/one");
+ integration.setSourceStreamArn("arn:aws:dynamodb:us-east-1:111111111111:table/" + tableName + "/stream/one");
integration.setTargetClusterIdentifier("warehouse");
integration.setLandingTableName("floci_zetl_orders");
integration.setPollingEnabled(true);
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftIntegrationsIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftIntegrationsIntegrationTest.java
index 1845b4a979..e72b0062e9 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftIntegrationsIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftIntegrationsIntegrationTest.java
@@ -1,7 +1,16 @@
package io.github.hectorvent.floci.services.redshift;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import org.awaitility.Awaitility;
import io.restassured.RestAssured;
import io.restassured.response.Response;
import io.quarkus.test.junit.QuarkusTest;
@@ -115,11 +124,81 @@ class RedshiftIntegrationsIntegrationTest {
.body(containsString("zetl-described"))
.body(containsString(source))
.body(containsString(TARGET))
- // Lower case on real Redshift, not ACTIVE.
- .body(containsString("<Status>active</Status>"))
+ // Lower case on real Redshift, not ACTIVE. `syncing` right after creation: the
+ // zero-ETL consumer has not yet confirmed the backfill scan of the source table.
+ .body(containsString("<Status>syncing</Status>"))
.body(containsString("<Errors></Errors>"));
}
+ @Test
+ void anEmptySourceTableEventuallyReachesActiveStatus() {
+ String arn = arnOf(createIntegration("zetl-eventually-active"));
+
+ Awaitility.await()
+ .atMost(Duration.ofSeconds(30))
+ .pollInterval(Duration.ofMillis(500))
+ .untilAsserted(() -> query("Action", "DescribeIntegrations", "IntegrationArn", arn)
+ .then().statusCode(200)
+ .body(containsString("<Status>active</Status>")));
+ }
+
+ @Test
+ void preExistingDynamoDbItemsAreBackfilledIntoTheLandingTable() {
+ RestAssuredJsonUtils.configureAwsContentTypes();
+ String tableName = "keystone-backfill-" + System.nanoTime();
+ String streamArn = createDynamoTable(tableName);
+ given()
+ .header("X-Amz-Target", "DynamoDB_20120810.PutItem")
+ .contentType("application/x-amz-json-1.0")
+ .body("""
+ {
+ "TableName": "%s",
+ "Item": {"id": {"S": "pre-existing-1"}, "note": {"S": "seeded before integration"}}
+ }
+ """.formatted(tableName))
+ .when().post("/")
+ .then().statusCode(200);
+
+ String createResponse = query("Action", "CreateIntegration", "IntegrationName", "zetl-backfill-proof",
+ "SourceArn", streamArn, "TargetArn", TARGET)
+ .then().statusCode(200)
+ .extract().body().asString();
+ String arn = arnOf(createResponse);
+ String landingTable = "floci_zetl_" + arn.substring(arn.lastIndexOf(':') + 1).replace('-', '_');
+ int port = clusterPort("zero-etl-cluster");
+
+ Awaitility.await()
+ .atMost(Duration.ofSeconds(30))
+ .pollInterval(Duration.ofMillis(500))
+ .ignoreExceptions()
+ .untilAsserted(() -> assertTrue(
+ landingTableHasBackfilledRow(port, landingTable, "pre-existing-1")));
+ }
+
+ private static int clusterPort(String clusterIdentifier) {
+ String xml = query("Action", "DescribeClusters", "ClusterIdentifier", clusterIdentifier)
+ .then().statusCode(200).extract().body().asString();
+ Matcher matcher = Pattern.compile("<Endpoint>.*?<Port>(\\d+)</Port>", Pattern.DOTALL).matcher(xml);
+ assertTrue(matcher.find(), "DescribeClusters returned no endpoint port: " + xml);
+ return Integer.parseInt(matcher.group(1));
+ }
+
+ private static boolean landingTableHasBackfilledRow(int port, String landingTable, String itemId)
+ throws SQLException {
+ String url = "jdbc:postgresql://127.0.0.1:" + port + "/dev";
+ try (Connection connection = DriverManager.getConnection(url, "admin", "password123");
+ Statement statement = connection.createStatement();
+ ResultSet rows = statement.executeQuery("SELECT new_image_json FROM " + landingTable
+ + " WHERE event_id LIKE 'backfill#%'")) {
+ while (rows.next()) {
+ if (rows.getString(1).contains(itemId)) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
@Test
void describingByArnReturnsOnlyThatIntegration() {
String arn = arnOf(createIntegration("zetl-byarn"));
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java
index f6071fd8e4..2d20fcabc0 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java
@@ -90,6 +90,16 @@ class RedshiftInterceptorIntegrationTest {
return jdbcUrl(cluster) + "?prepareThreshold=1";
}
+ /**
+ * Catalog-based column discovery for {@code FORMAT AS JSON 'auto'} without an explicit column
+ * list only runs over the Simple Query protocol (see {@code docs/services/redshift.md}), so
+ * tests exercising that discovery must force it explicitly: pgjdbc's default query mode is
+ * Extended even for a plain {@link java.sql.Statement}.
+ */
+ private static String simpleQueryJdbcUrl(Cluster cluster) {
+ return jdbcUrl(cluster) + "?preferQueryMode=simple";
+ }
+
private static Connection waitForConnection(Cluster cluster, String username, String password) throws SQLException {
return waitForConnection(jdbcUrl(cluster), username, password);
}
@@ -443,4 +453,144 @@ class RedshiftInterceptorIntegrationTest {
"UNLOAD ('select id from o_src') TO 's3://redshift-unload-it-ow/o/' ALLOWOVERWRITE");
}
}
+
+ @Test
+ void copyFromS3JsonAutoExplicitColumnsLoadsRows() throws Exception {
+ clusterId = "it-copy-json-explicit";
+ Cluster cluster = service.createCluster(clusterId, "dc2.large", "admin", "Secret123");
+
+ String bucket = "redshift-copy-json-explicit";
+ s3.createBucket(bucket, "us-east-1");
+ String ndjson = "{\"id\": 1, \"name\": \"Alice\", \"score\": 95.5}\n"
+ + "{\"id\": 2, \"name\": \"Bob\", \"score\": 82.0}\n";
+ s3.putObject(bucket, "data/users.json", ndjson.getBytes(StandardCharsets.UTF_8), "application/json", Map.of());
+
+ try (Connection c = waitForConnection(cluster, "admin", "Secret123")) {
+ c.createStatement().execute("CREATE TABLE users (id int, name text, score numeric)");
+ c.createStatement().execute("COPY users (id, name, score) FROM 's3://" + bucket + "/data/users.json' FORMAT AS JSON 'auto'");
+ try (ResultSet rs = c.createStatement().executeQuery("SELECT count(*), sum(score) FROM users")) {
+ assertTrue(rs.next());
+ assertEquals(2, rs.getInt(1));
+ assertEquals(177.5, rs.getDouble(2), 0.001);
+ }
+ }
+ }
+
+ @Test
+ void copyFromS3JsonAutoWithoutColumnsDiscoversSchema() throws Exception {
+ clusterId = "it-copy-json-auto-schema";
+ Cluster cluster = service.createCluster(clusterId, "dc2.large", "admin", "Secret123");
+
+ String bucket = "redshift-copy-json-auto-schema";
+ s3.createBucket(bucket, "us-east-1");
+ String ndjson = "{\"id\": 10, \"name\": \"Charlie\", \"note\": \"hello\"}\n"
+ + "{\"id\": 20, \"name\": \"David\", \"note\": \"world\"}\n";
+ s3.putObject(bucket, "data/items.json", ndjson.getBytes(StandardCharsets.UTF_8), "application/json", Map.of());
+
+ // Column discovery only runs over Simple Query; force it explicitly since pgjdbc's
+ // default query mode is Extended even for a plain Statement.
+ try (Connection c = waitForConnection(simpleQueryJdbcUrl(cluster), "admin", "Secret123")) {
+ c.createStatement().execute("CREATE TABLE items (id int, name text, note text)");
+ c.createStatement().execute("COPY items FROM 's3://" + bucket + "/data/items.json' FORMAT AS JSON 'auto'");
+ try (ResultSet rs = c.createStatement().executeQuery("SELECT count(*), max(id) FROM items")) {
+ assertTrue(rs.next());
+ assertEquals(2, rs.getInt(1));
+ assertEquals(20, rs.getInt(2));
+ }
+ }
+ }
+
+ @Test
+ void copyFromS3JsonAutoWithoutColumnsOverExtendedQueryFailsClearly() throws Exception {
+ clusterId = "it-copy-json-auto-extended-rejects";
+ Cluster cluster = service.createCluster(clusterId, "dc2.large", "admin", "Secret123");
+
+ String bucket = "redshift-copy-json-auto-extended";
+ s3.createBucket(bucket, "us-east-1");
+ s3.putObject(bucket, "data/items.json",
+ "{\"id\": 1}\n".getBytes(StandardCharsets.UTF_8), "application/json", Map.of());
+
+ // Default connection: pgjdbc's default query mode is Extended even for a plain Statement,
+ // where the column list is fixed at Parse time and catalog discovery cannot run.
+ try (Connection c = waitForConnection(cluster, "admin", "Secret123")) {
+ c.createStatement().execute("CREATE TABLE ext_items (id int)");
+ SQLException ex = assertThrows(SQLException.class, () -> c.createStatement().execute(
+ "COPY ext_items FROM 's3://" + bucket + "/data/items.json' FORMAT AS JSON 'auto'"));
+ assertTrue(ex.getMessage().contains("explicit column list"));
+ }
+ }
+
+ @Test
+ void copyFromS3JsonAutoGzipLoadsRows() throws Exception {
+ clusterId = "it-copy-json-gzip";
+ Cluster cluster = service.createCluster(clusterId, "dc2.large", "admin", "Secret123");
+
+ String bucket = "redshift-copy-json-gzip";
+ s3.createBucket(bucket, "us-east-1");
+ ByteArrayOutputStream raw = new ByteArrayOutputStream();
+ try (GZIPOutputStream gz = new GZIPOutputStream(raw)) {
+ gz.write("{\"id\": 100, \"val\": \"zipped\"}\n".getBytes(StandardCharsets.UTF_8));
+ }
+ s3.putObject(bucket, "data/compressed.json.gz", raw.toByteArray(), "application/gzip", Map.of());
+
+ // Column discovery only runs over Simple Query; force it explicitly since pgjdbc's
+ // default query mode is Extended even for a plain Statement.
+ try (Connection c = waitForConnection(simpleQueryJdbcUrl(cluster), "admin", "Secret123")) {
+ c.createStatement().execute("CREATE TABLE gz_items (id int, val text)");
+ c.createStatement().execute("COPY gz_items FROM 's3://" + bucket + "/data/compressed.json.gz' GZIP FORMAT AS JSON 'auto'");
+ try (ResultSet rs = c.createStatement().executeQuery("SELECT count(*), max(val) FROM gz_items")) {
+ assertTrue(rs.next());
+ assertEquals(1, rs.getInt(1));
+ assertEquals("zipped", rs.getString(2));
+ }
+ }
+ }
+
+ @Test
+ void copyFromS3ManifestRoundtrip() throws Exception {
+ clusterId = "it-manifest-roundtrip";
+ Cluster cluster = service.createCluster(clusterId, "dc2.large", "admin", "Secret123");
+
+ String bucket = "redshift-manifest-roundtrip";
+ s3.createBucket(bucket, "us-east-1");
+
+ try (Connection c = waitForConnection(cluster, "admin", "Secret123")) {
+ c.createStatement().execute("CREATE TABLE src (id int, name text)");
+ c.createStatement().execute("INSERT INTO src VALUES (1, 'alpha'), (2, 'beta')");
+
+ // UNLOAD with MANIFEST
+ c.createStatement().execute("UNLOAD ('select id, name from src order by id') "
+ + "TO 's3://" + bucket + "/export/' MANIFEST ALLOWOVERWRITE");
+
+ // Target table
+ c.createStatement().execute("CREATE TABLE dst (id int, name text)");
+
+ // COPY using the generated manifest
+ c.createStatement().execute("COPY dst (id, name) FROM 's3://" + bucket + "/export/manifest' MANIFEST");
+
+ try (ResultSet rs = c.createStatement().executeQuery("SELECT count(*) FROM dst")) {
+ assertTrue(rs.next());
+ assertEquals(2, rs.getInt(1));
+ }
+ }
+ }
+
+ @Test
+ void copyFromS3ManifestMissingMandatoryFails() throws Exception {
+ clusterId = "it-manifest-missing";
+ Cluster cluster = service.createCluster(clusterId, "dc2.large", "admin", "Secret123");
+
+ String bucket = "redshift-manifest-missing";
+ s3.createBucket(bucket, "us-east-1");
+ String manifestJson = "{\"entries\": [{\"url\": \"s3://" + bucket + "/missing.csv\", \"mandatory\": true}]}";
+ s3.putObject(bucket, "manifest", manifestJson.getBytes(StandardCharsets.UTF_8), "application/json", Map.of());
+
+ try (Connection c = waitForConnection(cluster, "admin", "Secret123")) {
+ c.createStatement().execute("CREATE TABLE t_mand (id int)");
+ SQLException ex = assertThrows(SQLException.class,
+ () -> c.createStatement().execute("COPY t_mand FROM 's3://" + bucket + "/manifest' MANIFEST"));
+ assertTrue(ex.getMessage().toLowerCase().contains("does not exist") || ex.getMessage().toLowerCase().contains("missing.csv"), ex.getMessage());
+ }
+ }
}
+
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftServiceTest.java
index 871f629dcd..fbb41fee35 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftServiceTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftServiceTest.java
@@ -35,7 +35,9 @@ import java.util.Map;
import java.util.Optional;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.Mockito.*;
@@ -126,6 +128,51 @@ class RedshiftServiceTest {
assertEquals(streamArn, integration.getSourceStreamArn());
assertEquals("warehouse", integration.getTargetClusterIdentifier());
assertNotNull(integration.getLandingTableName());
+ assertEquals("syncing", integration.getStatus());
+ assertFalse(integration.isBackfillCompleted());
+ }
+
+ @Test
+ void updateIntegrationBackfillProgressPersistsCheckpointAndFlipsStatusOnCompletion() {
+ String streamArn = "arn:aws:dynamodb:us-east-1:111111111111:table/orders/stream/2026-09-18T00:00:00.000";
+ String targetArn = "arn:aws:redshift:us-east-1:111111111111:cluster:warehouse";
+ StreamDescription stream = new StreamDescription();
+ stream.setStreamArn(streamArn);
+ stream.setTableName("orders");
+ stream.setStreamStatus("ENABLED");
+ when(streamService.describeStream(streamArn)).thenReturn(stream);
+ Cluster cluster = new Cluster();
+ cluster.setClusterIdentifier("warehouse");
+ when(clusterBackend.get("warehouse")).thenReturn(Optional.of(cluster));
+ when(integrationBackend.scan(any())).thenReturn(List.of());
+
+ Integration integration = service.createIntegration("orders-to-warehouse", streamArn, targetArn,
+ null, null, Map.of(), Map.of(), "us-east-1");
+
+ ArgumentCaptor<String> keyCaptor = ArgumentCaptor.forClass(String.class);
+ verify(integrationBackend).put(keyCaptor.capture(), eq(integration));
+ String storageKey = keyCaptor.getValue();
+ when(integrationBackend.keysForAccount("111111111111")).thenReturn(java.util.Set.of(storageKey));
+ when(integrationBackend.getForAccount("111111111111", storageKey)).thenReturn(Optional.of(integration));
+
+ service.updateIntegrationBackfillProgress("111111111111", integration.getIntegrationArn(),
+ "{\"id\":{\"S\":\"1\"}}", false);
+ assertEquals("syncing", integration.getStatus());
+ assertEquals("{\"id\":{\"S\":\"1\"}}", integration.getBackfillLastEvaluatedKey());
+ assertFalse(integration.isBackfillCompleted());
+
+ service.updateIntegrationBackfillProgress("111111111111", integration.getIntegrationArn(), null, true);
+ assertEquals("active", integration.getStatus());
+ assertNull(integration.getBackfillLastEvaluatedKey());
+ assertTrue(integration.isBackfillCompleted());
+ }
+
+ @Test
+ void updateIntegrationBackfillProgressOnUnknownIntegrationThrows() {
+ when(integrationBackend.keysForAccount("111111111111")).thenReturn(java.util.Set.of());
+
+ assertThrows(AwsException.class, () -> service.updateIntegrationBackfillProgress(
+ "111111111111", "arn:aws:redshift:us-east-1:111111111111:integration:missing", null, true));
}
@Test
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.java
index ae88d7d8b5..75270541b0 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/CopyStatementParserTest.java
@@ -163,12 +163,13 @@ class CopyStatementParserTest {
@Test
void returnsNullForUnsupportedClauses() {
assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' FORMAT AS PARQUET"));
- assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' JSON 'auto'"));
+ assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' JSON 's3://mybucket/jsonpaths.json'"));
assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' FIXEDWIDTH 'a:1,b:2'"));
assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' GZIP MAXERROR 10"));
assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k' DATEFORMAT 'YYYY-MM-DD'"));
}
+
@Test
void returnsNullWhenAStatementFollowsTheCopy() {
assertNull(CopyStatementParser.parse("COPY t FROM 's3://b/k'; DROP TABLE staging"));
@@ -325,4 +326,67 @@ class CopyStatementParserTest {
assertFalse(u.csv());
assertEquals("csv", u.nullAs());
}
+
+ @Test
+ void parseCopy_jsonAuto_explicitColumns() {
+ CopyStatementParser.S3CopyFrom c = copyFrom("COPY users (id, name) FROM 's3://mybucket/users.json' "
+ + "IAM_ROLE 'arn:aws:iam::123456789012:role/RedshiftRole' "
+ + "FORMAT AS JSON 'auto'");
+ assertEquals("users", c.targetTable());
+ assertEquals(List.of("id", "name"), c.columns());
+ assertEquals("mybucket", c.bucket());
+ assertEquals("users.json", c.keyOrPrefix());
+ assertTrue(c.jsonAuto());
+ assertFalse(c.manifest());
+ }
+
+ @Test
+ void parseCopy_jsonAuto_variations() {
+ CopyStatementParser.S3CopyFrom c1 = copyFrom("COPY tbl FROM 's3://b/data.json' JSON 'auto'");
+ assertTrue(c1.jsonAuto());
+ assertFalse(c1.jsonAutoIgnoreCase());
+
+ CopyStatementParser.S3CopyFrom c2 = copyFrom("COPY tbl FROM 's3://b/data.json' JSON AS 'auto'");
+ assertTrue(c2.jsonAuto());
+ assertFalse(c2.jsonAutoIgnoreCase());
+
+ CopyStatementParser.S3CopyFrom c3 = copyFrom("COPY tbl FROM 's3://b/data.json' FORMAT JSON 'auto'");
+ assertTrue(c3.jsonAuto());
+ assertFalse(c3.jsonAutoIgnoreCase());
+
+ CopyStatementParser.S3CopyFrom c4 = copyFrom("COPY tbl FROM 's3://b/data.json' JSON 'auto ignorecase'");
+ assertTrue(c4.jsonAuto());
+ assertTrue(c4.jsonAutoIgnoreCase());
+
+ CopyStatementParser.S3CopyFrom c5 = copyFrom("COPY tbl FROM 's3://b/data.json' FORMAT AS JSON 'auto ignorecase'");
+ assertTrue(c5.jsonAuto());
+ assertTrue(c5.jsonAutoIgnoreCase());
+ }
+
+ @Test
+ void parseCopy_manifest_gzip() {
+ CopyStatementParser.S3CopyFrom c = copyFrom("COPY sales FROM 's3://mybucket/manifest.json' "
+ + "IAM_ROLE 'arn:aws:iam::123456789012:role/RedshiftRole' "
+ + "MANIFEST GZIP CSV");
+ assertTrue(c.manifest());
+ assertTrue(c.gzip());
+ assertTrue(c.csv());
+ assertFalse(c.jsonAuto());
+ }
+
+ @Test
+ void parseCopy_jsonAutoAndManifest() {
+ CopyStatementParser.S3CopyFrom c = copyFrom("COPY tbl FROM 's3://b/manifest' "
+ + "MANIFEST FORMAT AS JSON 'auto'");
+ assertTrue(c.manifest());
+ assertTrue(c.jsonAuto());
+ }
+
+ @Test
+ void parseCopy_rejectsUnsupportedJsonPathsAndParquet() {
+ // Only JSON 'auto' is supported; jsonpaths or bare JSON must be rejected
+ assertNull(CopyStatementParser.parse("COPY tbl FROM 's3://b/k' JSON 's3://b/paths.json'"));
+ assertNull(CopyStatementParser.parse("COPY tbl FROM 's3://b/k' PARQUET"));
+ }
}
+
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/JsonLinesToCsvConverterTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/JsonLinesToCsvConverterTest.java
new file mode 100644
index 0000000000..58be74c40b
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/JsonLinesToCsvConverterTest.java
@@ -0,0 +1,139 @@
+package io.github.hectorvent.floci.services.redshift.proxy;
+
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+class JsonLinesToCsvConverterTest {
+
+ @Test
+ void convert_caseInsensitiveMappingAndEscaping() throws IOException {
+ String ndjson = "{\"Id\": 1, \"Name\": \"Alice\", \"note\": \"hello, world\"}\n"
+ + "{\"id\": 2, \"NAME\": \"Bob, \\\"The Builder\\\"\", \"note\": \"multi\\nline\"}\n"
+ + "{\"ID\": 3, \"name\": null}\n";
+
+ ByteArrayInputStream in = new ByteArrayInputStream(ndjson.getBytes(StandardCharsets.UTF_8));
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+
+ List<String> columns = List.of("id", "name", "note");
+ JsonLinesToCsvConverter.convert(in, columns, out, true);
+
+ String result = out.toString(StandardCharsets.UTF_8);
+ String[] lines = result.split("\n");
+ assertEquals("1,Alice,\"hello, world\"", lines[0].trim());
+ assertEquals("2,\"Bob, \"\"The Builder\"\"\",\"multi\nline\"", lines[1].trim() + "\n" + lines[2].trim());
+ assertEquals("3,,", lines[3].trim());
+ }
+
+ @Test
+ void convert_caseSensitiveDefault_matchesExactCase() throws IOException {
+ String ndjson = "{\"Id\": 1, \"id\": 10, \"NAME\": \"Alice\", \"name\": \"Bob\"}\n"
+ + "{\"Id\": 2, \"name\": \"Charlie\"}\n";
+
+ ByteArrayInputStream in = new ByteArrayInputStream(ndjson.getBytes(StandardCharsets.UTF_8));
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+
+ List<String> columns = List.of("id", "name");
+ // default ignoreCase = false
+ JsonLinesToCsvConverter.convert(in, columns, out);
+
+ String result = out.toString(StandardCharsets.UTF_8);
+ String[] lines = result.split("\n");
+ assertEquals("10,Bob", lines[0].trim());
+ // Second line has "Id" but not "id", so "id" column is null (empty)
+ assertEquals(",Charlie", lines[1].trim());
+ }
+
+ @Test
+ void convert_prettyPrintedMultilineJson() throws IOException {
+ String prettyJson = "{\n \"id\": 1,\n \"name\": \"Alice\"\n}\n"
+ + "{\n \"id\": 2,\n \"name\": \"Bob\"\n}\n";
+
+ ByteArrayInputStream in = new ByteArrayInputStream(prettyJson.getBytes(StandardCharsets.UTF_8));
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+
+ List<String> columns = List.of("id", "name");
+ JsonLinesToCsvConverter.convert(in, columns, out);
+
+ String result = out.toString(StandardCharsets.UTF_8);
+ String[] lines = result.split("\n");
+ assertEquals("1,Alice", lines[0].trim());
+ assertEquals("2,Bob", lines[1].trim());
+ }
+
+ @Test
+ void convert_nonObjectRoot_throwsException() {
+ String jsonArray = "[{\"id\": 1, \"name\": \"Alice\"}]\n";
+ ByteArrayInputStream in = new ByteArrayInputStream(jsonArray.getBytes(StandardCharsets.UTF_8));
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+
+ List<String> columns = List.of("id", "name");
+ S3CopySimulator.S3TransferException ex = assertThrows(
+ S3CopySimulator.S3TransferException.class,
+ () -> JsonLinesToCsvConverter.convert(in, columns, out));
+
+ assertTrue(ex.getMessage().contains("JSON value is not an object"));
+ }
+
+ @Test
+ void convert_nestedJsonSerializedToString() throws IOException {
+ String ndjson = "{\"id\": 10, \"payload\": {\"subKey\": \"val\", \"count\": 5}}\n";
+ ByteArrayInputStream in = new ByteArrayInputStream(ndjson.getBytes(StandardCharsets.UTF_8));
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+
+ List<String> columns = List.of("id", "payload");
+ JsonLinesToCsvConverter.convert(in, columns, out);
+
+ String result = out.toString(StandardCharsets.UTF_8);
+ assertEquals("10,\"{\"\"subKey\"\":\"\"val\"\",\"\"count\"\":5}\"", result.trim());
+ }
+
+ @Test
+ void convert_booleanAndNumericFormats() throws IOException {
+ String ndjson = "{\"flag\": true, \"val\": 12.345, \"zero\": 0}\n";
+ ByteArrayInputStream in = new ByteArrayInputStream(ndjson.getBytes(StandardCharsets.UTF_8));
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+
+ List<String> columns = List.of("flag", "val", "zero");
+ JsonLinesToCsvConverter.convert(in, columns, out);
+
+ String result = out.toString(StandardCharsets.UTF_8);
+ assertEquals("true,12.345,0", result.trim());
+ }
+
+ @Test
+ void convert_malformedJson_includesLinePreviewInException() {
+ String badNdjson = "{\"id\": 1, \"name\": invalid_json}\n";
+ ByteArrayInputStream in = new ByteArrayInputStream(badNdjson.getBytes(StandardCharsets.UTF_8));
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+
+ List<String> columns = List.of("id", "name");
+ S3CopySimulator.S3TransferException ex = assertThrows(
+ S3CopySimulator.S3TransferException.class,
+ () -> JsonLinesToCsvConverter.convert(in, columns, out));
+
+ assertTrue(ex.getMessage().contains("invalid_json"));
+ }
+
+ @Test
+ void convert_emptyOrNullColumns_throwsException() {
+ String ndjson = "{\"id\": 1}\n";
+ ByteArrayInputStream in = new ByteArrayInputStream(ndjson.getBytes(StandardCharsets.UTF_8));
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+
+ assertThrows(
+ S3CopySimulator.S3TransferException.class,
+ () -> JsonLinesToCsvConverter.convert(in, List.of(), out));
+ assertThrows(
+ S3CopySimulator.S3TransferException.class,
+ () -> JsonLinesToCsvConverter.convert(in, null, out));
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/ManifestReaderTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/ManifestReaderTest.java
new file mode 100644
index 0000000000..7f28259a00
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/ManifestReaderTest.java
@@ -0,0 +1,78 @@
+package io.github.hectorvent.floci.services.redshift.proxy;
+
+import io.github.hectorvent.floci.services.s3.S3Service;
+import io.github.hectorvent.floci.services.s3.model.S3Object;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.when;
+
+class ManifestReaderTest {
+
+ @Test
+ void parseManifest_validEntries() {
+ S3Service s3 = Mockito.mock(S3Service.class);
+ String manifestJson = "{\"entries\": ["
+ + "{\"url\": \"s3://mybucket/data/file1.csv\", \"mandatory\": true},"
+ + "{\"url\": \"s3://mybucket/data/file2.csv\", \"mandatory\": false}"
+ + "]}";
+ S3Object obj = new S3Object();
+ obj.setData(manifestJson.getBytes(StandardCharsets.UTF_8));
+ when(s3.getObject("mybucket", "manifest.json")).thenReturn(obj);
+ when(s3.objectExists("mybucket", "data/file1.csv")).thenReturn(true);
+ when(s3.objectExists("mybucket", "data/file2.csv")).thenReturn(false);
+
+ List<String> keys = ManifestReader.resolveManifestKeys("mybucket", "manifest.json", s3);
+ assertEquals(List.of("data/file1.csv"), keys);
+ }
+
+ @Test
+ void parseManifest_missingMandatory_throwsException() {
+ S3Service s3 = Mockito.mock(S3Service.class);
+ String manifestJson = "{\"entries\": ["
+ + "{\"url\": \"s3://mybucket/data/missing.csv\", \"mandatory\": true}"
+ + "]}";
+ S3Object obj = new S3Object();
+ obj.setData(manifestJson.getBytes(StandardCharsets.UTF_8));
+ when(s3.getObject("mybucket", "manifest.json")).thenReturn(obj);
+ when(s3.objectExists("mybucket", "data/missing.csv")).thenReturn(false);
+
+ S3CopySimulator.S3TransferException ex = assertThrows(S3CopySimulator.S3TransferException.class, () ->
+ ManifestReader.resolveManifestKeys("mybucket", "manifest.json", s3));
+ assertTrue(ex.getMessage().contains("missing.csv"));
+ }
+
+ @Test
+ void parseManifest_malformedJson_throwsException() {
+ S3Service s3 = Mockito.mock(S3Service.class);
+ S3Object obj = new S3Object();
+ obj.setData("not valid json".getBytes(StandardCharsets.UTF_8));
+ when(s3.getObject("mybucket", "manifest.json")).thenReturn(obj);
+
+ assertThrows(S3CopySimulator.S3TransferException.class, () ->
+ ManifestReader.resolveManifestKeys("mybucket", "manifest.json", s3));
+ }
+
+ @Test
+ void parseManifest_missingMandatoryFlagDefaultsToFalse_skipsMissingFile() {
+ S3Service s3 = Mockito.mock(S3Service.class);
+ String manifestJson = "{\"entries\": ["
+ + "{\"url\": \"s3://mybucket/data/file1.csv\"},"
+ + "{\"url\": \"s3://mybucket/data/missing.csv\"}"
+ + "]}";
+ S3Object obj = new S3Object();
+ obj.setData(manifestJson.getBytes(StandardCharsets.UTF_8));
+ when(s3.getObject("mybucket", "manifest.json")).thenReturn(obj);
+ when(s3.objectExists("mybucket", "data/file1.csv")).thenReturn(true);
+ when(s3.objectExists("mybucket", "data/missing.csv")).thenReturn(false);
+
+ List<String> keys = ManifestReader.resolveManifestKeys("mybucket", "manifest.json", s3);
+ assertEquals(List.of("data/file1.csv"), keys);
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java
index 9aad9aca8f..d87eb18d73 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java
@@ -1327,4 +1327,292 @@ class S3CopySimulatorTest {
assertNull(in.nextMessage(), "Client must not receive an unconfirmed ReadyForQuery");
}
+ @Test
+ void prepareCopy_withManifest_resolvesManifestKeys() {
+ String manifestJson = "{\"entries\": [{\"url\": \"s3://wh/data1.csv\", \"mandatory\": true}]}";
+ S3Object manifestObj = new S3Object("wh", "manifest.json", manifestJson.getBytes(StandardCharsets.UTF_8), "application/json");
+ when(s3.getObject("wh", "manifest.json")).thenReturn(manifestObj);
+ when(s3.objectExists("wh", "data1.csv")).thenReturn(true);
+
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "t", List.of(), "wh", "manifest.json", ",", 0, false, true, null, null, false, true);
+
+ S3CopySimulator.CopyInput input = S3CopySimulator.prepareCopy(spec, s3, null);
+ assertEquals(List.of("data1.csv"), input.keys());
+ }
+
+ @Test
+ void copyBackendSql_and_stream_withJsonAuto() throws Exception {
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "users", List.of("id", "name"), "wh", "users.json", null, 0, false, false, null, null, true, false);
+
+ assertEquals("COPY users (id, name) FROM STDIN WITH (FORMAT csv, DELIMITER ',')",
+ S3CopySimulator.copyBackendSql(spec));
+
+ String ndjson = "{\"id\": 1, \"name\": \"Alice\"}\n{\"id\": 2, \"name\": \"Bob\"}\n";
+ S3Object dataObj = new S3Object("wh", "users.json", ndjson.getBytes(StandardCharsets.UTF_8), "application/json");
+ when(s3.getObject("wh", "users.json")).thenReturn(dataObj);
+
+ S3CopySimulator.CopyInput input = new S3CopySimulator.CopyInput(spec, List.of("users.json"), s3, null, null);
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ S3CopySimulator.streamCopyInput(input, out);
+
+ byte[] rawBytes = out.toByteArray();
+ // The output is framed as Postgres 'd' CopyData messages
+ assertTrue(rawBytes.length > 0);
+ assertEquals('d', rawBytes[0]);
+ }
+
+ @Test
+ void copyBackendSql_quotesDiscoveredColumnIdentifiers() {
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "orders", List.of(), "wh", "orders.csv", ",", 0, false, true, null, null);
+ assertEquals("COPY orders (\"order\", \"user\", \"col\"\"name\") FROM STDIN WITH (FORMAT csv, DELIMITER ',')",
+ S3CopySimulator.copyBackendSql(spec, List.of("order", "user", "col\"name")));
+ }
+
+ @Test
+ void copyBackendSql_passesUserSuppliedColumnListThroughAsWritten() {
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "orders", List.of("\"col one\"", "MixedCase", "\"User\""), "wh", "orders.csv", ",", 0, false, true, null, null);
+ assertEquals("COPY orders (\"col one\", MixedCase, \"User\") FROM STDIN WITH (FORMAT csv, DELIMITER ',')",
+ S3CopySimulator.copyBackendSql(spec));
+ }
+
+ @Test
+ void copyBackendSql_withJsonAuto_forcesCommaDelimiter() {
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "users", List.of("id"), "wh", "users.json", "\t", 0, false, false, null, null, true, false);
+
+ assertEquals("COPY users (id) FROM STDIN WITH (FORMAT csv, DELIMITER ',')",
+ S3CopySimulator.copyBackendSql(spec));
+ }
+
+ @Test
+ void streamCopyInput_withJsonAuto_unquotesQuotedColumnIdentifiers() throws Exception {
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "users", List.of("\"col one\"", "Id"), "wh", "users.json", null, 0, false, false, null, null, true, false);
+
+ String ndjson = "{\"col one\": \"Alice\", \"Id\": 10}\n";
+ S3Object dataObj = new S3Object("wh", "users.json", ndjson.getBytes(StandardCharsets.UTF_8), "application/json");
+ when(s3.getObject("wh", "users.json")).thenReturn(dataObj);
+
+ S3CopySimulator.CopyInput input = new S3CopySimulator.CopyInput(spec, List.of("users.json"), s3, null, null);
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ S3CopySimulator.streamCopyInput(input, out);
+
+ byte[] rawBytes = out.toByteArray();
+ assertTrue(rawBytes.length > 5);
+ assertEquals('d', rawBytes[0]);
+ String payload = new String(rawBytes, 5, rawBytes.length - 5, StandardCharsets.UTF_8);
+ assertEquals("Alice,10\n", payload);
+ }
+
+ @Test
+ void streamCopyInput_withJsonAutoAndGzip() throws Exception {
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "users", List.of("id", "name"), "wh", "users.json.gz", null, 0, true, false, null, null, true, false);
+
+ String ndjson = "{\"id\": 1, \"name\": \"Alice\"}\n";
+ ByteArrayOutputStream gzippedOut = new ByteArrayOutputStream();
+ try (java.util.zip.GZIPOutputStream gzip = new java.util.zip.GZIPOutputStream(gzippedOut)) {
+ gzip.write(ndjson.getBytes(StandardCharsets.UTF_8));
+ }
+
+ S3Object dataObj = new S3Object("wh", "users.json.gz", gzippedOut.toByteArray(), "application/gzip");
+ when(s3.getObject("wh", "users.json.gz")).thenReturn(dataObj);
+
+ S3CopySimulator.CopyInput input = new S3CopySimulator.CopyInput(spec, List.of("users.json.gz"), s3, null, null);
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ S3CopySimulator.streamCopyInput(input, out);
+
+ byte[] rawBytes = out.toByteArray();
+ assertTrue(rawBytes.length > 0);
+ assertEquals('d', rawBytes[0]);
+ }
+
+ @Test
+ void runCopyFrom_withJsonAutoWithoutColumns_discoversSchemaAndCopies() throws Exception {
+ String ndjson = "{\"id\": 10, \"name\": \"Alice\"}\n";
+ S3Object dataObj = new S3Object("wh", "data.json", ndjson.getBytes(StandardCharsets.UTF_8), "application/json");
+ when(s3.objectExists("wh", "data.json")).thenReturn(true);
+ when(s3.getObject("wh", "data.json")).thenReturn(dataObj);
+
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "public.items", List.of(), "wh", "data.json", null, 0, false, false, null, null, true, false);
+
+ Thread backend = backendThread(() -> {
+ PostgresWireDecoder in = new PostgresWireDecoder(testBackend.getInputStream());
+ OutputStream out = testBackend.getOutputStream();
+
+ PostgresWireDecoder.FrontendMessage q1 = in.nextMessage();
+ assertEquals('Q', q1.type());
+ assertTrue(q1.getSql().contains("pg_catalog.pg_attribute"));
+ assertTrue(q1.getSql().contains("items"));
+
+ writeSingleColumnDataRow(out, "id");
+ writeSingleColumnDataRow(out, "name");
+ writeCommandComplete(out, "SELECT 2");
+ writeReadyForQuery(out, 'I');
+
+ PostgresWireDecoder.FrontendMessage q2 = in.nextMessage();
+ assertEquals('Q', q2.type());
+ assertEquals("COPY public.items (\"id\", \"name\") FROM STDIN WITH (FORMAT csv, DELIMITER ',')", q2.getSql());
+
+ out.write(new byte[]{'G', 0, 0, 0, 4});
+ out.flush();
+
+ PostgresWireDecoder.FrontendMessage d = in.nextMessage();
+ assertEquals('d', d.type());
+
+ PostgresWireDecoder.FrontendMessage c = in.nextMessage();
+ assertEquals('c', c.type());
+
+ writeCommandComplete(out, "COPY 1");
+ writeReadyForQuery(out, 'I');
+ });
+
+ boolean handled = S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, null, 'I');
+ joinBackend(backend);
+ assertTrue(handled);
+
+ PostgresWireDecoder clientIn = new PostgresWireDecoder(testClient.getInputStream());
+ PostgresWireDecoder.FrontendMessage msg1 = clientIn.nextMessage();
+ assertEquals('C', msg1.type());
+ PostgresWireDecoder.FrontendMessage msg2 = clientIn.nextMessage();
+ assertEquals('Z', msg2.type());
+ }
+
+ @Test
+ void runCopyFrom_withQuotedAndMixedCaseColumns_passesThroughToBackendAndMatchesJsonKeys() throws Exception {
+ String ndjson = "{\"col one\": \"Alice\", \"MixedCase\": 10}\n";
+ S3Object dataObj = new S3Object("wh", "users.json", ndjson.getBytes(StandardCharsets.UTF_8), "application/json");
+ when(s3.objectExists("wh", "users.json")).thenReturn(true);
+ when(s3.getObject("wh", "users.json")).thenReturn(dataObj);
+
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "public.users", List.of("\"col one\"", "MixedCase"), "wh", "users.json", null, 0, false, false, null, null, true, false);
+
+ ByteArrayOutputStream streamedData = new ByteArrayOutputStream();
+ Thread backend = backendThread(() -> {
+ PostgresWireDecoder in = new PostgresWireDecoder(testBackend.getInputStream());
+ OutputStream out = testBackend.getOutputStream();
+
+ PostgresWireDecoder.FrontendMessage q = in.nextMessage();
+ assertEquals('Q', q.type());
+ assertEquals("COPY public.users (\"col one\", MixedCase) FROM STDIN WITH (FORMAT csv, DELIMITER ',')", q.getSql());
+
+ out.write(new byte[]{'G', 0, 0, 0, 4});
+ out.flush();
+
+ while (true) {
+ PostgresWireDecoder.FrontendMessage m = in.nextMessage();
+ if (m == null || m.type() == 'c') {
+ break;
+ }
+ if (m.type() == 'd') {
+ streamedData.write(m.body());
+ }
+ }
+
+ writeCommandComplete(out, "COPY 1");
+ writeReadyForQuery(out, 'I');
+ });
+
+ boolean handled = S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, null, 'I');
+ joinBackend(backend);
+ assertTrue(handled);
+
+ assertEquals("Alice,10\n", streamedData.toString(StandardCharsets.UTF_8));
+
+ PostgresWireDecoder clientIn = new PostgresWireDecoder(testClient.getInputStream());
+ assertEquals('C', clientIn.nextMessage().type());
+ assertEquals('Z', clientIn.nextMessage().type());
+ }
+
+ @Test
+ void runCopyFrom_withJsonAuto_releasesIamSessionEvenWhenCatalogDiscoveryFails() throws Exception {
+ when(s3.objectExists("wh", "data.json")).thenReturn(true);
+ IamService iamService = mock(IamService.class);
+ IamRole role = mock(IamRole.class);
+ when(role.getAssumeRolePolicyDocument()).thenReturn(
+ "{\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"redshift.amazonaws.com\"}}]}");
+ when(iamService.findRole("000000000000", "Role")).thenReturn(java.util.Optional.of(role));
+
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "missing_table", List.of(), "wh", "data.json", null, 0, false, false, null,
+ "arn:aws:iam::000000000000:role/Role", true, false);
+
+ Thread backend = backendThread(() -> {
+ PostgresWireDecoder in = new PostgresWireDecoder(testBackend.getInputStream());
+ OutputStream out = testBackend.getOutputStream();
+
+ PostgresWireDecoder.FrontendMessage q1 = in.nextMessage();
+ assertEquals('Q', q1.type());
+ writeCommandComplete(out, "SELECT 0");
+ writeReadyForQuery(out, 'I');
+ });
+
+ boolean handled = S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, iamService, 'I');
+ joinBackend(backend);
+ assertTrue(handled);
+
+ verify(iamService, times(1)).registerSessionForAccount(eq("000000000000"), any(), any(), any(), any(), any(), any());
+ verify(iamService, times(1)).unregisterSession(eq("000000000000"), any());
+ }
+
+ @Test
+ void runCopyFrom_withJsonAutoWithoutColumns_whenTableNotFound_sendsError() throws Exception {
+ when(s3.objectExists("wh", "data.json")).thenReturn(true);
+
+ CopyStatementParser.S3CopyFrom spec = new CopyStatementParser.S3CopyFrom(
+ "missing_table", List.of(), "wh", "data.json", null, 0, false, false, null, null, true, false);
+
+ Thread backend = backendThread(() -> {
+ PostgresWireDecoder in = new PostgresWireDecoder(testBackend.getInputStream());
+ OutputStream out = testBackend.getOutputStream();
+
+ PostgresWireDecoder.FrontendMessage q1 = in.nextMessage();
+ assertEquals('Q', q1.type());
+ assertTrue(q1.getSql().contains("missing_table"));
+
+ writeCommandComplete(out, "SELECT 0");
+ writeReadyForQuery(out, 'I');
+ });
+
+ boolean handled = S3CopySimulator.runCopyFrom(simClient, simBackend, spec, s3, null, 'I');
+ joinBackend(backend);
+ assertTrue(handled);
+
+ PostgresWireDecoder clientIn = new PostgresWireDecoder(testClient.getInputStream());
+ PostgresWireDecoder.FrontendMessage err = clientIn.nextMessage();
+ assertEquals('E', err.type());
+ PostgresWireDecoder.FrontendMessage ready = clientIn.nextMessage();
+ assertEquals('Z', ready.type());
+ }
+
+ private static void writeSingleColumnDataRow(OutputStream out, String val) throws IOException {
+ byte[] b = val.getBytes(StandardCharsets.UTF_8);
+ int len = 4 + 2 + 4 + b.length;
+ out.write('D');
+ out.write(intBytes(len));
+ out.write(new byte[]{0, 1});
+ out.write(intBytes(b.length));
+ out.write(b);
+ out.flush();
+ }
+
+ private static void writeCommandComplete(OutputStream out, String tag) throws IOException {
+ byte[] b = (tag + "\0").getBytes(StandardCharsets.US_ASCII);
+ out.write('C');
+ out.write(intBytes(4 + b.length));
+ out.write(b);
+ out.flush();
+ }
+
+ private static void writeReadyForQuery(OutputStream out, char status) throws IOException {
+ out.write(new byte[]{'Z', 0, 0, 0, 5, (byte) status});
+ out.flush();
+ }
}
+