Light, fluffy, and always free - The AWS Local Emulator alternative
feat(redshift): seed catalog and system views (#3552)
Vũ Thế Huyd9abefb83fd5
diff --git a/docs/services/redshift.md b/docs/services/redshift.mdindex 9e7b94c999..724863367b 100644--- a/docs/services/redshift.md+++ b/docs/services/redshift.md@@ -255,6 +255,31 @@ the result to S3 as one or more objects under `<prefix>`.- Extended Query UNLOAD is supported when the complete statement is present in `Parse` and has nobind parameters. Parameterized statements are forwarded unchanged.+## Catalog Views++When a Redshift cluster container starts, Floci bootstraps common Redshift system and catalog views into both `template1` (ensuring any future `CREATE DATABASE` inherits them automatically) and the active cluster database (`dev`). This ensures BI tools (Tableau, Looker, DBeaver), ORMs, and migration tools (Flyway, Liquibase, dbt) can introspect database schema metadata without missing-relation errors:++- `pg_table_def`: Table and column metadata (`schemaname`, `tablename`, `column`, `type`, `encoding`, `distkey`, `sortkey`, `notnull`).+- `svv_table_info`: Table-level summary metadata (`database`, `schema`, `table_id`, `table`, `encoded`, `diststyle`, `sortkey1`, `max_varchar`, `tbl_rows`, `size`).+- `svv_all_columns`: All columns across database schemas (`database_name`, `schema_name`, `table_name`, `column_name`, `data_type`, `is_nullable`).+- `svv_columns`: Column catalog list (`table_catalog`, `table_schema`, `table_name`, `column_name`, `ordinal_position`, `column_default`, `is_nullable`, `data_type`).+- `svv_tables`: Table catalog list (`table_catalog`, `table_schema`, `table_name`, `table_type`).+- `stv_tbl_perm`: Table persistence metadata (`id`, `name`, `db_id`, `temp`, `backup`).+- `stl_load_errors`: Table exposing the documented Redshift load-error schema for catalog and tooling compatibility.+- `svl_qlog`: Query execution log view (`userid`, `query`, `xid`, `pid`, `starttime`, `endtime`, `elapsed`, `aborted`, `label`).+- `pg_user_info`: User catalog information (`usesysid`, `usename`, `usecreatedb`, `usesuper`, `useconnlimit`, `syslogaccess`).+- `svl_user_info`: Standard Redshift user information view matching AWS documented columns.+- `pg_database_info`: Database catalog information (`datid`, `datname`, `datdba`, `encoding`, `datconnlimit`).+- `stv_sessions`: Active database sessions (`process`, `user_name`, `db_name`, `starttime`, `timeout_sec`).+- `stv_recents`: Recently executed queries (`userid`, `pid`, `process`, `query`, `starttime`, `duration`, `status`).+- `svv_transactions`: Current transaction status (`txn_owner`, `txn_db`, `xid`, `pid`, `txn_start`, `lock_mode`, `relation`, `granted`).+- `stv_slices`: Cluster slice metadata (`node`, `slice`, `localslice`, `type`).+- `stl_query`: Dynamic query execution log view mapped from `pg_stat_activity` (`query`, `xid`, `pid`, `userid`, `starttime`, `endtime`, `elapsed`, `querytxt`, `database`, `aborted`, `insert_pristine`, `concurrency_scaling_status`).+- `stv_wlm_query_state`: Dynamic WLM query state view (`xid`, `task`, `query`, `service_class`, `slot_count`, `wlm_start_time`, `queue_time`, `exec_time`, `state`, `query_priority`).+- `svv_diskusage`: Disk space usage summary per relation exposing full documented Redshift block layout columns (`db_id`, `name`, `slice`, `col`, `tbl`, `blocknum`, `num_values`, `minvalue`, `maxvalue`, `sb_pos`, `pinned`, `on_disk`, `modified`, `hdr_modified`, `unsorted`, `tombstone`, `preferred_diskno`, `temporary`, `newblock`) as well as compatibility aliases (`database`, `schema`, `table_id`, `size`, `used`).++These views expose the documented Redshift column names, types, and ordering mapped from PostgreSQL internal catalogs (`pg_catalog`, `information_schema`, `pg_stat_activity`), with deterministic placeholders where PostgreSQL cannot provide multi-node metrics.+## Out of Scope- Real Redshift SQL semantics: the data plane is stock PostgreSQL. Redshift-only table DDL keywords (DISTSTYLE / DISTKEY / SORTKEY / ENCODE) are stripped so CREATE TABLE / ALTER TABLE executes (see [SQL Interceptor](#sql-interceptor)), but the distribution/sort behavior they request is not; SUPER/SPECTRUM are not emulated.diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/container/RedshiftContainerManager.java b/src/main/java/io/github/hectorvent/floci/services/redshift/container/RedshiftContainerManager.javaindex eb8b81ad04..08012a458c 100644--- a/src/main/java/io/github/hectorvent/floci/services/redshift/container/RedshiftContainerManager.java+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/container/RedshiftContainerManager.java@@ -75,6 +75,7 @@ public class RedshiftContainerManager {ContainerBuilder.Builder specBuilder = containerBuilder.newContainer(image).withName(containerName).withEnv(envVars)+ .withCmd(List.of("postgres", "-c", "allow_system_table_mods=on")).withDockerNetwork(config.services().redshift().dockerNetwork()).withLogRotation();@@ -140,6 +141,7 @@ public class RedshiftContainerManager {}waitForReady(containerName, info.containerId(), masterUsername, "dev");+ bootstrapCatalog(info.containerId(), masterUsername, "dev");containers.put(containerKey(accountId, clusterIdentifier), handle);return handle;@@ -415,5 +417,48 @@ public class RedshiftContainerManager {throw new IllegalStateException("Timed out initializing " + description + " in " + containerName + ": " + lastOutput);}+ /**+ * Bootstraps Redshift catalog and system views in the dev database so BI+ * and migration tooling can inspect metadata without relation-does-not-exist errors.+ */+ void bootstrapCatalog(String containerId, String username, String dbName) {+ String effectiveUser = (username != null && !username.isBlank()) ? username : "postgres";+ String effectiveDb = (dbName != null && !dbName.isBlank()) ? dbName : "dev";+ try (InputStream in = getClass().getResourceAsStream("/redshift/bootstrap-catalog.sql")) {+ if (in == null) {+ LOG.warnv("Redshift bootstrap-catalog.sql not found on classpath; skipping catalog seed for container {0}", containerId);+ return;+ }+ byte[] sqlBytes = in.readAllBytes();+ byte[] tarBytes = buildSingleFileTar("bootstrap-catalog.sql", sqlBytes, 0644);+ lifecycleManager.getDockerClient().copyArchiveToContainerCmd(containerId)+ .withTarInputStream(new ByteArrayInputStream(tarBytes))+ .withRemotePath("/tmp")+ .exec();++ List<String> targetDbs = effectiveDb.equals("template1")+ ? List.of("template1")+ : List.of("template1", effectiveDb);+ for (String db : targetDbs) {+ String[] cmd = new String[]{+ "psql",+ "-h", "127.0.0.1",+ "-v", "ON_ERROR_STOP=1",+ "-U", effectiveUser,+ "-d", db,+ "-f", "/tmp/bootstrap-catalog.sql"+ };+ ExecResult result = execInContainer(containerId, cmd, 30);+ if (result.exitCode() != 0) {+ LOG.warnv("Redshift catalog bootstrap for {0} exited with code {1}: {2}", db, result.exitCode(), result.stderr());+ } else {+ LOG.infov("Redshift catalog views bootstrapped successfully for {0} in container {1}", db, containerId);+ }+ }+ } catch (Exception e) {+ LOG.warnv(e, "Error bootstrapping Redshift catalog views for container {0}", containerId);+ }+ }+public record ExecResult(long exitCode, String stdout, String stderr) {}}diff --git a/src/main/resources/redshift/bootstrap-catalog.sql b/src/main/resources/redshift/bootstrap-catalog.sqlnew file mode 100644index 0000000000..fbeaab20d7--- /dev/null+++ b/src/main/resources/redshift/bootstrap-catalog.sql@@ -0,0 +1,364 @@+-- Bootstrap Redshift catalog and system views for PostgreSQL 15+-- Provides metadata compatibility for BI and migration tooling (Flyway, Liquibase, dbt, DBeaver, Tableau)++-- 1. pg_table_def: inspect column definitions across tables+DROP VIEW IF EXISTS pg_catalog.pg_table_def CASCADE;+CREATE OR REPLACE VIEW pg_catalog.pg_table_def AS+SELECT+ n.nspname::name AS schemaname,+ c.relname::name AS tablename,+ a.attname::name AS "column",+ format_type(a.atttypid, a.atttypmod)::character varying(256) AS type,+ 'none'::character varying(32) AS encoding,+ false::boolean AS distkey,+ 0::integer AS sortkey,+ a.attnotnull::boolean AS notnull+FROM pg_catalog.pg_attribute a+JOIN pg_catalog.pg_class c ON a.attrelid = c.oid+JOIN pg_catalog.pg_namespace n ON c.relnamespace = n.oid+WHERE a.attnum > 0+ AND NOT a.attisdropped+ AND c.relkind IN ('r', 'v', 'm', 'p')+ AND n.nspname NOT IN ('pg_toast');++-- 2. svv_table_info: table-level metadata and storage layout summary+DROP VIEW IF EXISTS pg_catalog.svv_table_info CASCADE;+CREATE OR REPLACE VIEW pg_catalog.svv_table_info AS+SELECT+ current_database()::name AS "database",+ n.nspname::name AS "schema",+ c.oid::integer AS table_id,+ c.relname::name AS "table",+ 'N'::character varying(1) AS encoded,+ 'EVEN'::character varying(20) AS diststyle,+ NULL::character varying(128) AS sortkey1,+ 0::integer AS max_varchar,+ 'none'::character varying(32) AS sortkey1_enc,+ 0::integer AS sortkey_num,+ 1::bigint AS size,+ 0.0000::numeric(10,4) AS pct_used,+ 0::bigint AS empty,+ 0.00::numeric(5,2) AS unsorted,+ 0.00::numeric(5,2) AS stats_off,+ COALESCE(c.reltuples::bigint, 0::bigint)::numeric(38,0) AS tbl_rows,+ 1.00::numeric(19,2) AS skew_sortkey1,+ 1.00::numeric(19,2) AS skew_rows,+ COALESCE(c.reltuples::bigint, 0::bigint)::numeric(38,0) AS estimated_visible_rows,+ NULL::text AS risk_event,+ 0.00::numeric(12,2) AS vacuum_sort_benefit,+ now()::timestamp without time zone AS create_time+FROM pg_catalog.pg_class c+JOIN pg_catalog.pg_namespace n ON c.relnamespace = n.oid+WHERE c.relkind = 'r'+ AND n.nspname NOT IN ('pg_catalog', 'information_schema', 'pg_toast');++-- 3. svv_all_columns: all columns across database and schemas+DROP VIEW IF EXISTS pg_catalog.svv_all_columns CASCADE;+CREATE OR REPLACE VIEW pg_catalog.svv_all_columns AS+SELECT+ current_database()::character varying(128) AS database_name,+ table_schema::character varying(128) AS schema_name,+ table_name::character varying(128) AS table_name,+ column_name::character varying(128) AS column_name,+ ordinal_position::integer AS ordinal_position,+ column_default::character varying(4000) AS column_default,+ is_nullable::character varying(3) AS is_nullable,+ data_type::character varying(128) AS data_type,+ character_maximum_length::integer AS character_maximum_length,+ numeric_precision::integer AS numeric_precision,+ numeric_scale::integer AS numeric_scale,+ NULL::character varying(256) AS remarks+FROM information_schema.columns;++-- 4. svv_tables: table catalog list+DROP VIEW IF EXISTS pg_catalog.svv_tables CASCADE;+CREATE OR REPLACE VIEW pg_catalog.svv_tables AS+SELECT+ table_catalog::text AS table_catalog,+ table_schema::text AS table_schema,+ table_name::text AS table_name,+ table_type::text AS table_type,+ NULL::text AS remarks+FROM information_schema.tables;++-- 5. stv_tbl_perm: physical table persistence information+DROP VIEW IF EXISTS pg_catalog.stv_tbl_perm CASCADE;+CREATE OR REPLACE VIEW pg_catalog.stv_tbl_perm AS+SELECT+ 0::integer AS slice,+ c.oid::integer AS id,+ c.relname::character varying(72) AS name,+ COALESCE(c.reltuples::bigint, 0::bigint) AS rows,+ COALESCE(c.reltuples::bigint, 0::bigint) AS sorted_rows,+ CASE WHEN c.relpersistence = 't' THEN 1 ELSE 0 END::integer AS temp,+ d.oid::integer AS db_id,+ 0::integer AS insert_pristine,+ 0::integer AS delete_pristine,+ 1::integer AS backup,+ 0::integer AS dist_style,+ (pg_relation_size(c.oid) / 1048576)::integer AS block_count,+ 0::bigint AS insert_prn,+ 0::bigint AS delete_prn+FROM pg_catalog.pg_class c+CROSS JOIN (SELECT oid FROM pg_catalog.pg_database WHERE datname = current_database()) d+WHERE c.relkind = 'r';++-- 6. stl_load_errors: load error log table for COPY commands+CREATE TABLE IF NOT EXISTS pg_catalog.stl_load_errors (+ userid integer DEFAULT 1,+ slice integer DEFAULT 0,+ tbl integer DEFAULT 0,+ starttime timestamp without time zone DEFAULT now(),+ session integer DEFAULT 0,+ query integer DEFAULT 0,+ filename character varying(256),+ line_number bigint,+ colname character varying(127),+ type character varying(10),+ col_length character varying(10),+ position integer,+ raw_line character varying(1024),+ raw_field_value character varying(1024),+ err_code integer,+ err_reason character varying(100),+ is_partial integer DEFAULT 0,+ start_offset bigint DEFAULT 0,+ copy_job_id bigint DEFAULT 0+);++-- 7. svl_qlog: dynamic query execution log view+DROP VIEW IF EXISTS pg_catalog.svl_qlog CASCADE;+CREATE OR REPLACE VIEW pg_catalog.svl_qlog AS+SELECT+ a.usesysid::integer AS userid,+ a.pid AS query,+ 0::bigint AS xid,+ a.pid AS pid,+ COALESCE(a.query_start, now()) AS starttime,+ COALESCE(a.state_change, now()) AS endtime,+ COALESCE(EXTRACT(EPOCH FROM (COALESCE(a.state_change, now()) - a.query_start))::bigint * 1000000, 0::bigint) AS elapsed,+ 0::integer AS aborted,+ 0::integer AS insert_pristine,+ 0::integer AS concurrency_scaling_status,+ 0::integer AS source_query,+ 'default'::character varying(320) AS label,+ SUBSTRING(COALESCE(a.query, ''), 1, 60)::character varying(60) AS substring,+ 'Disabled'::text AS concurrency_scaling_status_txt,+ 0::integer AS from_sp_call,+ 0::integer AS insert_prn+FROM pg_catalog.pg_stat_activity a+WHERE a.query IS NOT NULL AND a.query != '';++-- 8. pg_user_info: user information view+DROP VIEW IF EXISTS pg_catalog.pg_user_info CASCADE;+CREATE OR REPLACE VIEW pg_catalog.pg_user_info AS+SELECT+ usename::name AS usename,+ usesysid::integer AS usesysid,+ usecreatedb::boolean AS usecreatedb,+ usesuper::boolean AS usesuper,+ false::boolean AS usecatupd,+ valuntil::timestamp without time zone AS valuntil,+ NULL::text AS useconfig,+ '-1'::text AS useconnlimit,+ 'UNRESTRICTED'::text AS syslogaccess,+ NULL::timestamp without time zone AS last_ddl_ts,+ 0::integer AS sessiontimeout,+ NULL::text AS external_id+FROM pg_catalog.pg_user;++-- 8b. svl_user_info: standard Redshift user information view+DROP VIEW IF EXISTS pg_catalog.svl_user_info CASCADE;+CREATE OR REPLACE VIEW pg_catalog.svl_user_info AS+SELECT+ usename::text AS usename,+ usesysid::integer AS usesysid,+ usecreatedb::boolean AS usecreatedb,+ usesuper::boolean AS usesuper,+ false::boolean AS usecatupd,+ '-1'::text AS useconnlimit,+ 'UNRESTRICTED'::text AS syslogaccess,+ NULL::timestamp without time zone AS last_ddl_ts,+ 0::integer AS sessiontimeout,+ NULL::text AS external_id+FROM pg_catalog.pg_user;++-- 9. stv_sessions: active database sessions+DROP VIEW IF EXISTS pg_catalog.stv_sessions CASCADE;+CREATE OR REPLACE VIEW pg_catalog.stv_sessions AS+SELECT+ backend_start AS starttime,+ pid AS process,+ usename::character varying(50) AS user_name,+ datname::character varying(50) AS db_name,+ 0::integer AS timeout_sec,+ COALESCE(client_addr::text, '127.0.0.1')::character varying(45) AS remotehost,+ COALESCE(client_port::text, '0')::character varying(10) AS remoteport+FROM pg_catalog.pg_stat_activity+WHERE datname IS NOT NULL;++-- 10. stv_recents: current and recent queries+DROP VIEW IF EXISTS pg_catalog.stv_recents CASCADE;+CREATE OR REPLACE VIEW pg_catalog.stv_recents AS+SELECT+ a.usesysid::integer AS userid,+ CASE WHEN a.state = 'active' THEN 'Running'::character varying(20) ELSE 'Completed'::character varying(20) END AS status,+ COALESCE(a.query_start, now()) AS starttime,+ COALESCE(EXTRACT(EPOCH FROM (now() - a.query_start))::integer * 1000000, 0) AS duration,+ a.usename::character varying(50) AS user_name,+ a.datname::character varying(50) AS db_name,+ COALESCE(a.query, '')::character varying(4000) AS query,+ a.pid AS pid,+ a.pid AS process,+ a.state_change AS endtime+FROM pg_catalog.pg_stat_activity a+WHERE a.datname IS NOT NULL;++-- 11. pg_database_info: database metadata catalog+DROP VIEW IF EXISTS pg_catalog.pg_database_info CASCADE;+CREATE OR REPLACE VIEW pg_catalog.pg_database_info AS+SELECT+ datname,+ oid::integer AS datid,+ datdba::integer AS datdba,+ encoding,+ datconnlimit::text AS datconnlimit,+ datistemplate,+ datallowconn,+ dattablespace::integer AS dattablespace+FROM pg_catalog.pg_database;++-- 12. svv_columns: user-accessible table columns (full AWS Redshift contract)+DROP VIEW IF EXISTS pg_catalog.svv_columns CASCADE;+CREATE OR REPLACE VIEW pg_catalog.svv_columns AS+SELECT+ table_catalog::text AS table_catalog,+ table_schema::text AS table_schema,+ table_name::text AS table_name,+ column_name::text AS column_name,+ ordinal_position::integer AS ordinal_position,+ column_default::text AS column_default,+ is_nullable::text AS is_nullable,+ data_type::text AS data_type,+ character_maximum_length::integer AS character_maximum_length,+ numeric_precision::integer AS numeric_precision,+ numeric_precision_radix::integer AS numeric_precision_radix,+ numeric_scale::integer AS numeric_scale,+ datetime_precision::integer AS datetime_precision,+ interval_type::text AS interval_type,+ interval_precision::text AS interval_precision,+ character_set_catalog::text AS character_set_catalog,+ character_set_schema::text AS character_set_schema,+ character_set_name::text AS character_set_name,+ collation_catalog::text AS collation_catalog,+ collation_schema::text AS collation_schema,+ collation_name::text AS collation_name,+ domain_name::text AS domain_name,+ NULL::text AS remarks,+ table_schema::text AS schema_name+FROM information_schema.columns+WHERE table_schema NOT IN ('pg_catalog', 'information_schema', 'pg_toast');++-- 13. svv_transactions: active transactions and locks+DROP VIEW IF EXISTS pg_catalog.svv_transactions CASCADE;+CREATE OR REPLACE VIEW pg_catalog.svv_transactions AS+SELECT+ a.usename::text AS txn_owner,+ a.datname::text AS txn_db,+ COALESCE(NULLIF(a.backend_xid::text, '')::bigint, 0::bigint) AS xid,+ a.pid AS pid,+ a.xact_start AS txn_start,+ l.mode::text AS lock_mode,+ 'relation'::text AS lockable_object_type,+ l.relation::integer AS relation,+ l.granted::boolean AS granted+FROM pg_catalog.pg_stat_activity a+JOIN pg_catalog.pg_locks l ON a.pid = l.pid+WHERE a.xact_start IS NOT NULL;++-- 14. stv_slices: cluster slice topology (single-node emulator)+DROP VIEW IF EXISTS pg_catalog.stv_slices CASCADE;+CREATE OR REPLACE VIEW pg_catalog.stv_slices AS+SELECT+ 0::integer AS node,+ 0::integer AS slice,+ 0::integer AS localslice,+ 'C'::character varying(1) AS type,+ 1::integer AS cpu;++-- 15. stl_query: completed and running query execution log view+DROP VIEW IF EXISTS pg_catalog.stl_query CASCADE;+CREATE OR REPLACE VIEW pg_catalog.stl_query AS+SELECT+ a.usesysid::integer AS userid,+ a.pid AS query,+ 'default'::character varying(320) AS label,+ 0::bigint AS xid,+ a.pid AS pid,+ COALESCE(a.datname, current_database())::character varying(32) AS database,+ COALESCE(a.query, '')::character varying(4000) AS querytxt,+ COALESCE(a.query_start, now()) AS starttime,+ COALESCE(a.state_change, now()) AS endtime,+ CASE WHEN a.state = 'active' THEN 0 ELSE 0 END::integer AS aborted,+ 0::integer AS insert_pristine,+ 0::integer AS concurrency_scaling_status,+ COALESCE(EXTRACT(EPOCH FROM (COALESCE(a.state_change, now()) - a.query_start))::bigint * 1000000, 0::bigint) AS elapsed+FROM pg_catalog.pg_stat_activity a+WHERE a.query IS NOT NULL AND a.query != '';++-- 16. stv_wlm_query_state: WLM query state view+DROP VIEW IF EXISTS pg_catalog.stv_wlm_query_state CASCADE;+CREATE OR REPLACE VIEW pg_catalog.stv_wlm_query_state AS+SELECT+ 0::bigint AS xid,+ 0::integer AS task,+ a.pid AS query,+ 1::integer AS service_class,+ 1::integer AS slot_count,+ COALESCE(a.query_start, now()) AS wlm_start_time,+ CASE WHEN a.state = 'active' THEN 'Running'::character varying(16) ELSE 'Returning'::character varying(16) END AS state,+ 0::bigint AS queue_time,+ COALESCE(EXTRACT(EPOCH FROM (now() - a.query_start))::bigint * 1000000, 0::bigint) AS exec_time,+ 'Normal'::character varying(20) AS query_priority,+ COALESCE(a.query_start, now()) AS service_class_start_time+FROM pg_catalog.pg_stat_activity a+WHERE a.state = 'active';++-- 17. svv_diskusage: disk space usage per relation (full Redshift columns + compatibility aliases)+DROP VIEW IF EXISTS pg_catalog.svv_diskusage CASCADE;+CREATE OR REPLACE VIEW pg_catalog.svv_diskusage AS+SELECT+ d.oid::integer AS db_id,+ c.relname::character varying(72) AS name,+ 0::integer AS slice,+ 0::integer AS col,+ c.oid::integer AS tbl,+ 0::integer AS blocknum,+ COALESCE(c.reltuples::integer, 0::integer) AS num_values,+ 0::bigint AS minvalue,+ 0::bigint AS maxvalue,+ 0::integer AS sb_pos,+ 0::integer AS pinned,+ 0::integer AS on_disk,+ 0::integer AS modified,+ 0::integer AS hdr_modified,+ 1::integer AS unsorted,+ 0::integer AS tombstone,+ 0::integer AS preferred_diskno,+ CASE WHEN c.relpersistence = 't' THEN 1 ELSE 0 END::integer AS temporary,+ 0::integer AS newblock,+ current_database()::name AS "database",+ n.nspname::name AS "schema",+ c.oid::integer AS table_id,+ (pg_total_relation_size(c.oid) / 1048576)::bigint AS size,+ (pg_relation_size(c.oid) / 1048576)::bigint AS used+FROM pg_catalog.pg_class c+JOIN pg_catalog.pg_namespace n ON c.relnamespace = n.oid+CROSS JOIN (SELECT oid FROM pg_catalog.pg_database WHERE datname = current_database()) d+WHERE c.relkind IN ('r', 'm')+ AND n.nspname NOT IN ('pg_catalog', 'information_schema', 'pg_toast');++-- Grant access to all database users (including IAM temporary users)+GRANT USAGE ON SCHEMA pg_catalog TO PUBLIC;+GRANT ALL ON TABLE pg_catalog.stl_load_errors TO PUBLIC;+GRANT SELECT ON ALL TABLES IN SCHEMA pg_catalog TO PUBLIC;diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftCatalogSeedIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftCatalogSeedIntegrationTest.javanew file mode 100644index 0000000000..0d2eb86a2e--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftCatalogSeedIntegrationTest.java@@ -0,0 +1,325 @@+package io.github.hectorvent.floci.services.redshift;++import io.github.hectorvent.floci.services.redshift.model.Cluster;+import io.quarkus.test.junit.QuarkusTest;+import jakarta.inject.Inject;+import org.awaitility.Awaitility;+import org.awaitility.core.ConditionTimeoutException;+import org.junit.jupiter.api.AfterEach;+import org.junit.jupiter.api.Assumptions;+import org.junit.jupiter.api.BeforeAll;+import org.junit.jupiter.api.Test;++import java.sql.Connection;+import java.sql.DriverManager;+import java.sql.PreparedStatement;+import java.sql.ResultSet;+import java.sql.SQLException;+import java.sql.Statement;+import java.time.Duration;+import java.util.Objects;+import java.util.UUID;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertNotNull;+import static org.junit.jupiter.api.Assertions.assertTrue;++@QuarkusTest+class RedshiftCatalogSeedIntegrationTest {++ @Inject+ RedshiftService service;++ private String clusterId;++ @BeforeAll+ static void requireDocker() {+ Assumptions.assumeTrue(isDockerAvailable(), "Docker daemon must be available for Redshift catalog integration tests");+ }++ private static boolean isDockerAvailable() {+ try {+ Process process = new ProcessBuilder("docker", "version", "--format", "{{.Server.Version}}")+ .redirectErrorStream(true)+ .start();+ int exit = process.waitFor();+ return exit == 0;+ } catch (Exception e) {+ return false;+ }+ }++ @AfterEach+ void cleanUp() {+ if (clusterId != null) {+ service.deleteCluster(clusterId);+ }+ }++ private static String jdbcUrl(Cluster cluster) {+ return "jdbc:postgresql://127.0.0.1:" + cluster.getEndpoint().getPort() + "/dev";+ }++ private static Connection waitForConnection(Cluster cluster, String username, String password) throws SQLException {+ try {+ return Awaitility.await()+ .atMost(Duration.ofSeconds(30))+ .pollInterval(Duration.ofMillis(500))+ .ignoreExceptions()+ .until(() -> DriverManager.getConnection(jdbcUrl(cluster), username, password), Objects::nonNull);+ } catch (ConditionTimeoutException e) {+ return DriverManager.getConnection(jdbcUrl(cluster), username, password);+ }+ }++ @Test+ void testCatalogViewsIntrospection() throws Exception {+ clusterId = "cat-seed-" + UUID.randomUUID().toString().substring(0, 8);+ Cluster cluster = service.createCluster(clusterId, "dc2.large", "admin", "Secret123");++ try (Connection conn = waitForConnection(cluster, "admin", "Secret123");+ Statement stmt = conn.createStatement()) {++ // Create sample table for metadata verification+ stmt.execute("CREATE TABLE test_catalog_users (id integer, username varchar(50) NOT NULL)");++ // 1. Verify pg_table_def+ try (PreparedStatement ps = conn.prepareStatement(+ "SELECT \"column\", type, \"notnull\" FROM pg_table_def WHERE tablename = ? ORDER BY \"column\"")) {+ ps.setString(1, "test_catalog_users");+ try (ResultSet rs = ps.executeQuery()) {+ assertTrue(rs.next(), "pg_table_def must contain at least one column");+ assertEquals("id", rs.getString("column"));+ assertFalse(rs.getBoolean("notnull"));++ assertTrue(rs.next(), "pg_table_def must contain second column");+ assertEquals("username", rs.getString("column"));+ assertTrue(rs.getBoolean("notnull"));+ }+ }++ // 2. Verify svv_table_info+ try (PreparedStatement ps = conn.prepareStatement(+ "SELECT \"table\", \"schema\", encoded, diststyle FROM svv_table_info WHERE \"table\" = ?")) {+ ps.setString(1, "test_catalog_users");+ try (ResultSet rs = ps.executeQuery()) {+ assertTrue(rs.next(), "svv_table_info must contain a record for test_catalog_users");+ assertEquals("test_catalog_users", rs.getString("table"));+ assertEquals("public", rs.getString("schema"));+ assertEquals("EVEN", rs.getString("diststyle"));+ }+ }++ // 3. Verify svv_all_columns+ try (PreparedStatement ps = conn.prepareStatement(+ "SELECT column_name, is_nullable FROM svv_all_columns WHERE table_name = ? ORDER BY column_name")) {+ ps.setString(1, "test_catalog_users");+ try (ResultSet rs = ps.executeQuery()) {+ assertTrue(rs.next());+ assertEquals("id", rs.getString("column_name"));+ assertEquals("YES", rs.getString("is_nullable"));++ assertTrue(rs.next());+ assertEquals("username", rs.getString("column_name"));+ assertEquals("NO", rs.getString("is_nullable"));+ }+ }++ // 4. Verify svv_tables+ try (PreparedStatement ps = conn.prepareStatement(+ "SELECT table_name FROM svv_tables WHERE table_name = ?")) {+ ps.setString(1, "test_catalog_users");+ try (ResultSet rs = ps.executeQuery()) {+ assertTrue(rs.next(), "svv_tables must contain test_catalog_users");+ }+ }++ // 5. Verify stv_tbl_perm+ try (PreparedStatement ps = conn.prepareStatement(+ "SELECT name FROM stv_tbl_perm WHERE name = ?")) {+ ps.setString(1, "test_catalog_users");+ try (ResultSet rs = ps.executeQuery()) {+ assertTrue(rs.next(), "stv_tbl_perm must contain test_catalog_users");+ }+ }++ // 6. Verify stl_load_errors+ try (ResultSet rs = stmt.executeQuery("SELECT count(*) FROM stl_load_errors")) {+ assertTrue(rs.next());+ assertEquals(0, rs.getInt(1));+ }++ // 7. Verify svl_qlog+ try (ResultSet rs = stmt.executeQuery("SELECT count(*) FROM svl_qlog")) {+ assertTrue(rs.next());+ assertTrue(rs.getInt(1) >= 1);+ }++ // 8. Verify pg_user_info+ try (ResultSet rs = stmt.executeQuery("SELECT usename FROM pg_user_info WHERE usename = 'admin'")) {+ assertTrue(rs.next(), "pg_user_info must contain admin user");+ }++ // 9. Verify stv_sessions+ try (ResultSet rs = stmt.executeQuery("SELECT process, user_name, db_name FROM stv_sessions WHERE user_name = 'admin'")) {+ assertTrue(rs.next(), "stv_sessions must contain admin session");+ assertEquals("admin", rs.getString("user_name"));+ assertEquals("dev", rs.getString("db_name"));+ }++ // 10. Verify stv_recents+ try (ResultSet rs = stmt.executeQuery("SELECT status, user_name, db_name FROM stv_recents WHERE user_name = 'admin'")) {+ assertTrue(rs.next(), "stv_recents must contain admin queries");+ assertEquals("admin", rs.getString("user_name"));+ }++ // 11. Verify pg_database_info+ try (ResultSet rs = stmt.executeQuery("SELECT datname FROM pg_database_info WHERE datname = 'dev'")) {+ assertTrue(rs.next(), "pg_database_info must contain dev database");+ }++ // 12. Verify svv_columns (AWS Redshift contract)+ try (PreparedStatement ps = conn.prepareStatement(+ "SELECT table_catalog, table_schema, table_name, column_name, ordinal_position, data_type " ++ "FROM svv_columns WHERE table_name = ? ORDER BY column_name")) {+ ps.setString(1, "test_catalog_users");+ try (ResultSet rs = ps.executeQuery()) {+ assertTrue(rs.next(), "svv_columns must contain id column");+ assertEquals("dev", rs.getString("table_catalog"));+ assertEquals("public", rs.getString("table_schema"));+ assertEquals("id", rs.getString("column_name"));+ assertEquals(1, rs.getInt("ordinal_position"));+ assertTrue(rs.next(), "svv_columns must contain username column");+ assertEquals("username", rs.getString("column_name"));+ assertEquals(2, rs.getInt("ordinal_position"));+ }+ }++ // 13. Verify svv_transactions+ try (ResultSet rs = stmt.executeQuery("SELECT count(*) FROM svv_transactions")) {+ assertTrue(rs.next());+ }++ // 14. Verify stv_slices+ try (ResultSet rs = stmt.executeQuery("SELECT slice, node FROM stv_slices")) {+ assertTrue(rs.next());+ assertEquals(0, rs.getInt("slice"));+ assertEquals(0, rs.getInt("node"));+ }++ // 15. Verify stl_query (dynamic view over pg_stat_activity)+ try (ResultSet rs = stmt.executeQuery("SELECT count(*) FROM stl_query")) {+ assertTrue(rs.next());+ assertTrue(rs.getInt(1) >= 1, "stl_query must report active or recent queries");+ }++ // 16. Verify stv_wlm_query_state+ try (ResultSet rs = stmt.executeQuery("SELECT count(*) FROM stv_wlm_query_state")) {+ assertTrue(rs.next());+ }++ // 17. Verify svv_diskusage+ try (PreparedStatement ps = conn.prepareStatement(+ "SELECT name, size, used FROM svv_diskusage WHERE name = ?")) {+ ps.setString(1, "test_catalog_users");+ try (ResultSet rs = ps.executeQuery()) {+ assertTrue(rs.next(), "svv_diskusage must contain test_catalog_users");+ assertEquals("test_catalog_users", rs.getString("name"));+ }+ }++ // 18. Verify newly created database inherits catalog views from template1+ stmt.execute("CREATE DATABASE test_clone_db");+ }++ String cloneDbUrl = "jdbc:postgresql://127.0.0.1:" + cluster.getEndpoint().getPort() + "/test_clone_db";+ try (Connection cloneConn = DriverManager.getConnection(cloneDbUrl, "admin", "Secret123");+ Statement cloneStmt = cloneConn.createStatement()) {+ try (ResultSet rs = cloneStmt.executeQuery("SELECT count(*) FROM pg_table_def")) {+ assertTrue(rs.next(), "cloned database must have pg_table_def from template1");+ }+ try (ResultSet rs = cloneStmt.executeQuery("SELECT count(*) FROM svv_table_info")) {+ assertTrue(rs.next(), "cloned database must have svv_table_info from template1");+ }+ try (ResultSet rs = cloneStmt.executeQuery("SELECT count(*) FROM stv_sessions")) {+ assertTrue(rs.next(), "cloned database must have stv_sessions from template1");+ }+ }+ }++ @Test+ void testAllDocumentedRedshiftColumnsContract() throws Exception {+ clusterId = "cat-contract-" + UUID.randomUUID().toString().substring(0, 8);+ Cluster cluster = service.createCluster(clusterId, "dc2.large", "admin", "Secret123");++ try (Connection conn = waitForConnection(cluster, "admin", "Secret123");+ Statement stmt = conn.createStatement()) {++ // Pre-seed a test table so metadata views have rows+ stmt.execute("CREATE TABLE contract_test_table (col_int integer, col_varchar varchar(100))");++ String[] contractQueries = new String[] {+ // 1. pg_table_def (AWS Redshift documented columns)+ "SELECT schemaname, tablename, \"column\", type, encoding, distkey, sortkey, \"notnull\" FROM pg_table_def LIMIT 1",++ // 2. svv_table_info (AWS Redshift documented columns)+ "SELECT \"database\", \"schema\", table_id, \"table\", encoded, diststyle, sortkey1, max_varchar, sortkey1_enc, sortkey_num, size, pct_used, empty, unsorted, stats_off, tbl_rows, skew_sortkey1, skew_rows, estimated_visible_rows, risk_event, vacuum_sort_benefit, create_time FROM svv_table_info LIMIT 1",++ // 3. svv_all_columns (AWS Redshift documented columns)+ "SELECT database_name, schema_name, table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, numeric_precision, numeric_scale, remarks FROM svv_all_columns LIMIT 1",++ // 4. svv_tables (AWS Redshift documented columns)+ "SELECT table_catalog, table_schema, table_name, table_type, remarks FROM svv_tables LIMIT 1",++ // 5. stv_tbl_perm (AWS Redshift documented columns)+ "SELECT slice, id, name, rows, sorted_rows, temp, db_id, insert_pristine, delete_pristine, backup, dist_style, block_count FROM stv_tbl_perm LIMIT 1",++ // 6. stl_load_errors (AWS Redshift documented columns)+ "SELECT userid, slice, tbl, starttime, session, query, filename, line_number, colname, type, col_length, position, raw_line, raw_field_value, err_code, err_reason, is_partial, start_offset, copy_job_id FROM stl_load_errors LIMIT 1",++ // 7. svl_qlog (AWS Redshift documented columns)+ "SELECT userid, query, xid, pid, starttime, endtime, elapsed, aborted, insert_pristine, concurrency_scaling_status, source_query, label, substring, concurrency_scaling_status_txt, from_sp_call FROM svl_qlog LIMIT 1",++ // 8. pg_user_info (Redshift user catalog view)+ "SELECT usename, usesysid, usecreatedb, usesuper, usecatupd, valuntil, useconfig, useconnlimit, syslogaccess, last_ddl_ts, sessiontimeout, external_id FROM pg_user_info LIMIT 1",++ // 8b. svl_user_info (AWS Redshift documented columns)+ "SELECT usename, usesysid, usecreatedb, usesuper, usecatupd, useconnlimit, syslogaccess, last_ddl_ts, sessiontimeout, external_id FROM svl_user_info LIMIT 1",++ // 9. stv_sessions (AWS Redshift documented columns)+ "SELECT starttime, process, user_name, db_name, timeout_sec FROM stv_sessions LIMIT 1",++ // 10. stv_recents (AWS Redshift documented columns)+ "SELECT userid, status, starttime, duration, user_name, db_name, query, pid FROM stv_recents LIMIT 1",++ // 11. pg_database_info (AWS Redshift documented columns)+ "SELECT datname, datid, datdba, encoding, datconnlimit, datistemplate, datallowconn, dattablespace FROM pg_database_info LIMIT 1",++ // 12. svv_columns (AWS Redshift documented columns)+ "SELECT table_catalog, table_schema, table_name, column_name, ordinal_position, column_default, is_nullable, data_type, character_maximum_length, numeric_precision, numeric_precision_radix, numeric_scale, datetime_precision, interval_type, interval_precision, character_set_catalog, character_set_schema, character_set_name, collation_catalog, collation_schema, collation_name, domain_name, remarks FROM svv_columns LIMIT 1",++ // 13. svv_transactions (AWS Redshift documented columns)+ "SELECT txn_owner, txn_db, xid, pid, txn_start, lock_mode, lockable_object_type, relation, granted FROM svv_transactions LIMIT 1",++ // 14. stv_slices (AWS Redshift documented columns)+ "SELECT node, slice, localslice, type FROM stv_slices LIMIT 1",++ // 15. stl_query (AWS Redshift documented columns)+ "SELECT userid, query, label, xid, pid, database, querytxt, starttime, endtime, aborted, insert_pristine, concurrency_scaling_status FROM stl_query LIMIT 1",++ // 16. stv_wlm_query_state (AWS Redshift documented columns)+ "SELECT xid, task, query, service_class, slot_count, wlm_start_time, state, queue_time, exec_time, query_priority FROM stv_wlm_query_state LIMIT 1",++ // 17. svv_diskusage (AWS Redshift documented columns)+ "SELECT db_id, name, slice, col, tbl, blocknum, num_values, minvalue, maxvalue, sb_pos, pinned, on_disk, modified, hdr_modified, unsorted, tombstone, preferred_diskno, temporary, newblock FROM svv_diskusage LIMIT 1"+ };++ for (String sql : contractQueries) {+ try (ResultSet rs = stmt.executeQuery(sql)) {+ assertNotNull(rs.getMetaData(), "ResultSet metadata must not be null for query: " + sql);+ }+ }+ }+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/container/RedshiftCatalogResourceTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/container/RedshiftCatalogResourceTest.javanew file mode 100644index 0000000000..063eef6758--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/container/RedshiftCatalogResourceTest.java@@ -0,0 +1,38 @@+package io.github.hectorvent.floci.services.redshift.container;++import org.junit.jupiter.api.Test;++import java.io.InputStream;+import java.nio.charset.StandardCharsets;++import static org.junit.jupiter.api.Assertions.assertNotNull;+import static org.junit.jupiter.api.Assertions.assertTrue;++class RedshiftCatalogResourceTest {++ @Test+ void testBootstrapCatalogResourceExistsAndContainsExpectedViews() throws Exception {+ try (InputStream in = getClass().getResourceAsStream("/redshift/bootstrap-catalog.sql")) {+ assertNotNull(in, "bootstrap-catalog.sql must exist on classpath");+ String sql = new String(in.readAllBytes(), StandardCharsets.UTF_8);+ assertTrue(sql.contains("pg_table_def"), "Missing DDL for pg_table_def");+ assertTrue(sql.contains("svv_table_info"), "Missing DDL for svv_table_info");+ assertTrue(sql.contains("svv_all_columns"), "Missing DDL for svv_all_columns");+ assertTrue(sql.contains("svv_tables"), "Missing DDL for svv_tables");+ assertTrue(sql.contains("stv_tbl_perm"), "Missing DDL for stv_tbl_perm");+ assertTrue(sql.contains("stl_load_errors"), "Missing DDL for stl_load_errors");+ assertTrue(sql.contains("svl_qlog"), "Missing DDL for svl_qlog");+ assertTrue(sql.contains("pg_user_info"), "Missing DDL for pg_user_info");+ assertTrue(sql.contains("svl_user_info"), "Missing DDL for svl_user_info");+ assertTrue(sql.contains("stv_sessions"), "Missing DDL for stv_sessions");+ assertTrue(sql.contains("stv_recents"), "Missing DDL for stv_recents");+ assertTrue(sql.contains("pg_database_info"), "Missing DDL for pg_database_info");+ assertTrue(sql.contains("svv_columns"), "Missing DDL for svv_columns");+ assertTrue(sql.contains("svv_transactions"), "Missing DDL for svv_transactions");+ assertTrue(sql.contains("stv_slices"), "Missing DDL for stv_slices");+ assertTrue(sql.contains("stl_query"), "Missing DDL for stl_query");+ assertTrue(sql.contains("stv_wlm_query_state"), "Missing DDL for stv_wlm_query_state");+ assertTrue(sql.contains("svv_diskusage"), "Missing DDL for svv_diskusage");+ }+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/container/RedshiftContainerManagerTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/container/RedshiftContainerManagerTest.javaindex 111b7d280d..462f460379 100644--- a/src/test/java/io/github/hectorvent/floci/services/redshift/container/RedshiftContainerManagerTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/container/RedshiftContainerManagerTest.java@@ -95,6 +95,9 @@ class RedshiftContainerManagerTest {when(defaultInspectCmd.exec()).thenReturn(defaultInspectResponse);when(dockerClient.inspectExecCmd(anyString())).thenReturn(defaultInspectCmd);+ CopyArchiveToContainerCmd defaultCopyCmd = mock(CopyArchiveToContainerCmd.class, org.mockito.Mockito.RETURNS_SELF);+ when(dockerClient.copyArchiveToContainerCmd(anyString())).thenReturn(defaultCopyCmd);+manager = new RedshiftContainerManager(containerBuilder,lifecycleManager,@@ -287,8 +290,8 @@ class RedshiftContainerManagerTest {Path tempFile = Files.createTempFile("test-restore-snapshot", ".sql");try {manager.restoreSnapshot(ACCOUNT_ID, "test-cluster", "admin", "dev",tempFile);- verify(dockerClient).copyArchiveToContainerCmd("cont-123");- verify(dockerClient, org.mockito.Mockito.times(2)).execCreateCmd("cont-123");+ verify(dockerClient, org.mockito.Mockito.times(2)).copyArchiveToContainerCmd("cont-123");+ verify(dockerClient, org.mockito.Mockito.times(4)).execCreateCmd("cont-123");} finally {Files.deleteIfExists(tempFile);}@@ -393,7 +396,7 @@ class RedshiftContainerManagerTest {manager.alterUserPassword(ACCOUNT_ID, "test-cluster", "admin", "NewSecret1");- verify(dockerClient, org.mockito.Mockito.times(2)).execCreateCmd("cont-123");+ verify(dockerClient, org.mockito.Mockito.times(4)).execCreateCmd("cont-123");}@Test@@ -542,4 +545,38 @@ class RedshiftContainerManagerTest {assertNotNull(handleTolerated);assertNull(handleTolerated.getLogStream());}++ @Test+ void testBootstrapCatalogSuccess() {+ CopyArchiveToContainerCmd copyCmd = mock(CopyArchiveToContainerCmd.class, org.mockito.Mockito.RETURNS_SELF);+ when(dockerClient.copyArchiveToContainerCmd("cont-bootstrap")).thenReturn(copyCmd);++ ExecCreateCmd createCmd = mock(ExecCreateCmd.class, org.mockito.Mockito.RETURNS_SELF);+ ExecCreateCmdResponse createResponse = mock(ExecCreateCmdResponse.class);+ when(createResponse.getId()).thenReturn("exec-bootstrap");+ when(createCmd.exec()).thenReturn(createResponse);+ when(dockerClient.execCreateCmd("cont-bootstrap")).thenReturn(createCmd);++ InspectExecCmd inspectCmd = mock(InspectExecCmd.class);+ InspectExecResponse inspectResponse = mock(InspectExecResponse.class);+ when(inspectResponse.getExitCodeLong()).thenReturn(0L);+ when(inspectCmd.exec()).thenReturn(inspectResponse);+ when(dockerClient.inspectExecCmd("exec-bootstrap")).thenReturn(inspectCmd);++ manager.bootstrapCatalog("cont-bootstrap", "admin", "dev");++ verify(dockerClient).copyArchiveToContainerCmd("cont-bootstrap");+ verify(dockerClient, org.mockito.Mockito.times(2)).execCreateCmd("cont-bootstrap");+ }++ @Test+ void testBootstrapCatalogFailureDoesNotThrow() {+ CopyArchiveToContainerCmd copyCmd = mock(CopyArchiveToContainerCmd.class, org.mockito.Mockito.RETURNS_SELF);+ when(copyCmd.exec()).thenThrow(new RuntimeException("Docker copy error"));+ when(dockerClient.copyArchiveToContainerCmd("cont-bootstrap-fail")).thenReturn(copyCmd);++ // Must not bubble an exception that fails cluster startup+ org.junit.jupiter.api.Assertions.assertDoesNotThrow(() ->+ manager.bootstrapCatalog("cont-bootstrap-fail", "admin", "dev"));+ }}diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.javaindex 6b19bf101b..b80d175b94 100644--- a/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/proxy/S3CopySimulatorTest.java@@ -1196,4 +1196,5 @@ class S3CopySimulatorTest {assertEquals('E', err.type());assertNull(in.nextMessage(), "Client must not receive an unconfirmed ReadyForQuery");}+}