Skip to content

Business intelligence made simple

Added support for Apache Hive and Apache Spark

Andrew Kanec18bdc9f9837
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b61b33c2..d1b60d08 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,6 +1,7 @@
## 2.4.1 (unreleased)
- Added cohorts for MySQL
+- Added support for Apache Hive and Apache Spark
- Fixed deprecation warning with Active Record 6.1
## 2.4.0 (2020-12-15)
diff --git a/README.md b/README.md
index 0202abe5..9edb4e3d 100644
--- a/README.md
+++ b/README.md
@@ -566,6 +566,8 @@ data_sources:
- [Amazon Athena](#amazon-athena)
- [Amazon Redshift](#amazon-redshift)
- [Apache Drill](#apache-drill)
+- [Apache Hive](#apache-hive) [master]
+- [Apache Spark](#apache-spark) [master]
- [Cassandra](#cassandra)
- [Druid](#druid)
- [Elasticsearch](#elasticsearch)
@@ -627,6 +629,32 @@ data_sources:
url: http://hostname:8047
```
+### Apache Hive
+
+Add [hexspace](https://github.com/ankane/hexspace) to your Gemfile and set:
+
+```yml
+data_sources:
+ my_source:
+ adapter: hive
+ url: sasl://user:password@hostname:10000/database
+```
+
+Use a read-only user. Requires [HiveServer2](https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2).
+
+### Apache Spark
+
+Add [hexspace](https://github.com/ankane/hexspace) to your Gemfile and set:
+
+```yml
+data_sources:
+ my_source:
+ adapter: spark
+ url: sasl://user:password@hostname:10000/database
+```
+
+Use a read-only user. Requires the [Thrift server](https://spark.apache.org/docs/latest/sql-distributed-sql-engine.html).
+
### Cassandra
Add [cassandra-driver](https://github.com/datastax/ruby-driver) to your Gemfile and set:
diff --git a/lib/blazer.rb b/lib/blazer.rb
index bef51238..d1f72bf7 100644
--- a/lib/blazer.rb
+++ b/lib/blazer.rb
@@ -18,12 +18,14 @@ require "blazer/adapters/cassandra_adapter"
require "blazer/adapters/drill_adapter"
require "blazer/adapters/druid_adapter"
require "blazer/adapters/elasticsearch_adapter"
+require "blazer/adapters/hive_adapter"
require "blazer/adapters/influxdb_adapter"
require "blazer/adapters/mongodb_adapter"
require "blazer/adapters/neo4j_adapter"
require "blazer/adapters/presto_adapter"
require "blazer/adapters/salesforce_adapter"
require "blazer/adapters/soda_adapter"
+require "blazer/adapters/spark_adapter"
require "blazer/adapters/sql_adapter"
require "blazer/adapters/snowflake_adapter"
@@ -239,11 +241,13 @@ Blazer.register_adapter "cassandra", Blazer::Adapters::CassandraAdapter
Blazer.register_adapter "drill", Blazer::Adapters::DrillAdapter
Blazer.register_adapter "druid", Blazer::Adapters::DruidAdapter
Blazer.register_adapter "elasticsearch", Blazer::Adapters::ElasticsearchAdapter
+Blazer.register_adapter "hive", Blazer::Adapters::HiveAdapter
Blazer.register_adapter "influxdb", Blazer::Adapters::InfluxdbAdapter
Blazer.register_adapter "neo4j", Blazer::Adapters::Neo4jAdapter
Blazer.register_adapter "presto", Blazer::Adapters::PrestoAdapter
Blazer.register_adapter "mongodb", Blazer::Adapters::MongodbAdapter
Blazer.register_adapter "salesforce", Blazer::Adapters::SalesforceAdapter
Blazer.register_adapter "soda", Blazer::Adapters::SodaAdapter
+Blazer.register_adapter "spark", Blazer::Adapters::SparkAdapter
Blazer.register_adapter "sql", Blazer::Adapters::SqlAdapter
Blazer.register_adapter "snowflake", Blazer::Adapters::SnowflakeAdapter
diff --git a/lib/blazer/adapters/hive_adapter.rb b/lib/blazer/adapters/hive_adapter.rb
new file mode 100644
index 00000000..467ede91
--- /dev/null
+++ b/lib/blazer/adapters/hive_adapter.rb
@@ -0,0 +1,45 @@
+module Blazer
+ module Adapters
+ class HiveAdapter < BaseAdapter
+ def run_statement(statement, comment)
+ columns = []
+ rows = []
+ error = nil
+
+ begin
+ result = client.execute("#{statement} /*#{comment}*/")
+ columns = result.any? ? result.first.keys : []
+ rows = result.map(&:values)
+ rescue => e
+ error = e.message
+ end
+
+ [columns, rows, error]
+ end
+
+ def tables
+ client.execute("SHOW TABLES").map { |r| r["tableName"] }
+ end
+
+ def preview_statement
+ "SELECT * FROM {table} LIMIT 10"
+ end
+
+ protected
+
+ def client
+ @client ||= begin
+ uri = URI.parse(settings["url"])
+ Hexspace::Client.new(
+ host: uri.host,
+ port: uri.port,
+ username: uri.user,
+ password: uri.password,
+ database: uri.path.sub(/\A\//, ""),
+ mode: uri.scheme.to_sym
+ )
+ end
+ end
+ end
+ end
+end
diff --git a/lib/blazer/adapters/spark_adapter.rb b/lib/blazer/adapters/spark_adapter.rb
new file mode 100644
index 00000000..794dac62
--- /dev/null
+++ b/lib/blazer/adapters/spark_adapter.rb
@@ -0,0 +1,6 @@
+module Blazer
+ module Adapters
+ class SparkAdapter < HiveAdapter
+ end
+ end
+end