Skip to content

Business intelligence made simple

Fixed quoting issue with variables - fixes #391

Andrew Kanef49fbfed7b9e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ca54e23f..84f671ba 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,7 @@
-## 2.5.1 (unreleased)
+## 2.6.0 (unreleased)
+- Fixed quoting issue with variables
+- Custom adapters now need to specify how to quote variables in queries
- Added experimental support for Propshaft
- Fixed error with empty results with InfluxDB
diff --git a/README.md b/README.md
index 88b69a12..2914eb34 100644
--- a/README.md
+++ b/README.md
@@ -61,7 +61,7 @@ For production, specify your database:
ENV["BLAZER_DATABASE_URL"] = "postgres://user:password@hostname:5432/database"
```
-Blazer tries to protect against queries which modify data (by running each query in a transaction and rolling it back), but a safer approach is to use a read-only user. [See how to create one](#permissions).
+When possible, Blazer tries to protect against queries which modify data by running each query in a transaction and rolling it back, but a safer approach is to use a read-only user. [See how to create one](#permissions).
#### Checks (optional)
@@ -144,11 +144,9 @@ Be sure to render or redirect for unauthorized users.
## Permissions
-Blazer runs each query in a transaction and rolls it back to prevent queries from modifying data. As an additional line of defense, we recommend using a read only user.
-
### PostgreSQL
-Create a user with read only permissions:
+Create a user with read-only permissions:
```sql
BEGIN;
@@ -162,7 +160,7 @@ COMMIT;
### MySQL
-Create a user with read only permissions:
+Create a user with read-only permissions:
```sql
GRANT SELECT, SHOW VIEW ON database_name.* TO blazer@’127.0.0.1′ IDENTIFIED BY ‘secret123‘;
@@ -171,7 +169,7 @@ FLUSH PRIVILEGES;
### MongoDB
-Create a user with read only permissions:
+Create a user with read-only permissions:
```
db.createUser({user: "blazer", pwd: "password", roles: ["read"]})
@@ -656,6 +654,8 @@ data_sources:
url: redshift://user:password@hostname:5439/database
```
+Use a [read-only user](https://docs.aws.amazon.com/redshift/latest/dg/r_GRANT.html).
+
### Apache Drill
Add [drill-sergeant](https://github.com/ankane/drill-sergeant) to your Gemfile and set:
@@ -667,6 +667,8 @@ data_sources:
url: http://hostname:8047
```
+Use a [read-only user](https://drill.apache.org/docs/roles-and-privileges/).
+
### Apache Hive
Add [hexspace](https://github.com/ankane/hexspace) to your Gemfile and set:
@@ -690,6 +692,8 @@ data_sources:
url: ignite://user:password@hostname:10800
```
+Use a [read-only user](https://www.gridgain.com/docs/latest/administrators-guide/security/authorization-permissions) (requires a third-party plugin).
+
### Apache Spark
Add [hexspace](https://github.com/ankane/hexspace) to your Gemfile and set:
@@ -713,6 +717,8 @@ data_sources:
url: cassandra://user:password@hostname:9042/keyspace
```
+Use a [read-only role](https://docs.datastax.com/en/cql-oss/3.3/cql/cql_using/useSecurePermission.html).
+
### Druid
Enable [SQL support](http://druid.io/docs/latest/querying/sql.html#configuration) on the broker and set:
@@ -724,6 +730,8 @@ data_sources:
url: http://hostname:8082
```
+Use a [read-only role](https://druid.apache.org/docs/latest/development/extensions-core/druid-basic-security.html).
+
### Elasticsearch
Add [elasticsearch](https://github.com/elastic/elasticsearch-ruby) to your Gemfile and set:
@@ -735,6 +743,8 @@ data_sources:
url: http://user:password@hostname:9200
```
+Use a [read-only role](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html).
+
### Google BigQuery
Add [google-cloud-bigquery](https://github.com/GoogleCloudPlatform/google-cloud-ruby/tree/master/google-cloud-bigquery) to your Gemfile and set:
@@ -757,6 +767,8 @@ data_sources:
url: ibm-db://user:password@hostname:50000/database
```
+Use a [read-only user](https://www.ibm.com/support/pages/creating-read-only-database-permissions-user).
+
### InfluxDB
Add [influxdb](https://github.com/influxdata/influxdb-ruby) to your Gemfile and set:
@@ -768,7 +780,7 @@ data_sources:
url: http://user:password@hostname:8086/database
```
-Supports [InfluxQL](https://docs.influxdata.com/influxdb/v1.8/query_language/explore-data/)
+Use a [read-only user](https://docs.influxdata.com/influxdb/v1.8/administration/authentication_and_authorization/). Supports [InfluxQL](https://docs.influxdata.com/influxdb/v1.8/query_language/explore-data/).
### MongoDB
@@ -782,6 +794,8 @@ data_sources:
url: mongodb://user:password@hostname:27017/database
```
+Use a [read-only user](#mongodb).
+
### MySQL
Add [mysql2](https://github.com/brianmario/mysql2) to your Gemfile (if it’s not there) and set:
@@ -792,6 +806,8 @@ data_sources:
url: mysql2://user:password@hostname:3306/database
```
+Use a [read-only user](#mysql).
+
### Neo4j
Add [neo4j-core](https://github.com/neo4jrb/neo4j-core) to your Gemfile and set:
@@ -803,6 +819,8 @@ data_sources:
url: http://user:password@hostname:7474
```
+Use a [read-only user](https://neo4j.com/docs/cypher-manual/current/access-control/manage-privileges/).
+
### OpenSearch
Add [opensearch-ruby](https://github.com/opensearch-project/opensearch-ruby) to your Gemfile and set:
@@ -814,6 +832,8 @@ data_sources:
url: http://user:password@hostname:9200
```
+Use a [read-only user](https://opensearch.org/docs/latest/security-plugin/access-control/permissions/).
+
### Oracle
Add [activerecord-oracle_enhanced-adapter](https://github.com/rsim/oracle-enhanced) and [ruby-oci8](https://github.com/kubo/ruby-oci8) to your Gemfile and set:
@@ -824,6 +844,8 @@ data_sources:
url: oracle-enhanced://user:password@hostname:1521/database
```
+Use a [read-only user](https://docs.oracle.com/cd/B19306_01/network.102/b14266/authoriz.htm).
+
### PostgreSQL
Add [pg](https://github.com/ged/ruby-pg) to your Gemfile (if it’s not there) and set:
@@ -834,6 +856,8 @@ data_sources:
url: postgres://user:password@hostname:5432/database
```
+Use a [read-only user](#postgresql).
+
### Presto
Add [presto-client](https://github.com/treasure-data/presto-client-ruby) to your Gemfile and set:
@@ -844,6 +868,8 @@ data_sources:
url: presto://user@hostname:8080/catalog
```
+Use a [read-only user](https://prestodb.io/docs/current/security/built-in-system-access-control.html).
+
### Salesforce
Add [restforce](https://github.com/restforce/restforce) to your Gemfile and set:
@@ -865,7 +891,7 @@ SALESFORCE_CLIENT_SECRET="client secret"
SALESFORCE_API_VERSION="41.0"
```
-Supports [SOQL](https://developer.salesforce.com/docs/atlas.en-us.soql_sosl.meta/soql_sosl/sforce_api_calls_soql.htm)
+Use a read-only user. Supports [SOQL](https://developer.salesforce.com/docs/atlas.en-us.soql_sosl.meta/soql_sosl/sforce_api_calls_soql.htm).
### Socrata Open Data API (SODA)
@@ -879,7 +905,7 @@ data_sources:
app_token: ...
```
-Supports [SoQL](https://dev.socrata.com/docs/functions/)
+Supports [SoQL](https://dev.socrata.com/docs/functions/).
### Snowflake
@@ -913,6 +939,8 @@ data_sources:
conn_str: Driver=/path/to/libSnowflake.so;uid=user;pwd=password;server=host.snowflakecomputing.com
```
+Use a [read-only role](https://docs.snowflake.com/en/user-guide/security-access-control-configure.html).
+
### SQLite
Add [sqlite3](https://github.com/sparklemotion/sqlite3-ruby) to your Gemfile and set:
@@ -933,6 +961,8 @@ data_sources:
url: sqlserver://user:password@hostname:1433/database
```
+Use a [read-only user](https://docs.microsoft.com/en-us/sql/relational-databases/security/authentication-access/getting-started-with-database-engine-permissions?view=sql-server-ver15).
+
## Creating an Adapter
Create an adapter for any data store with:
@@ -1009,6 +1039,22 @@ override_csp: true
## Upgrading
+### 2.6
+
+Custom adapters now need to specify how to quote variables in queries (there is no longer a default)
+
+```ruby
+class FooAdapter < Blazer::Adapters::BaseAdapter
+ def quoting
+ :backslash_escape # single quote strings and convert ' to \' and \ to \\
+ # or
+ :single_quote_escape # single quote strings and convert ' to ''
+ # or
+ ->(value) { ... } # custom method
+ end
+end
+```
+
### 2.3
To archive queries, create a migration
diff --git a/app/controllers/blazer/base_controller.rb b/app/controllers/blazer/base_controller.rb
index 1b359544..fc22511b 100644
--- a/app/controllers/blazer/base_controller.rb
+++ b/app/controllers/blazer/base_controller.rb
@@ -32,46 +32,32 @@ module Blazer
private
- def process_vars(statement, data_source, var_params = nil)
+ def process_vars(statement, var_params = nil)
var_params ||= request.query_parameters
- (@bind_vars ||= []).concat(Blazer.extract_vars(statement)).uniq!
+ (@bind_vars ||= []).concat(statement.variables).uniq!
# update in-place so populated in view and consistent across queries on dashboard
@bind_vars.each do |var|
- var_params[var] ||= Blazer.data_sources[data_source].variable_defaults[var]
- end
- @success = @bind_vars.all? { |v| var_params[v] }
-
- if @success
- @bind_vars.each do |var|
- value = var_params[var].presence
- if value
- if ["start_time", "end_time"].include?(var)
- value = value.to_s.gsub(" ", "+") # fix for Quip bug
- end
-
- if var.end_with?("_at")
- begin
- value = Blazer.time_zone.parse(value)
- rescue
- # do nothing
- end
- end
-
- if value =~ /\A\d+\z/
- value = value.to_i
- elsif value =~ /\A\d+\.\d+\z/
- value = value.to_f
- end
- end
- value = Blazer.transform_variable.call(var, value) if Blazer.transform_variable
- statement.gsub!("{#{var}}", ActiveRecord::Base.connection.quote(value))
+ if !var_params[var]
+ default = statement.data_source.variable_defaults[var]
+ # only add if default exists
+ var_params[var] = default if default
end
end
+ runnable = @bind_vars.all? { |v| var_params[v] }
+ statement.add_values(var_params) if runnable
+ runnable
+ end
+
+ def refresh_query(query)
+ statement = query.statement_object
+ runnable = process_vars(statement)
+ cohort_analysis_statement(statement) if statement.cohort_analysis?
+ statement.clear_cache if runnable
end
def add_cohort_analysis_vars
@bind_vars << "cohort_period" unless @bind_vars.include?("cohort_period")
- @smart_vars["cohort_period"] = ["day", "week", "month"]
+ @smart_vars["cohort_period"] = ["day", "week", "month"] if @smart_vars
# TODO create var_params method
request.query_parameters["cohort_period"] ||= "week"
end
@@ -97,6 +83,25 @@ module Blazer
[smart_var, error]
end
+ def cohort_analysis_statement(statement)
+ @cohort_period = params["cohort_period"] || "week"
+ @cohort_period = "week" unless ["day", "week", "month"].include?(@cohort_period)
+
+ # for now
+ @conversion_period = @cohort_period
+ @cohort_days =
+ case @cohort_period
+ when "day"
+ 1
+ when "week"
+ 7
+ when "month"
+ 30
+ end
+
+ statement.apply_cohort_analysis(period: @cohort_period, days: @cohort_days)
+ end
+
# TODO allow all keys
# or show error message for disallowed keys
UNPERMITTED_KEYS = [:controller, :action, :id, :host, :query, :dashboard, :query_id, :query_ids, :table_names, :authenticity_token, :utf8, :_method, :commit, :statement, :data_source, :name, :fork_query_id, :blazer, :run_id, :script_name, :original_script_name]
diff --git a/app/controllers/blazer/dashboards_controller.rb b/app/controllers/blazer/dashboards_controller.rb
index c752a8cc..5cb5564d 100644
--- a/app/controllers/blazer/dashboards_controller.rb
+++ b/app/controllers/blazer/dashboards_controller.rb
@@ -21,11 +21,8 @@ module Blazer
def show
@queries = @dashboard.dashboard_queries.order(:position).preload(:query).map(&:query)
- @statements = []
@queries.each do |query|
- statement = query.statement.dup
- process_vars(statement, query.data_source)
- @statements << statement
+ @success = process_vars(query.statement_object)
end
@bind_vars ||= []
@@ -61,11 +58,7 @@ module Blazer
def refresh
@dashboard.queries.each do |query|
- data_source = Blazer.data_sources[query.data_source]
- statement = query.statement.dup
- process_vars(statement, query.data_source)
- Blazer.transform_statement.call(data_source, statement) if Blazer.transform_statement
- data_source.clear_cache(statement)
+ refresh_query(query)
end
redirect_to dashboard_path(@dashboard, params: variable_params(@dashboard))
end
diff --git a/app/controllers/blazer/queries_controller.rb b/app/controllers/blazer/queries_controller.rb
index d980747a..04e4c0a7 100644
--- a/app/controllers/blazer/queries_controller.rb
+++ b/app/controllers/blazer/queries_controller.rb
@@ -59,22 +59,19 @@ module Blazer
end
def show
- @statement = @query.statement.dup
- process_vars(@statement, @query.data_source)
+ @statement = @query.statement_object
+ @success = process_vars(@statement)
@smart_vars = {}
@sql_errors = []
- data_source = Blazer.data_sources[@query.data_source]
@bind_vars.each do |var|
- smart_var, error = parse_smart_variables(var, data_source)
+ smart_var, error = parse_smart_variables(var, @statement.data_source)
@smart_vars[var] = smart_var if smart_var
@sql_errors << error if error
end
@query.update!(status: "active") if @query.respond_to?(:status) && @query.status.in?(["archived", nil])
- Blazer.transform_statement.call(data_source, @statement) if Blazer.transform_statement
-
add_cohort_analysis_vars if @query.cohort_analysis?
end
@@ -90,12 +87,13 @@ module Blazer
data_source ||= params[:data_source]
@data_source = Blazer.data_sources[data_source]
- @statement = params[:statement]
+ @statement = Blazer::Statement.new(params[:statement], @data_source)
# before process_vars
- @cohort_analysis = Query.new(statement: @statement).cohort_analysis?
+ @cohort_analysis = @statement.cohort_analysis?
+
# fallback for now for users with open tabs
@var_params = request.request_parameters["variables"] || request.request_parameters
- process_vars(@statement, data_source, @var_params)
+ @success = process_vars(@statement, @var_params)
@only_chart = params[:only_chart]
@run_id = blazer_params[:run_id]
@@ -134,7 +132,7 @@ module Blazer
options = {user: blazer_user, query: @query, refresh_cache: params[:check], run_id: @run_id, async: Blazer.async}
if Blazer.async && request.format.symbol != :csv
- Blazer::RunStatementJob.perform_later(@data_source.id, @statement, options)
+ Blazer::RunStatementJob.perform_later(@data_source.id, @statement.statement, options.merge(values: @statement.values))
wait_start = Time.now
loop do
sleep(0.1)
@@ -142,7 +140,7 @@ module Blazer
break if @result || Time.now - wait_start > 3
end
else
- @result = Blazer::RunStatement.new.perform(@data_source, @statement, options)
+ @result = Blazer::RunStatement.new.perform(@statement, options)
end
if @result
@@ -172,12 +170,7 @@ module Blazer
end
def refresh
- data_source = Blazer.data_sources[@query.data_source]
- @statement = @query.statement.dup
- process_vars(@statement, @query.data_source)
- Blazer.transform_statement.call(data_source, @statement) if Blazer.transform_statement
- @statement = cohort_analysis_statement(data_source, @statement) if @query.cohort_analysis?
- data_source.clear_cache(@statement)
+ refresh_query(@query)
redirect_to query_path(@query, params: variable_params(@query))
end
@@ -288,6 +281,9 @@ module Blazer
render layout: false
end
format.csv do
+ # not ideal, but useful for testing
+ raise Error, @error if @error && Rails.env.test?
+
send_data csv_data(@columns, @rows, @data_source), type: "text/csv; charset=utf-8; header=present", disposition: "attachment; filename=\"#{@query.try(:name).try(:parameterize).presence || 'query'}.csv\""
end
end
@@ -369,34 +365,12 @@ module Blazer
end
def run_cohort_analysis
- unless @data_source.supports_cohort_analysis?
+ unless @statement.data_source.supports_cohort_analysis?
@cohort_error = "This data source does not support cohort analysis"
end
@show_cohort_rows = !params[:query_id] || @cohort_error
-
- unless @show_cohort_rows
- @statement = cohort_analysis_statement(@data_source, @statement)
- end
- end
-
- def cohort_analysis_statement(data_source, statement)
- @cohort_period = params["cohort_period"] || "week"
- @cohort_period = "week" unless ["day", "week", "month"].include?(@cohort_period)
-
- # for now
- @conversion_period = @cohort_period
- @cohort_days =
- case @cohort_period
- when "day"
- 1
- when "week"
- 7
- when "month"
- 30
- end
-
- data_source.cohort_analysis_statement(statement, period: @cohort_period, days: @cohort_days)
+ cohort_analysis_statement(@statement) unless @show_cohort_rows
end
def render_cohort_analysis
diff --git a/app/models/blazer/query.rb b/app/models/blazer/query.rb
index 0605f3e5..f51fd8f9 100644
--- a/app/models/blazer/query.rb
+++ b/app/models/blazer/query.rb
@@ -35,13 +35,19 @@ module Blazer
end
def variables
- variables = Blazer.extract_vars(statement)
+ # don't require data_source to be loaded
+ variables = Statement.new(statement).variables
variables += ["cohort_period"] if cohort_analysis?
variables
end
def cohort_analysis?
- /\/\*\s*cohort analysis\s*\*\//i.match?(statement)
+ # don't require data_source to be loaded
+ Statement.new(statement).cohort_analysis?
+ end
+
+ def statement_object
+ Statement.new(statement, data_source)
end
end
end
diff --git a/app/views/blazer/dashboards/show.html.erb b/app/views/blazer/dashboards/show.html.erb
index d93494ed..4afbfc60 100644
--- a/app/views/blazer/dashboards/show.html.erb
+++ b/app/views/blazer/dashboards/show.html.erb
@@ -39,7 +39,7 @@
</div>
</div>
<script>
- <%= blazer_js_var "data", {statement: @statements[i], query_id: query.id, data_source: query.data_source, only_chart: true, cohort_period: params[:cohort_period]} %>
+ <%= blazer_js_var "data", {statement: query.statement, query_id: query.id, variables: variable_params(query), only_chart: true, cohort_period: params[:cohort_period]} %>
runQuery(data, function (data) {
$("#chart-<%= i %>").html(data)
diff --git a/app/views/blazer/queries/show.html.erb b/app/views/blazer/queries/show.html.erb
index 0a80a0d4..d7c74a2c 100644
--- a/app/views/blazer/queries/show.html.erb
+++ b/app/views/blazer/queries/show.html.erb
@@ -1,5 +1,11 @@
<% blazer_title @query.name %>
+<% if @success %>
+ <% run_data = {statement: @query.statement, query_id: @query.id, variables: variable_params(@query)} %>
+ <% run_data.merge!(forecast: "t") if params[:forecast] %>
+ <% run_data.merge!(cohort_period: params[:cohort_period]) if params[:cohort_period] %>
+<% end %>
+
<div class="topbar">
<div class="container">
<div class="row" style="padding-top: 13px;">
@@ -14,7 +20,7 @@
<%= link_to "Fork", new_query_path(params: variable_params(@query).merge(fork_query_id: @query.id, data_source: @query.data_source, name: @query.name)), class: "btn btn-info" %>
<% if !@error && @success %>
- <%= button_to "Download", run_queries_path(query_id: @query.id, format: "csv", forecast: params[:forecast], cohort_period: params[:cohort_period]), params: {statement: @statement}, class: "btn btn-primary" %>
+ <%= button_to "Download", run_queries_path(format: "csv"), params: run_data, class: "btn btn-primary" %>
<% end %>
</div>
</div>
@@ -39,7 +45,7 @@
<%= render partial: "blazer/variables", locals: {action: query_path(@query)} %>
-<pre id="code"><code><%= @statement %></code></pre>
+<pre id="code"><code><%= @statement.display_statement %></code></pre>
<% if @success %>
<div id="results">
@@ -56,9 +62,7 @@
$("#results").addClass("query-error").html(message)
}
- <% data = {statement: @statement, query_id: @query.id, data_source: @query.data_source, variables: variable_params(@query)} %>
- <% data.merge!(forecast: "t") if params[:forecast] %>
- <%= blazer_js_var "data", data %>
+ <%= blazer_js_var "data", run_data %>
runQuery(data, showRun, showError)
</script>
diff --git a/lib/blazer.rb b/lib/blazer.rb
index 10a69e79..3c6eaf90 100644
--- a/lib/blazer.rb
+++ b/lib/blazer.rb
@@ -1,14 +1,18 @@
# dependencies
-require "csv"
-require "yaml"
require "chartkick"
require "safely/core"
+# stdlib
+require "csv"
+require "json"
+require "yaml"
+
# modules
require "blazer/version"
require "blazer/data_source"
require "blazer/result"
require "blazer/run_statement"
+require "blazer/statement"
# adapters
require "blazer/adapters/base_adapter"
@@ -78,6 +82,7 @@ module Blazer
self.images = false
self.override_csp = false
+ VARIABLE_MESSAGE = "Variable cannot be used in this position"
TIMEOUT_MESSAGE = "Query timed out :("
TIMEOUT_ERRORS = [
"canceling statement due to statement timeout", # postgres
@@ -130,6 +135,7 @@ module Blazer
end
end
+ # TODO move to Statement and remove in 3.0.0
def self.extract_vars(statement)
# strip commented out lines
# and regex {1} or {1,2}
@@ -150,9 +156,8 @@ module Blazer
ActiveSupport::Notifications.instrument("run_check.blazer", check_id: check.id, query_id: check.query.id, state_was: check.state) do |instrument|
# try 3 times on timeout errors
- data_source = data_sources[check.query.data_source]
- statement = check.query.statement
- Blazer.transform_statement.call(data_source, statement) if Blazer.transform_statement
+ statement = check.query.statement_object
+ data_source = statement.data_source
while tries <= 3
result = data_source.run_statement(statement, refresh_cache: true, check: check, query: check.query)
@@ -180,7 +185,8 @@ module Blazer
# TODO use proper logfmt
Rails.logger.info "[blazer check] query=#{check.query.name} state=#{check.state} rows=#{result.rows.try(:size)} error=#{result.error}"
- instrument[:statement] = statement
+ # should be no variables
+ instrument[:statement] = statement.bind_statement
instrument[:data_source] = data_source
instrument[:state] = check.state
instrument[:rows] = result.rows.try(:size)
diff --git a/lib/blazer/adapters/athena_adapter.rb b/lib/blazer/adapters/athena_adapter.rb
index 115564c9..c8b80b93 100644
--- a/lib/blazer/adapters/athena_adapter.rb
+++ b/lib/blazer/adapters/athena_adapter.rb
@@ -1,31 +1,50 @@
module Blazer
module Adapters
class AthenaAdapter < BaseAdapter
- def run_statement(statement, comment)
+ def run_statement(statement, comment, bind_params = [])
require "digest/md5"
columns = []
rows = []
error = nil
- query_options = {
- query_string: statement,
- # use token so we fetch cached results after query is run
- client_request_token: Digest::MD5.hexdigest([statement, data_source.id, settings["workgroup"]].compact.join("/")),
- query_execution_context: {
- database: database,
+ begin
+ # use empty? since any? doesn't work for [nil]
+ if !bind_params.empty?
+ request_token = Digest::MD5.hexdigest([statement, bind_params.to_json, data_source.id, settings["workgroup"]].compact.join("/"))
+ statement_name = "blazer_#{request_token}"
+ begin
+ client.create_prepared_statement({
+ statement_name: statement_name,
+ work_group: settings["workgroup"],
+ query_statement: statement
+ })
+ rescue Aws::Athena::Errors::InvalidRequestException => e
+ raise e unless e.message.include?("already exists in WorkGroup")
+ end
+ using_statement = bind_params.map { |v| data_source.quote(v) }.join(", ")
+ statement = "EXECUTE #{statement_name} USING #{using_statement}"
+ else
+ request_token = Digest::MD5.hexdigest([statement, data_source.id, settings["workgroup"]].compact.join("/"))
+ end
+
+ query_options = {
+ query_string: statement,
+ # use token so we fetch cached results after query is run
+ client_request_token: request_token,
+ query_execution_context: {
+ database: database,
+ }
}
- }
- if settings["output_location"]
- query_options[:result_configuration] = {output_location: settings["output_location"]}
- end
+ if settings["output_location"]
+ query_options[:result_configuration] = {output_location: settings["output_location"]}
+ end
- if settings["workgroup"]
- query_options[:work_group] = settings["workgroup"]
- end
+ if settings["workgroup"]
+ query_options[:work_group] = settings["workgroup"]
+ end
- begin
resp = client.start_query_execution(**query_options)
query_execution_id = resp.query_execution_id
@@ -111,12 +130,29 @@ module Blazer
"SELECT * FROM {table} LIMIT 10"
end
+ # https://docs.aws.amazon.com/athena/latest/ug/select.html#select-escaping
+ def quoting
+ :single_quote_escape
+ end
+
+ # https://docs.aws.amazon.com/athena/latest/ug/querying-with-prepared-statements.html
+ def parameter_binding
+ engine_version > 1 ? :positional : nil
+ end
+
private
def database
@database ||= settings["database"] || "default"
end
+ # note: this setting is experimental
+ # it does *not* need to be set to use engine version 2
+ # prepared statements must be manually deleted if enabled
+ def engine_version
+ @engine_version ||= (settings["engine_version"] || 1).to_i
+ end
+
def fetch_error(query_execution_id)
client.get_query_execution(
query_execution_id: query_execution_id
diff --git a/lib/blazer/adapters/base_adapter.rb b/lib/blazer/adapters/base_adapter.rb
index 631c943e..231ef8a7 100644
--- a/lib/blazer/adapters/base_adapter.rb
+++ b/lib/blazer/adapters/base_adapter.rb
@@ -8,7 +8,22 @@ module Blazer
end
def run_statement(statement, comment)
- # the one required method
+ # required
+ end
+
+ def quoting
+ # required, how to quote variables
+ # :backslash_escape - single quote strings and convert ' to \' and \ to \\
+ # :single_quote_escape - single quote strings and convert ' to ''
+ # ->(value) { ... } - custom method
+ end
+
+ def parameter_binding
+ # optional, but recommended when possible for security
+ # if specified, quoting is only used for display
+ # :positional - ?
+ # :numeric - $1
+ # ->(statement, values) { ... } - custom method
end
def tables
diff --git a/lib/blazer/adapters/bigquery_adapter.rb b/lib/blazer/adapters/bigquery_adapter.rb
index 49db073c..b035da82 100644
--- a/lib/blazer/adapters/bigquery_adapter.rb
+++ b/lib/blazer/adapters/bigquery_adapter.rb
@@ -1,13 +1,13 @@
module Blazer
module Adapters
class BigQueryAdapter < BaseAdapter
- def run_statement(statement, comment)
+ def run_statement(statement, comment, bind_params)
columns = []
rows = []
error = nil
begin
- results = bigquery.query(statement)
+ results = bigquery.query(statement, params: bind_params)
# complete? was removed in google-cloud-bigquery 0.29.0
# code is for backward compatibility
@@ -19,6 +19,7 @@ module Blazer
end
rescue => e
error = e.message
+ error = Blazer::VARIABLE_MESSAGE if error.include?("Syntax error: Unexpected \"?\"")
end
[columns, rows, error]
@@ -42,6 +43,16 @@ module Blazer
"SELECT * FROM `{table}` LIMIT 10"
end
+ # https://cloud.google.com/bigquery/docs/reference/standard-sql/lexical#string_and_bytes_literals
+ def quoting
+ :backslash_escape
+ end
+
+ # https://cloud.google.com/bigquery/docs/parameterized-queries
+ def parameter_binding
+ :positional
+ end
+
private
def bigquery
diff --git a/lib/blazer/adapters/cassandra_adapter.rb b/lib/blazer/adapters/cassandra_adapter.rb
index 3228c737..736c3c1c 100644
--- a/lib/blazer/adapters/cassandra_adapter.rb
+++ b/lib/blazer/adapters/cassandra_adapter.rb
@@ -1,28 +1,29 @@
module Blazer
module Adapters
class CassandraAdapter < BaseAdapter
- def run_statement(statement, comment)
+ def run_statement(statement, comment, bind_params)
columns = []
rows = []
error = nil
begin
- response = session.execute("#{statement} /*#{comment}*/")
+ response = session.execute("#{statement} /*#{comment}*/", arguments: bind_params)
rows = response.map { |r| r.values }
columns = rows.any? ? response.first.keys : []
rescue => e
error = e.message
+ error = Blazer::VARIABLE_MESSAGE if error.include?("no viable alternative at input '?'")
end
[columns, rows, error]
end
def tables
- session.execute("SELECT table_name FROM system_schema.tables WHERE keyspace_name = '#{keyspace}'").map { |r| r["table_name"] }
+ session.execute("SELECT table_name FROM system_schema.tables WHERE keyspace_name = #{data_source.quote(keyspace)}").map { |r| r["table_name"] }
end
def schema
- result = session.execute("SELECT keyspace_name, table_name, column_name, type, position FROM system_schema.columns WHERE keyspace_name = '#{keyspace}'")
+ result = session.execute("SELECT keyspace_name, table_name, column_name, type, position FROM system_schema.columns WHERE keyspace_name = #{data_source.quote(keyspace)}")
result.map(&:values).group_by { |r| [r[0], r[1]] }.map { |k, vs| {schema: k[0], table: k[1], columns: vs.sort_by { |v| v[2] }.map { |v| {name: v[2], data_type: v[3]} }} }
end
@@ -30,6 +31,16 @@ module Blazer
"SELECT * FROM {table} LIMIT 10"
end
+ # https://docs.datastax.com/en/cql-oss/3.3/cql/cql_reference/escape_char_r.html
+ def quoting
+ :single_quote_escape
+ end
+
+ # https://docs.datastax.com/en/developer/nodejs-driver/3.0/features/parameterized-queries/
+ def parameter_binding
+ :positional
+ end
+
private
def cluster
diff --git a/lib/blazer/adapters/drill_adapter.rb b/lib/blazer/adapters/drill_adapter.rb
index 7db9c623..d5358d66 100644
--- a/lib/blazer/adapters/drill_adapter.rb
+++ b/lib/blazer/adapters/drill_adapter.rb
@@ -18,6 +18,16 @@ module Blazer
[columns, rows, error]
end
+ # https://drill.apache.org/docs/lexical-structure/#string
+ def quoting
+ :single_quote_escape
+ end
+
+ # https://issues.apache.org/jira/browse/DRILL-5079
+ def parameter_binding
+ # not supported
+ end
+
private
def drill
diff --git a/lib/blazer/adapters/druid_adapter.rb b/lib/blazer/adapters/druid_adapter.rb
index a99e6619..1eb3a4af 100644
--- a/lib/blazer/adapters/druid_adapter.rb
+++ b/lib/blazer/adapters/druid_adapter.rb
@@ -3,7 +3,7 @@ module Blazer
class DruidAdapter < BaseAdapter
TIMESTAMP_REGEX = /\A\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z\z/
- def run_statement(statement, comment)
+ def run_statement(statement, comment, bind_params)
require "json"
require "net/http"
require "uri"
@@ -12,10 +12,28 @@ module Blazer
rows = []
error = nil
+ params =
+ bind_params.map do |v|
+ type =
+ case v
+ when Integer
+ "BIGINT"
+ when Float
+ "DOUBLE"
+ when ActiveSupport::TimeWithZone
+ v = (v.to_f * 1000).round
+ "TIMESTAMP"
+ else
+ "VARCHAR"
+ end
+ {type: type, value: v}
+ end
+
header = {"Content-Type" => "application/json", "Accept" => "application/json"}
timeout = data_source.timeout ? data_source.timeout.to_i : 300
data = {
query: statement,
+ parameters: params,
context: {
timeout: timeout * 1000
}
@@ -31,6 +49,8 @@ module Blazer
error = response["errorMessage"] || "Unknown error: #{response.inspect}"
if error.include?("timed out")
error = Blazer::TIMEOUT_MESSAGE
+ elsif error.include?("Encountered \"?\" at")
+ error = Blazer::VARIABLE_MESSAGE
end
else
columns = (response.first || {}).keys
@@ -66,6 +86,17 @@ module Blazer
def preview_statement
"SELECT * FROM {table} LIMIT 10"
end
+
+ # https://druid.apache.org/docs/latest/querying/sql.html#identifiers-and-literals
+ # docs only mention double quotes
+ def quoting
+ :single_quote_escape
+ end
+
+ # https://druid.apache.org/docs/latest/querying/sql.html#dynamic-parameters
+ def parameter_binding
+ :positional
+ end
end
end
end
diff --git a/lib/blazer/adapters/elasticsearch_adapter.rb b/lib/blazer/adapters/elasticsearch_adapter.rb
index 7e469d47..6488ae02 100644
--- a/lib/blazer/adapters/elasticsearch_adapter.rb
+++ b/lib/blazer/adapters/elasticsearch_adapter.rb
@@ -1,13 +1,13 @@
module Blazer
module Adapters
class ElasticsearchAdapter < BaseAdapter
- def run_statement(statement, comment)
+ def run_statement(statement, comment, bind_params)
columns = []
rows = []
error = nil
begin
- response = client.transport.perform_request("POST", endpoint, {}, {query: "#{statement} /*#{comment}*/"}).body
+ response = client.transport.perform_request("POST", endpoint, {}, {query: "#{statement} /*#{comment}*/", params: bind_params}).body
columns = response["columns"].map { |v| v["name"] }
# Elasticsearch does not differentiate between dates and times
date_indexes = response["columns"].each_index.select { |i| ["date", "datetime"].include?(response["columns"][i]["type"]) }
@@ -21,6 +21,7 @@ module Blazer
end
rescue => e
error = e.message
+ error = Blazer::VARIABLE_MESSAGE if error.include?("mismatched input '?'")
end
[columns, rows, error]
@@ -36,6 +37,16 @@ module Blazer
"SELECT * FROM \"{table}\" LIMIT 10"
end
+ # https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-lexical-structure.html#sql-syntax-string-literals
+ def quoting
+ :single_quote_escape
+ end
+
+ # https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-rest-params.html
+ def parameter_binding
+ :positional
+ end
+
protected
def endpoint
diff --git a/lib/blazer/adapters/hive_adapter.rb b/lib/blazer/adapters/hive_adapter.rb
index bd8cab16..4f2b4c81 100644
--- a/lib/blazer/adapters/hive_adapter.rb
+++ b/lib/blazer/adapters/hive_adapter.rb
@@ -25,6 +25,16 @@ module Blazer
"SELECT * FROM {table} LIMIT 10"
end
+ # https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-StringsstringStrings
+ def quoting
+ :backslash_escape
+ end
+
+ # has variable substitution, but sets for session
+ # https://cwiki.apache.org/confluence/display/Hive/LanguageManual+VariableSubstitution
+ def parameter_binding
+ end
+
protected
def client
diff --git a/lib/blazer/adapters/ignite_adapter.rb b/lib/blazer/adapters/ignite_adapter.rb
index f345bf10..3b480f54 100644
--- a/lib/blazer/adapters/ignite_adapter.rb
+++ b/lib/blazer/adapters/ignite_adapter.rb
@@ -1,13 +1,13 @@
module Blazer
module Adapters
class IgniteAdapter < BaseAdapter
- def run_statement(statement, comment)
+ def run_statement(statement, comment, bind_params)
columns = []
rows = []
error = nil
begin
- result = client.query("#{statement} /*#{comment}*/", schema: default_schema, statement_type: :select, timeout: data_source.timeout)
+ result = client.query("#{statement} /*#{comment}*/", bind_params, schema: default_schema, statement_type: :select, timeout: data_source.timeout)
columns = result.any? ? result.first.keys : []
rows = result.map(&:values)
rescue => e
@@ -37,6 +37,16 @@ module Blazer
# result.rows.group_by { |r| [r[0], r[1]] }.map { |k, vs| {schema: k[0], table: k[1], columns: vs.sort_by { |v| v[2] }.map { |v| {name: v[2], data_type: v[3]} }} }.sort_by { |t| [t[:schema] == default_schema ? "" : t[:schema], t[:table]] }
# end
+ def quoting
+ :single_quote_escape
+ end
+
+ # query arguments
+ # https://ignite.apache.org/docs/latest/binary-client-protocol/sql-and-scan-queries#op_query_sql
+ def parameter_binding
+ :positional
+ end
+
private
def default_schema
diff --git a/lib/blazer/adapters/influxdb_adapter.rb b/lib/blazer/adapters/influxdb_adapter.rb
index ab59aa9b..1bab4f6d 100644
--- a/lib/blazer/adapters/influxdb_adapter.rb
+++ b/lib/blazer/adapters/influxdb_adapter.rb
@@ -38,6 +38,15 @@ module Blazer
"SELECT * FROM {table} LIMIT 10"
end
+ # https://docs.influxdata.com/influxdb/v1.8/query_language/spec/#strings
+ def quoting
+ :backslash_escape
+ end
+
+ def parameter_binding
+ # not supported
+ end
+
protected
def client
diff --git a/lib/blazer/adapters/mongodb_adapter.rb b/lib/blazer/adapters/mongodb_adapter.rb
index 350897f9..d4f188a0 100644
--- a/lib/blazer/adapters/mongodb_adapter.rb
+++ b/lib/blazer/adapters/mongodb_adapter.rb
@@ -25,6 +25,10 @@ module Blazer
"db.{table}.find().limit(10)"
end
+ def quoting
+ :backslash_escape
+ end
+
protected
def client
diff --git a/lib/blazer/adapters/neo4j_adapter.rb b/lib/blazer/adapters/neo4j_adapter.rb
index c2800dc6..35c79634 100644
--- a/lib/blazer/adapters/neo4j_adapter.rb
+++ b/lib/blazer/adapters/neo4j_adapter.rb
@@ -1,13 +1,13 @@
module Blazer
module Adapters
class Neo4jAdapter < BaseAdapter
- def run_statement(statement, comment)
+ def run_statement(statement, comment, bind_params)
columns = []
rows = []
error = nil
begin
- result = session.query("#{statement} /*#{comment}*/")
+ result = session.query("#{statement} /*#{comment}*/", bind_params)
columns = result.columns.map(&:to_s)
rows = []
result.each do |row|
@@ -19,6 +19,7 @@ module Blazer
end
rescue => e
error = e.message
+ error = Blazer::VARIABLE_MESSAGE if error.include?("Invalid input '$'")
end
[columns, rows, error]
@@ -33,6 +34,20 @@ module Blazer
"MATCH (n:{table}) RETURN n LIMIT 10"
end
+ # https://neo4j.com/docs/cypher-manual/current/syntax/expressions/#cypher-expressions-string-literals
+ def quoting
+ :backslash_escape
+ end
+
+ def parameter_binding
+ proc do |statement, variables|
+ variables.each_key do |k|
+ statement = statement.gsub("{#{k}}") { "$#{k} " }
+ end
+ [statement, variables]
+ end
+ end
+
protected
def session
diff --git a/lib/blazer/adapters/opensearch_adapter.rb b/lib/blazer/adapters/opensearch_adapter.rb
index 5f6032e7..bca6181b 100644
--- a/lib/blazer/adapters/opensearch_adapter.rb
+++ b/lib/blazer/adapters/opensearch_adapter.rb
@@ -38,6 +38,10 @@ module Blazer
"SELECT * FROM `{table}` LIMIT 10"
end
+ def quoting
+ # unknown
+ end
+
protected
def client
diff --git a/lib/blazer/adapters/presto_adapter.rb b/lib/blazer/adapters/presto_adapter.rb
index 1ee0eddb..ab5bdc3c 100644
--- a/lib/blazer/adapters/presto_adapter.rb
+++ b/lib/blazer/adapters/presto_adapter.rb
@@ -25,6 +25,15 @@ module Blazer
"SELECT * FROM {table} LIMIT 10"
end
+ def quoting
+ :single_quote_escape
+ end
+
+ # TODO support prepared statements - https://prestodb.io/docs/current/sql/prepare.html
+ # feature request for variables - https://github.com/prestodb/presto/issues/5918
+ def parameter_binding
+ end
+
protected
def client
diff --git a/lib/blazer/adapters/salesforce_adapter.rb b/lib/blazer/adapters/salesforce_adapter.rb
index a0630ead..fdef968f 100644
--- a/lib/blazer/adapters/salesforce_adapter.rb
+++ b/lib/blazer/adapters/salesforce_adapter.rb
@@ -35,6 +35,11 @@ module Blazer
"SELECT Id FROM {table} LIMIT 10"
end
+ # https://developer.salesforce.com/docs/atlas.en-us.soql_sosl.meta/soql_sosl/sforce_api_calls_soql_select_quotedstringescapes.htm
+ def quoting
+ :backslash_escape
+ end
+
protected
def client
diff --git a/lib/blazer/adapters/snowflake_adapter.rb b/lib/blazer/adapters/snowflake_adapter.rb
index d14c41cf..46e6dea8 100644
--- a/lib/blazer/adapters/snowflake_adapter.rb
+++ b/lib/blazer/adapters/snowflake_adapter.rb
@@ -68,6 +68,15 @@ module Blazer
def cancel(run_id)
# todo
end
+
+ # https://docs.snowflake.com/en/sql-reference/data-types-text.html#escape-sequences
+ def quoting
+ :backslash_escape
+ end
+
+ def parameter_binding
+ # TODO
+ end
end
end
end
diff --git a/lib/blazer/adapters/soda_adapter.rb b/lib/blazer/adapters/soda_adapter.rb
index 59b15a45..92050266 100644
--- a/lib/blazer/adapters/soda_adapter.rb
+++ b/lib/blazer/adapters/soda_adapter.rb
@@ -95,6 +95,11 @@ module Blazer
def tables
["all"]
end
+
+ # https://dev.socrata.com/docs/datatypes/text.html
+ def quoting
+ :single_quote_escape
+ end
end
end
end
diff --git a/lib/blazer/adapters/spark_adapter.rb b/lib/blazer/adapters/spark_adapter.rb
index 486982b0..0ac7b2fe 100644
--- a/lib/blazer/adapters/spark_adapter.rb
+++ b/lib/blazer/adapters/spark_adapter.rb
@@ -4,6 +4,11 @@ module Blazer
def tables
client.execute("SHOW TABLES").map { |r| r["tableName"] }
end
+
+ # https://spark.apache.org/docs/latest/sql-ref-literals.html
+ def quoting
+ :backslash_escape
+ end
end
end
end
diff --git a/lib/blazer/adapters/sql_adapter.rb b/lib/blazer/adapters/sql_adapter.rb
index 3df5a7ee..6908c695 100644
--- a/lib/blazer/adapters/sql_adapter.rb
+++ b/lib/blazer/adapters/sql_adapter.rb
@@ -15,7 +15,7 @@ module Blazer
end
end
- def run_statement(statement, comment)
+ def run_statement(statement, comment, bind_params = [])
columns = []
rows = []
error = nil
@@ -24,7 +24,8 @@ module Blazer
in_transaction do
set_timeout(data_source.timeout) if data_source.timeout
- result = select_all("#{statement} /*#{comment}*/")
+ binds = bind_params.map { |v| ActiveRecord::Relation::QueryAttribute.new(nil, v, ActiveRecord::Type::Value.new) }
+ result = connection_model.connection.select_all("#{statement} /*#{comment}*/", nil, binds)
columns = result.columns
result.rows.each do |untyped_row|
rows << (result.column_types.empty? ? untyped_row : columns.each_with_index.map { |c, i| untyped_row[i] && result.column_types[c] ? result.column_types[c].send(:cast_value, untyped_row[i]) : untyped_row[i] })
@@ -33,6 +34,7 @@ module Blazer
rescue => e
error = e.message.sub(/.+ERROR: /, "")
error = Blazer::TIMEOUT_MESSAGE if Blazer::TIMEOUT_ERRORS.any? { |e| error.include?(e) }
+ error = Blazer::VARIABLE_MESSAGE if error.include?("syntax error at or near \"$") || error.include?("Incorrect syntax near '@") || error.include?("your MySQL server version for the right syntax to use near '?")
reconnect if error.include?("PG::ConnectionBad")
end
@@ -156,7 +158,7 @@ module Blazer
# WITH not an optimization fence in Postgres 12+
statement = <<~SQL
WITH query AS (
- #{statement}
+ {placeholder}
),
cohorts AS (
SELECT user_id, MIN(#{cohort_column}) AS cohort_time FROM query
@@ -183,6 +185,27 @@ module Blazer
connection_model.send(:sanitize_sql_array, params)
end
+ def quoting
+ ->(value) { connection_model.connection.quote(value) }
+ end
+
+ # Redshift adapter silently ignores binds
+ def parameter_binding
+ if postgresql? || sqlite?
+ :numeric
+ elsif mysql? && connection_model.connection.prepared_statements?
+ # Active Record silently ignores binds with MySQL when prepared statements are disabled
+ :positional
+ elsif sqlserver?
+ proc do |statement, variables|
+ variables.each_with_index do |(k, _), i|
+ statement = statement.gsub("{#{k}}", "@#{i} ")
+ end
+ [statement, variables.values]
+ end
+ end
+ end
+
protected
def select_all(statement, params = [])
@@ -207,6 +230,10 @@ module Blazer
["MySQL", "Mysql2", "Mysql2Spatial"].include?(adapter_name)
end
+ def sqlite?
+ ["SQLite"].include?(adapter_name)
+ end
+
def sqlserver?
["SQLServer", "tinytds", "mssql"].include?(adapter_name)
end
diff --git a/lib/blazer/data_source.rb b/lib/blazer/data_source.rb
index b9de0b08..411a9486 100644
--- a/lib/blazer/data_source.rb
+++ b/lib/blazer/data_source.rb
@@ -89,7 +89,19 @@ module Blazer
Blazer.cache.delete(run_cache_key(run_id))
end
+ def sub_variables(statement, vars)
+ statement = statement.dup
+ vars.each do |var, value|
+ # use block form to disable back-references
+ statement.gsub!("{#{var}}") { quote(value) }
+ end
+ statement
+ end
+
def run_statement(statement, options = {})
+ statement = Statement.new(statement, self) if statement.is_a?(String)
+ statement.bind unless statement.bind_statement
+
async = options[:async]
result = nil
if cache_mode != "off"
@@ -118,7 +130,7 @@ module Blazer
if options[:run_id]
comment << ",run_id:#{options[:run_id]}"
end
- result = run_statement_helper(statement, comment, async ? options[:run_id] : nil)
+ result = run_statement_helper(statement, comment, async ? options[:run_id] : nil, options)
end
result
@@ -133,13 +145,68 @@ module Blazer
end
def statement_cache_key(statement)
- cache_key(["statement", id, Digest::MD5.hexdigest(statement.to_s.gsub("\r\n", "\n"))])
+ cache_key(["statement", id, Digest::MD5.hexdigest(statement.bind_statement.to_s.gsub("\r\n", "\n") + statement.bind_values.sort_by { |k, _| k }.to_json)])
end
def run_cache_key(run_id)
cache_key(["run", run_id])
end
+ def quote(value)
+ if quoting == :backslash_escape || quoting == :single_quote_escape
+ # only need to support types generated by process_vars
+ if value.is_a?(Integer) || value.is_a?(Float)
+ value.to_s
+ elsif value.nil?
+ "NULL"
+ else
+ value = value.to_formatted_s(:db) if value.is_a?(ActiveSupport::TimeWithZone)
+
+ if quoting == :backslash_escape
+ "'#{value.gsub("\\") { "\\\\" }.gsub("'") { "\\'" }}'"
+ else
+ "'#{value.gsub("'", "''")}'"
+ end
+ end
+ elsif quoting.respond_to?(:call)
+ quoting.call(value)
+ elsif quoting.nil?
+ raise Blazer::Error, "Quoting not specified"
+ else
+ raise Blazer::Error, "Unknown quoting"
+ end
+ end
+
+ def bind_params(statement, variables)
+ if parameter_binding == :positional
+ locations = []
+ variables.each do |k, v|
+ i = 0
+ while (idx = statement.index("{#{k}}", i))
+ locations << [v, idx]
+ i = idx + 1
+ end
+ end
+ variables.each do |k, v|
+ statement = statement.gsub("{#{k}}", "?")
+ end
+ [statement, locations.sort_by(&:last).map(&:first)]
+ elsif parameter_binding == :numeric
+ variables.each_with_index do |(k, v), i|
+ # add trailing space if followed by digit
+ # try to keep minimal to avoid fixing invalid queries like SELECT{var}
+ statement = statement.gsub(/#{Regexp.escape("{#{k}}")}(\d)/, "$#{i + 1} \\1").gsub("{#{k}}", "$#{i + 1}")
+ end
+ [statement, variables.values]
+ elsif parameter_binding.respond_to?(:call)
+ parameter_binding.call(statement, variables)
+ elsif parameter_binding.nil?
+ [sub_variables(statement, variables), []]
+ else
+ raise Blazer::Error, "Unknown bind parameters"
+ end
+ end
+
protected
def adapter_instance
@@ -157,9 +224,22 @@ module Blazer
end
end
- def run_statement_helper(statement, comment, run_id)
+ def quoting
+ @quoting ||= adapter_instance.quoting
+ end
+
+ def parameter_binding
+ @parameter_binding ||= adapter_instance.parameter_binding
+ end
+
+ def run_statement_helper(statement, comment, run_id, options)
start_time = Time.now
- columns, rows, error = adapter_instance.run_statement(statement, comment)
+ columns, rows, error =
+ if adapter_instance.parameter_binding
+ adapter_instance.run_statement(statement.bind_statement, comment, statement.bind_values)
+ else
+ adapter_instance.run_statement(statement.bind_statement, comment)
+ end
duration = Time.now - start_time
cache_data = nil
@@ -168,7 +248,7 @@ module Blazer
cache_data = Marshal.dump([columns, rows, error, cache ? Time.now : nil]) rescue nil
end
- if cache && cache_data && adapter_instance.cachable?(statement)
+ if cache && cache_data && adapter_instance.cachable?(statement.bind_statement)
Blazer.cache.write(statement_cache_key(statement), cache_data, expires_in: cache_expires_in.to_f * 60)
end
diff --git a/lib/blazer/run_statement.rb b/lib/blazer/run_statement.rb
index 8ba3e163..bc5727f5 100644
--- a/lib/blazer/run_statement.rb
+++ b/lib/blazer/run_statement.rb
@@ -1,12 +1,16 @@
module Blazer
class RunStatement
- def perform(data_source, statement, options = {})
+ def perform(statement, options = {})
query = options[:query]
- Blazer.transform_statement.call(data_source, statement) if Blazer.transform_statement
+
+ data_source = statement.data_source
+ statement.bind
# audit
if Blazer.audit
- audit = Blazer::Audit.new(statement: statement)
+ audit_statement = statement.bind_statement
+ audit_statement += "\n\n#{statement.bind_values.to_json}" if statement.bind_values.any?
+ audit = Blazer::Audit.new(statement: audit_statement)
audit.query = query
audit.data_source = data_source.id
audit.user = options[:user]
diff --git a/lib/blazer/run_statement_job.rb b/lib/blazer/run_statement_job.rb
index 8bc43921..7d7bfef4 100644
--- a/lib/blazer/run_statement_job.rb
+++ b/lib/blazer/run_statement_job.rb
@@ -3,10 +3,12 @@ module Blazer
self.queue_adapter = :async
def perform(data_source_id, statement, options)
- data_source = Blazer.data_sources[data_source_id]
+ statement = Blazer::Statement.new(statement, data_source_id)
+ statement.values = options.delete(:values)
+ data_source = statement.data_source
begin
ActiveRecord::Base.connection_pool.with_connection do
- Blazer::RunStatement.new.perform(data_source, statement, options)
+ Blazer::RunStatement.new.perform(statement, options)
end
rescue Exception => e
Blazer::Result.new(data_source, [], [], "Unknown error", nil, false)
diff --git a/lib/blazer/statement.rb b/lib/blazer/statement.rb
new file mode 100644
index 00000000..994d733e
--- /dev/null
+++ b/lib/blazer/statement.rb
@@ -0,0 +1,75 @@
+module Blazer
+ class Statement
+ attr_reader :statement, :data_source, :bind_statement, :bind_values
+ attr_accessor :values
+
+ def initialize(statement, data_source = nil)
+ @statement = statement
+ @data_source = data_source.is_a?(String) ? Blazer.data_sources[data_source] : data_source
+ @values = {}
+ end
+
+ def variables
+ @variables ||= Blazer.extract_vars(statement)
+ end
+
+ def add_values(var_params)
+ variables.each do |var|
+ value = var_params[var].presence
+ value = nil unless value.is_a?(String) # ignore arrays and hashes
+ if value
+ if ["start_time", "end_time"].include?(var)
+ value = value.to_s.gsub(" ", "+") # fix for Quip bug
+ end
+
+ if var.end_with?("_at")
+ begin
+ value = Blazer.time_zone.parse(value)
+ rescue
+ # do nothing
+ end
+ end
+
+ unless value.is_a?(ActiveSupport::TimeWithZone)
+ if value =~ /\A\d+\z/
+ value = value.to_i
+ elsif value =~ /\A\d+\.\d+\z/
+ value = value.to_f
+ end
+ end
+ end
+ value = Blazer.transform_variable.call(var, value) if Blazer.transform_variable
+ @values[var] = value
+ end
+ end
+
+ def cohort_analysis?
+ /\/\*\s*cohort analysis\s*\*\//i.match?(statement)
+ end
+
+ def apply_cohort_analysis(period:, days:)
+ @statement = data_source.cohort_analysis_statement(statement, period: period, days: days).sub("{placeholder}") { statement }
+ end
+
+ # should probably transform before cohort analysis
+ # but keep previous order for now
+ def transformed_statement
+ statement = self.statement.dup
+ Blazer.transform_statement.call(data_source, statement) if Blazer.transform_statement
+ statement
+ end
+
+ def bind
+ @bind_statement, @bind_values = data_source.bind_params(transformed_statement, values)
+ end
+
+ def display_statement
+ data_source.sub_variables(transformed_statement, values)
+ end
+
+ def clear_cache
+ bind if bind_statement.nil?
+ data_source.clear_cache(self)
+ end
+ end
+end
diff --git a/test/adapters/athena_test.rb b/test/adapters/athena_test.rb
index 684cd703..f8211071 100644
--- a/test/adapters/athena_test.rb
+++ b/test/adapters/athena_test.rb
@@ -10,4 +10,64 @@ class AthenaTest < ActionDispatch::IntegrationTest
def test_run
assert_result [{"hello" => "world"}], "SELECT 'world' AS hello"
end
+
+ def test_audit
+ if engine_version > 1
+ assert_audit "SELECT ? AS hello\n\n[\"world\"]", "SELECT {var} AS hello", var: "world"
+ else
+ assert_audit "SELECT 'world' AS hello", "SELECT {var} AS hello", var: "world"
+ end
+ end
+
+ def test_string
+ assert_result [{"hello" => "world"}], "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_integer
+ assert_result [{"hello" => "1"}], "SELECT {var} AS hello", var: "1"
+ end
+
+ def test_float
+ assert_result [{"hello" => "1.5"}], "SELECT {var} AS hello", var: "1.5"
+ end
+
+ def test_time
+ assert_result [{"hello" => "2022-01-01 08:00:00"}], "SELECT {created_at} AS hello", created_at: "2022-01-01 08:00:00"
+ end
+
+ def test_nil
+ assert_result [{"hello" => nil}], "SELECT {var} AS hello", var: ""
+ end
+
+ def test_single_quote
+ assert_result [{"hello" => "'"}], "SELECT {var} AS hello", var: "'"
+ end
+
+ def test_double_quote
+ assert_result [{"hello" => '"'}], "SELECT {var} AS hello", var: '"'
+ end
+
+ def test_backslash
+ assert_result [{"hello" => "\\"}], "SELECT {var} AS hello", var: "\\"
+ end
+
+ def test_bad_position
+ if engine_version > 1
+ assert_error "Exception parsing query", "SELECT 'world' AS {var}", var: "hello"
+ else
+ assert_error "mismatched input", "SELECT 'world' AS {var}", var: "hello"
+ end
+ end
+
+ def test_quoted
+ if engine_version > 1
+ assert_error "Incorrect number of parameters: expected 0 but found 1", "SELECT '{var}' AS hello", var: "world"
+ end
+ end
+
+ private
+
+ def engine_version
+ Blazer.data_sources[data_source].settings["engine_version"].to_i
+ end
end
diff --git a/test/adapters/bigquery_test.rb b/test/adapters/bigquery_test.rb
index 255982e0..ca3d16ea 100644
--- a/test/adapters/bigquery_test.rb
+++ b/test/adapters/bigquery_test.rb
@@ -10,4 +10,50 @@ class BigqueryTest < ActionDispatch::IntegrationTest
def test_run
assert_result [{"hello" => "world"}], "SELECT 'world' AS hello"
end
+
+ def test_audit
+ assert_audit "SELECT ? AS hello\n\n[\"world\"]", "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_string
+ assert_result [{"hello" => "world"}], "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_integer
+ assert_result [{"hello" => "1"}], "SELECT {var} AS hello", var: "1"
+ end
+
+ def test_float
+ assert_result [{"hello" => "1.5"}], "SELECT {var} AS hello", var: "1.5"
+ end
+
+ def test_time
+ assert_result [{"hello" => "2022-01-01 08:00:00 UTC"}], "SELECT {created_at} AS hello", created_at: "2022-01-01 08:00:00"
+ end
+
+ # TODO fix
+ def test_nil
+ assert_error "nil params are not supported, must assign optional type", "SELECT {var} AS hello", var: ""
+ end
+
+ def test_single_quote
+ assert_result [{"hello" => "'"}], "SELECT {var} AS hello", var: "'"
+ end
+
+ def test_double_quote
+ assert_result [{"hello" => '"'}], "SELECT {var} AS hello", var: '"'
+ end
+
+ def test_backslash
+ assert_result [{"hello" => "\\"}], "SELECT {var} AS hello", var: "\\"
+ end
+
+ def test_bad_position
+ assert_bad_position "SELECT 'world' AS {var}", var: "hello"
+ end
+
+ # does not raise error for too many params
+ def test_quoted
+ assert_result [{"hello" => "?"}], "SELECT '{var}' AS hello", var: "world"
+ end
end
diff --git a/test/adapters/cassandra_test.rb b/test/adapters/cassandra_test.rb
index 00e84b35..7223cd6c 100644
--- a/test/adapters/cassandra_test.rb
+++ b/test/adapters/cassandra_test.rb
@@ -17,12 +17,45 @@ class CassandraTest < ActionDispatch::IntegrationTest
session = cluster.connect("blazer_test")
session.execute("DROP TABLE IF EXISTS items")
- session.execute("CREATE TABLE items (id int, name text, PRIMARY KEY (id))")
+ session.execute("CREATE TABLE items (id int, hello text, PRIMARY KEY (id))")
+ session.execute("INSERT INTO items (id, hello) VALUES (1, 'world')")
+ session.execute("INSERT INTO items (id, hello) VALUES (2, '''')")
+ session.execute("INSERT INTO items (id, hello) VALUES (3, '\"')")
+ session.execute("INSERT INTO items (id, hello) VALUES (4, '\\')")
true
end
end
+ def test_tables
+ assert_equal ["items"], tables
+ end
+
def test_run
- assert_result [{"count" => "0"}], "SELECT COUNT(*) FROM items"
+ expected = [{"hello" => 'world'}]
+ assert_result expected, "SELECT hello FROM items WHERE hello = 'world' ALLOW FILTERING"
+ end
+
+ def test_audit
+ expected = "SELECT hello FROM items WHERE hello = ? ALLOW FILTERING\n\n[\"world\"]"
+ assert_audit expected, "SELECT hello FROM items WHERE hello = {var} ALLOW FILTERING", var: "world"
+ end
+
+ def test_single_quote
+ expected = [{"hello" => "'"}]
+ assert_result expected, "SELECT hello FROM items WHERE hello = {var} ALLOW FILTERING", var: "'"
+ end
+
+ def test_double_quote
+ expected = [{"hello" => '"'}]
+ assert_result expected, "SELECT hello FROM items WHERE hello = {var} ALLOW FILTERING", var: '"'
+ end
+
+ def test_backslash
+ expected = [{"hello" => "\\"}]
+ assert_result expected, "SELECT hello FROM items WHERE hello = {var} ALLOW FILTERING", var: "\\"
+ end
+
+ def test_bad_position
+ assert_bad_position "SELECT hello FROM items WHERE hello {var} 'world' ALLOW FILTERING", var: "="
end
end
diff --git a/test/adapters/drill_test.rb b/test/adapters/drill_test.rb
index 2132a8e4..bbbd5e5e 100644
--- a/test/adapters/drill_test.rb
+++ b/test/adapters/drill_test.rb
@@ -10,4 +10,20 @@ class DrillTest < ActionDispatch::IntegrationTest
def test_run
assert_result [{"hello" => "world"}], "SELECT 'world' AS hello"
end
+
+ def test_audit
+ assert_audit "SELECT 'world' AS hello", "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_single_quote
+ assert_result [{"hello" => "'"}], "SELECT {var} AS hello", var: "'"
+ end
+
+ def test_double_quote
+ assert_result [{"hello" => '"'}], "SELECT {var} AS hello", var: '"'
+ end
+
+ def test_backslash
+ assert_result [{"hello" => "\\"}], "SELECT {var} AS hello", var: "\\"
+ end
end
diff --git a/test/adapters/druid_test.rb b/test/adapters/druid_test.rb
index c5471b5f..c549bc9b 100644
--- a/test/adapters/druid_test.rb
+++ b/test/adapters/druid_test.rb
@@ -10,4 +10,45 @@ class DruidTest < ActionDispatch::IntegrationTest
def test_run
assert_result [{"hello" => "world"}], "SELECT 'world' AS hello"
end
+
+ def test_audit
+ assert_audit "SELECT ? AS hello\n\n[\"world\"]", "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_string
+ assert_result [{"hello" => "world"}], "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_integer
+ assert_result [{"hello" => "1"}], "SELECT {var} AS hello", var: "1"
+ end
+
+ def test_float
+ assert_result [{"hello" => "1.5"}], "SELECT {var} AS hello", var: "1.5"
+ end
+
+ def test_time
+ assert_result [{"hello" => "2022-01-01 08:00:00 UTC"}], "SELECT {created_at} AS hello", created_at: "2022-01-01 08:00:00"
+ end
+
+ # TODO fix
+ def test_nil
+ # assert_result [{"hello" => nil}], "SELECT {var} AS hello", var: ""
+ end
+
+ def test_single_quote
+ assert_result [{"hello" => "'"}], "SELECT {var} AS hello", var: "'"
+ end
+
+ def test_double_quote
+ assert_result [{"hello" => '"'}], "SELECT {var} AS hello", var: '"'
+ end
+
+ def test_backslash
+ assert_result [{"hello" => "\\"}], "SELECT {var} AS hello", var: "\\"
+ end
+
+ def test_bad_position
+ assert_bad_position "SELECT hello AS {var}", var: "world"
+ end
end
diff --git a/test/adapters/elasticsearch_test.rb b/test/adapters/elasticsearch_test.rb
index b48fee3b..d30c70ce 100644
--- a/test/adapters/elasticsearch_test.rb
+++ b/test/adapters/elasticsearch_test.rb
@@ -10,4 +10,44 @@ class ElasticsearchTest < ActionDispatch::IntegrationTest
def test_run
assert_result [{"hello" => "world"}], "SELECT 'world' AS hello"
end
+
+ def test_audit
+ assert_audit "SELECT ? AS hello\n\n[\"world\"]", "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_string
+ assert_result [{"hello" => "world"}], "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_integer
+ assert_result [{"hello" => "1"}], "SELECT {var} AS hello", var: "1"
+ end
+
+ def test_float
+ assert_result [{"hello" => "1.5"}], "SELECT {var} AS hello", var: "1.5"
+ end
+
+ def test_time
+ assert_result [{"hello" => "2022-01-01T08:00:00.000Z"}], "SELECT {created_at} AS hello", created_at: "2022-01-01 08:00:00"
+ end
+
+ def test_nil
+ assert_result [{"hello" => nil}], "SELECT {var} AS hello", var: ""
+ end
+
+ def test_single_quote
+ assert_result [{"hello" => "'"}], "SELECT {var} AS hello", var: "'"
+ end
+
+ def test_double_quote
+ assert_result [{"hello" => '"'}], "SELECT {var} AS hello", var: '"'
+ end
+
+ def test_backslash
+ assert_result [{"hello" => "\\"}], "SELECT {var} AS hello", var: "\\"
+ end
+
+ def test_bad_position
+ assert_bad_position "SELECT 'world' AS {var}", var: "hello"
+ end
end
diff --git a/test/adapters/hive_test.rb b/test/adapters/hive_test.rb
index 34b967f6..0e43aa70 100644
--- a/test/adapters/hive_test.rb
+++ b/test/adapters/hive_test.rb
@@ -1,5 +1,7 @@
require_relative "../test_helper"
+# bin/beeline -u jdbc:hive2://localhost:10000 -e 'CREATE DATABASE blazer_test;'
+
class HiveTest < ActionDispatch::IntegrationTest
include AdapterTest
@@ -10,4 +12,40 @@ class HiveTest < ActionDispatch::IntegrationTest
def test_run
assert_result [{"hello" => "world"}], "SELECT 'world' AS hello"
end
+
+ def test_audit
+ assert_audit "SELECT 'world' AS hello", "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_string
+ assert_result [{"hello" => "world"}], "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_integer
+ assert_result [{"hello" => "1"}], "SELECT {var} AS hello", var: "1"
+ end
+
+ def test_float
+ assert_result [{"hello" => "1.5"}], "SELECT {var} AS hello", var: "1.5"
+ end
+
+ def test_time
+ assert_result [{"hello" => "2022-01-01 08:00:00"}], "SELECT {created_at} AS hello", created_at: "2022-01-01 08:00:00"
+ end
+
+ def test_nil
+ assert_result [{"hello" => nil}], "SELECT {var} AS hello", var: ""
+ end
+
+ def test_single_quote
+ assert_result [{"hello" => "'"}], "SELECT {var} AS hello", var: "'"
+ end
+
+ def test_double_quote
+ assert_result [{"hello" => '"'}], "SELECT {var} AS hello", var: '"'
+ end
+
+ def test_backslash
+ assert_result [{"hello" => "\\"}], "SELECT {var} AS hello", var: "\\"
+ end
end
diff --git a/test/adapters/ignite_test.rb b/test/adapters/ignite_test.rb
index 8ecde393..bb1eb59d 100644
--- a/test/adapters/ignite_test.rb
+++ b/test/adapters/ignite_test.rb
@@ -10,4 +10,44 @@ class IgniteTest < ActionDispatch::IntegrationTest
def test_run
assert_result [{"HELLO" => "world"}], "SELECT 'world' AS hello"
end
+
+ def test_audit
+ assert_audit "SELECT ? AS hello\n\n[\"world\"]", "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_string
+ assert_result [{"HELLO" => "world"}], "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_integer
+ assert_result [{"HELLO" => "1"}], "SELECT {var} AS hello", var: "1"
+ end
+
+ def test_float
+ assert_result [{"HELLO" => "1.5"}], "SELECT {var} AS hello", var: "1.5"
+ end
+
+ def test_time
+ assert_result [{"HELLO" => "2022-01-01 08:00:00 UTC"}], "SELECT {created_at} AS hello", created_at: "2022-01-01 08:00:00"
+ end
+
+ def test_nil
+ assert_result [{"HELLO" => nil}], "SELECT {var} AS hello", var: ""
+ end
+
+ def test_single_quote
+ assert_result [{"HELLO" => "'"}], "SELECT {var} AS hello", var: "'"
+ end
+
+ def test_double_quote
+ assert_result [{"HELLO" => '"'}], "SELECT {var} AS hello", var: '"'
+ end
+
+ def test_backslash
+ assert_result [{"HELLO" => "\\"}], "SELECT {var} AS hello", var: "\\"
+ end
+
+ def test_bad_position
+ assert_error "Syntax error in SQL statement", "SELECT 'world' AS {var}", var: "hello"
+ end
end
diff --git a/test/adapters/influxdb_test.rb b/test/adapters/influxdb_test.rb
index d620e0dd..72b1103b 100644
--- a/test/adapters/influxdb_test.rb
+++ b/test/adapters/influxdb_test.rb
@@ -12,12 +12,37 @@ class InfluxdbTest < ActionDispatch::IntegrationTest
client = InfluxDB::Client.new(url: "http://localhost:8086/blazer_test")
client.delete_series("items")
client.write_point("items", {values: {value: 1}, tags: {hello: "world"}, timestamp: 0})
+ client.write_point("items", {values: {value: 1}, tags: {hello: "'"}, timestamp: 0})
+ client.write_point("items", {values: {value: 1}, tags: {hello: '"'}, timestamp: 0})
+ # InfluxDB does not like trailing backslashes
+ # https://github.com/influxdata/influxdb/issues/5231
+ # https://github.com/influxdata/influxdb-ruby/issues/225
+ client.write_point("items", {values: {value: 1}, tags: {hello: "\\a"}, timestamp: 0})
true
end
end
def test_run
- expected = [{"time" => "1970-01-01 00:00:00 UTC", "count_value" => "1"}]
- assert_result expected, "SELECT COUNT(*) FROM items WHERE hello = 'world'"
+ expected = [{"time" => "1970-01-01 00:00:00 UTC", "hello" => "world", "value" => "1"}]
+ assert_result expected, "SELECT * FROM items WHERE hello = 'world'"
+ end
+
+ def test_audit
+ assert_audit "SELECT * FROM items WHERE hello = 'world'", "SELECT * FROM items WHERE hello = {var}", var: "world"
+ end
+
+ def test_single_quote
+ expected = [{"time" => "1970-01-01 00:00:00 UTC", "hello" => "'", "value" => "1"}]
+ assert_result expected, "SELECT * FROM items WHERE hello = {var}", var: "'"
+ end
+
+ def test_double_quote
+ expected = [{"time" => "1970-01-01 00:00:00 UTC", "hello" => '"', "value" => "1"}]
+ assert_result expected, "SELECT * FROM items WHERE hello = {var}", var: '"'
+ end
+
+ def test_backslash
+ expected = [{"time" => "1970-01-01 00:00:00 UTC", "hello" => "\\a", "value" => "1"}]
+ assert_result expected, "SELECT * FROM items WHERE hello = {var}", var: "\\a"
end
end
diff --git a/test/adapters/mongodb_test.rb b/test/adapters/mongodb_test.rb
index 154a1136..f8040b13 100644
--- a/test/adapters/mongodb_test.rb
+++ b/test/adapters/mongodb_test.rb
@@ -12,11 +12,30 @@ class MongodbTest < ActionDispatch::IntegrationTest
client = Mongo::Client.new(["127.0.0.1:27017"], database: "blazer_test")
client[:items].drop
client[:items].insert_one({hello: "world"})
+ client[:items].insert_one({hello: "'"})
+ client[:items].insert_one({hello: '"'})
+ client[:items].insert_one({hello: "\\"})
true
end
end
def test_run
- assert_result [{"hello" => "world"}], "db.items.find({}, {hello: 1, _id: 0})"
+ assert_result [{"hello" => "world"}], "db.items.find({hello: 'world'}, {hello: 1, _id: 0})"
+ end
+
+ def test_audit
+ assert_audit "db.items.find({hello: 'world'}, {hello: 1, _id: 0})", "db.items.find({hello: {var}}, {hello: 1, _id: 0})", var: "world"
+ end
+
+ def test_single_quote
+ assert_result [{"hello" => "'"}], "db.items.find({hello: {var}}, {hello: 1, _id: 0})", var: "'"
+ end
+
+ def test_double_quote
+ assert_result [{"hello" => '"'}], "db.items.find({hello: {var}}, {hello: 1, _id: 0})", var: '"'
+ end
+
+ def test_backslash
+ assert_result [{"hello" => "\\"}], "db.items.find({hello: {var}}, {hello: 1, _id: 0})", var: "\\"
end
end
diff --git a/test/adapters/mysql_test.rb b/test/adapters/mysql_test.rb
index 77526267..99e661c2 100644
--- a/test/adapters/mysql_test.rb
+++ b/test/adapters/mysql_test.rb
@@ -10,4 +10,86 @@ class MysqlTest < ActionDispatch::IntegrationTest
def test_run
assert_result [{"hello" => "world"}], "SELECT 'world' AS hello"
end
+
+ def test_audit
+ if prepared_statements?
+ assert_audit "SELECT ? AS hello\n\n[\"world\"]", "SELECT {var} AS hello", var: "world"
+ else
+ assert_audit "SELECT 'world' AS hello", "SELECT {var} AS hello", var: "world"
+ end
+ end
+
+ def test_string
+ assert_result [{"hello" => "world"}], "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_integer
+ assert_result [{"hello" => "1"}], "SELECT {var} AS hello", var: "1"
+ end
+
+ def test_float
+ assert_result [{"hello" => "1.5"}], "SELECT {var} AS hello", var: "1.5"
+ end
+
+ def test_time
+ assert_result [{"hello" => "2022-01-01 08:00:00"}], "SELECT {created_at} AS hello", created_at: "2022-01-01 08:00:00"
+ end
+
+ def test_nil
+ assert_result [{"hello" => nil}], "SELECT {var} AS hello", var: ""
+ end
+
+ def test_single_quote
+ assert_result [{"hello" => "'"}], "SELECT {var} AS hello", var: "'"
+ end
+
+ def test_double_quote
+ assert_result [{"hello" => '"'}], "SELECT {var} AS hello", var: '"'
+ end
+
+ def test_backslash
+ assert_result [{"hello" => "\\"}], "SELECT {var} AS hello", var: "\\"
+ end
+
+ def test_multiple_variables
+ assert_result [{"c1" => "one", "c2" => "two", "c3" => "one"}], "SELECT {var} AS c1, {var2} AS c2, {var} AS c3", var: "one", var2: "two"
+ end
+
+ def test_bad_position
+ if prepared_statements?
+ assert_bad_position "SELECT 'world' AS {var}", var: "hello"
+ else
+ assert_result [{"hello"=>"world"}], "SELECT 'world' AS {var}", var: "hello"
+ end
+ end
+
+ def test_bad_position_before
+ if prepared_statements?
+ assert_result [{"?" => "world"}], "SELECT{var}", var: "world"
+ else
+ assert_result [{"world" => "world"}], "SELECT{var}", var: "world"
+ end
+ end
+
+ def test_bad_position_after
+ if prepared_statements?
+ assert_bad_position "SELECT {var}456", var: "world"
+ else
+ assert_error "You have an error in your SQL syntax", "SELECT {var}456", var: "world"
+ end
+ end
+
+ def test_quoted
+ if prepared_statements?
+ assert_error "Bind parameter count (0) doesn't match number of arguments (1)", "SELECT '{var}' AS hello", var: "world"
+ else
+ assert_error "You have an error in your SQL syntax", "SELECT '{var}' AS hello", var: "world"
+ end
+ end
+
+ private
+
+ def prepared_statements?
+ Blazer.data_sources[data_source].settings["url"].include?("prepared_statements=true")
+ end
end
diff --git a/test/adapters/neo4j_test.rb b/test/adapters/neo4j_test.rb
index c079a372..0a1e14af 100644
--- a/test/adapters/neo4j_test.rb
+++ b/test/adapters/neo4j_test.rb
@@ -10,4 +10,44 @@ class Neo4jTest < ActionDispatch::IntegrationTest
def test_run
assert_result [{"hello" => "world"}], "OPTIONAL MATCH () RETURN 'world' AS `hello`"
end
+
+ def test_audit
+ assert_audit "OPTIONAL MATCH () RETURN $var AS `hello`\n\n{\"var\":\"world\"}", "OPTIONAL MATCH () RETURN {var} AS `hello`", var: "world"
+ end
+
+ def test_string
+ assert_result [{"hello" => "world"}], "OPTIONAL MATCH () RETURN {var} AS `hello`", var: "world"
+ end
+
+ def test_integer
+ assert_result [{"hello" => "1"}], "OPTIONAL MATCH () RETURN {var} AS `hello`", var: "1"
+ end
+
+ def test_float
+ assert_result [{"hello" => "1.5"}], "OPTIONAL MATCH () RETURN {var} AS `hello`", var: "1.5"
+ end
+
+ def test_time
+ assert_result [{"hello" => "2022-01-01T08:00:00.000Z"}], "OPTIONAL MATCH () RETURN {created_at} AS `hello`", created_at: "2022-01-01 08:00:00"
+ end
+
+ def test_nil
+ assert_result [{"hello" => nil}], "OPTIONAL MATCH () RETURN {var} AS `hello`", var: ""
+ end
+
+ def test_single_quote
+ assert_result [{"hello" => "'"}], "OPTIONAL MATCH () RETURN {var} AS `hello`", var: "'"
+ end
+
+ def test_double_quote
+ assert_result [{"hello" => '"'}], "OPTIONAL MATCH () RETURN {var} AS `hello`", var: '"'
+ end
+
+ def test_backslash
+ assert_result [{"hello" => "\\"}], "OPTIONAL MATCH () RETURN {var} AS `hello`", var: "\\"
+ end
+
+ def test_bad_position
+ assert_bad_position "OPTIONAL MATCH () RETURN 'world' AS {var}", var: "hello"
+ end
end
diff --git a/test/adapters/opensearch_test.rb b/test/adapters/opensearch_test.rb
index 79ee678b..b2d06b13 100644
--- a/test/adapters/opensearch_test.rb
+++ b/test/adapters/opensearch_test.rb
@@ -10,4 +10,16 @@ class OpensearchTest < ActionDispatch::IntegrationTest
def test_run
assert_result [{"'world'" => "world"}], "SELECT 'world' AS hello"
end
+
+ def test_single_quote
+ assert_error "Quoting not specified", "SELECT {var} AS hello", var: "'"
+ end
+
+ def test_double_quote
+ assert_error "Quoting not specified", "SELECT {var} AS hello", var: '"'
+ end
+
+ def test_backslash
+ assert_error "Quoting not specified", "SELECT {var} AS hello", var: "\\"
+ end
end
diff --git a/test/adapters/postgresql_test.rb b/test/adapters/postgresql_test.rb
index 76600f88..6828ce58 100644
--- a/test/adapters/postgresql_test.rb
+++ b/test/adapters/postgresql_test.rb
@@ -10,4 +10,61 @@ class PostgresqlTest < ActionDispatch::IntegrationTest
def test_run
assert_result [{"hello" => "world"}], "SELECT 'world' AS hello"
end
+
+ def test_audit
+ assert_audit "SELECT $1 AS hello\n\n[\"world\"]", "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_string
+ assert_result [{"hello" => "world"}], "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_integer
+ assert_result [{"hello" => "1"}], "SELECT {var} AS hello", var: "1"
+ end
+
+ def test_float
+ assert_result [{"hello" => "1.5"}], "SELECT {var} AS hello", var: "1.5"
+ end
+
+ def test_time
+ assert_result [{"hello" => "2022-01-01 08:00:00"}], "SELECT {created_at} AS hello", created_at: "2022-01-01 08:00:00"
+ end
+
+ def test_nil
+ assert_result [{"hello" => nil}], "SELECT {var} AS hello", var: ""
+ end
+
+ def test_single_quote
+ assert_result [{"hello" => "'"}], "SELECT {var} AS hello", var: "'"
+ end
+
+ def test_double_quote
+ assert_result [{"hello" => '"'}], "SELECT {var} AS hello", var: '"'
+ end
+
+ def test_backslash
+ assert_result [{"hello" => "\\"}], "SELECT {var} AS hello", var: "\\"
+ end
+
+ def test_multiple_variables
+ assert_result [{"c1" => "one", "c2" => "two", "c3" => "one"}], "SELECT {var} AS c1, {var2} AS c2, {var} AS c3", var: "one", var2: "two"
+ end
+
+ def test_bad_position
+ assert_bad_position "SELECT 'world' AS {var}", var: "hello"
+ end
+
+ def test_bad_position_before
+ assert_error "syntax error at or near \"SELECT$1\"", "SELECT{var}", var: "world"
+ end
+
+ def test_bad_position_after
+ assert_error "syntax error at or near \"456\"\nLINE 1: SELECT $1 456", "SELECT {var}456", var: "world"
+ assert_equal "SELECT $1 456\n\n[\"world\"]", Blazer::Audit.last.statement
+ end
+
+ def test_quoted
+ assert_error "could not determine data type of parameter $1", "SELECT '{var}' AS hello", var: "world"
+ end
end
diff --git a/test/adapters/presto_test.rb b/test/adapters/presto_test.rb
index 33e93a35..518d385d 100644
--- a/test/adapters/presto_test.rb
+++ b/test/adapters/presto_test.rb
@@ -14,4 +14,20 @@ class PrestoTest < ActionDispatch::IntegrationTest
def test_run
assert_result [{"hello" => "world"}], "SELECT 'world' AS hello"
end
+
+ def test_audit
+ assert_audit "SELECT 'world' AS hello", "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_single_quote
+ assert_result [{"hello" => "'"}], "SELECT {var} AS hello", var: "'"
+ end
+
+ def test_double_quote
+ assert_result [{"hello" => '"'}], "SELECT {var} AS hello", var: '"'
+ end
+
+ def test_backslash
+ assert_result [{"hello" => "\\"}], "SELECT {var} AS hello", var: "\\"
+ end
end
diff --git a/test/adapters/redshift_test.rb b/test/adapters/redshift_test.rb
index 77dc6512..5e36df48 100644
--- a/test/adapters/redshift_test.rb
+++ b/test/adapters/redshift_test.rb
@@ -6,4 +6,52 @@ class RedshiftTest < ActionDispatch::IntegrationTest
def data_source
"redshift"
end
+
+ def test_run
+ assert_result [{"hello" => "world"}], "SELECT 'world' AS hello"
+ end
+
+ def test_audit
+ assert_audit "SELECT 'world' AS hello", "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_string
+ assert_result [{"hello" => "world"}], "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_integer
+ assert_result [{"hello" => "1"}], "SELECT {var} AS hello", var: "1"
+ end
+
+ def test_float
+ assert_result [{"hello" => "1.5"}], "SELECT {var} AS hello", var: "1.5"
+ end
+
+ def test_time
+ assert_result [{"hello" => "2022-01-01 08:00:00"}], "SELECT {created_at} AS hello", created_at: "2022-01-01 08:00:00"
+ end
+
+ def test_nil
+ assert_result [{"hello" => nil}], "SELECT {var} AS hello", var: ""
+ end
+
+ def test_single_quote
+ assert_result [{"hello" => "'"}], "SELECT {var} AS hello", var: "'"
+ end
+
+ def test_double_quote
+ assert_result [{"hello" => '"'}], "SELECT {var} AS hello", var: '"'
+ end
+
+ def test_backslash
+ assert_result [{"hello" => "\\"}], "SELECT {var} AS hello", var: "\\"
+ end
+
+ def test_multiple_variables
+ assert_result [{"c1" => "one", "c2" => "two", "c3" => "one"}], "SELECT {var} AS c1, {var2} AS c2, {var} AS c3", var: "one", var2: "two"
+ end
+
+ def test_bad_position
+ assert_error "syntax error at or near", "SELECT 'world' AS {var}", var: "hello"
+ end
end
diff --git a/test/adapters/salesforce_test.rb b/test/adapters/salesforce_test.rb
index 8c48d781..52ee193f 100644
--- a/test/adapters/salesforce_test.rb
+++ b/test/adapters/salesforce_test.rb
@@ -1,9 +1,35 @@
require_relative "../test_helper"
+# https://stackoverflow.com/questions/12794302/salesforce-authentication-failing/29112224#29112224
+# create accounts named world, ', ", and \
+
+# ENV["SALESFORCE_USERNAME"] = "username"
+# ENV["SALESFORCE_PASSWORD"] = "password"
+# ENV["SALESFORCE_SECURITY_TOKEN"] = "security token"
+# ENV["SALESFORCE_CLIENT_ID"] = "client id"
+# ENV["SALESFORCE_CLIENT_SECRET"] = "client secret"
+# ENV["SALESFORCE_API_VERSION"] = "41.0"
+
class SalesforceTest < ActionDispatch::IntegrationTest
include AdapterTest
def data_source
"salesforce"
end
+
+ def test_run
+ assert_result [{"Name" => "world"}], "SELECT Name FROM Account WHERE Name = 'world'"
+ end
+
+ def test_single_quote
+ assert_result [{"Name" => "'"}], "SELECT Name FROM Account WHERE Name = {var}", var: "'"
+ end
+
+ def test_double_quote
+ assert_result [{"Name" => '"'}], "SELECT Name FROM Account WHERE Name = {var}", var: '"'
+ end
+
+ def test_backslash
+ assert_result [{"Name" => "\\"}], "SELECT Name FROM Account WHERE Name = {var}", var: "\\"
+ end
end
diff --git a/test/adapters/soda_test.rb b/test/adapters/soda_test.rb
index 483d0097..16bbb358 100644
--- a/test/adapters/soda_test.rb
+++ b/test/adapters/soda_test.rb
@@ -14,4 +14,16 @@ class SodaTest < ActionDispatch::IntegrationTest
def test_run
assert_result [{"hello" => "world"}], "SELECT 'world' AS hello LIMIT 1"
end
+
+ def test_single_quote
+ assert_result [{"hello" => "'"}], "SELECT {var} AS hello LIMIT 1", var: "'"
+ end
+
+ def test_double_quote
+ assert_result [{"hello" => '"'}], "SELECT {var} AS hello LIMIT 1", var: '"'
+ end
+
+ def test_backslash
+ assert_result [{"hello" => "\\"}], "SELECT {var} AS hello LIMIT 1", var: "\\"
+ end
end
diff --git a/test/adapters/spark_test.rb b/test/adapters/spark_test.rb
index 662ae0a0..1ba86c7e 100644
--- a/test/adapters/spark_test.rb
+++ b/test/adapters/spark_test.rb
@@ -12,4 +12,40 @@ class SparkTest < ActionDispatch::IntegrationTest
def test_run
assert_result [{"hello" => "world"}], "SELECT 'world' AS hello"
end
+
+ def test_audit
+ assert_audit "SELECT 'world' AS hello", "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_string
+ assert_result [{"hello" => "world"}], "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_integer
+ assert_result [{"hello" => "1"}], "SELECT {var} AS hello", var: "1"
+ end
+
+ def test_float
+ assert_result [{"hello" => "1.5"}], "SELECT {var} AS hello", var: "1.5"
+ end
+
+ def test_time
+ assert_result [{"hello" => "2022-01-01 08:00:00"}], "SELECT {created_at} AS hello", created_at: "2022-01-01 08:00:00"
+ end
+
+ def test_nil
+ assert_result [{"hello" => nil}], "SELECT {var} AS hello", var: ""
+ end
+
+ def test_single_quote
+ assert_result [{"hello" => "'"}], "SELECT {var} AS hello", var: "'"
+ end
+
+ def test_double_quote
+ assert_result [{"hello" => '"'}], "SELECT {var} AS hello", var: '"'
+ end
+
+ def test_backslash
+ assert_result [{"hello" => "\\"}], "SELECT {var} AS hello", var: "\\"
+ end
end
diff --git a/test/adapters/sqlite_test.rb b/test/adapters/sqlite_test.rb
index 1f71f755..dbb1d331 100644
--- a/test/adapters/sqlite_test.rb
+++ b/test/adapters/sqlite_test.rb
@@ -10,4 +10,44 @@ class SqliteTest < ActionDispatch::IntegrationTest
def test_run
assert_result [{"hello" => "world"}], "SELECT 'world' AS hello"
end
+
+ def test_audit
+ assert_audit "SELECT $1 AS hello\n\n[\"world\"]", "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_string
+ assert_result [{"hello" => "world"}], "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_integer
+ assert_result [{"hello" => "1"}], "SELECT {var} AS hello", var: "1"
+ end
+
+ def test_float
+ assert_result [{"hello" => "1.5"}], "SELECT {var} AS hello", var: "1.5"
+ end
+
+ def test_time
+ assert_result [{"hello" => "2022-01-01 08:00:00"}], "SELECT {created_at} AS hello", created_at: "2022-01-01 08:00:00"
+ end
+
+ def test_nil
+ assert_result [{"hello" => nil}], "SELECT {var} AS hello", var: ""
+ end
+
+ def test_single_quote
+ assert_result [{"hello" => "'"}], "SELECT {var} AS hello", var: "'"
+ end
+
+ def test_double_quote
+ assert_result [{"hello" => '"'}], "SELECT {var} AS hello", var: '"'
+ end
+
+ def test_backslash
+ assert_result [{"hello" => "\\"}], "SELECT {var} AS hello", var: "\\"
+ end
+
+ def test_multiple_variables
+ assert_result [{"c1" => "one", "c2" => "two", "c3" => "one"}], "SELECT {var} AS c1, {var2} AS c2, {var} AS c3", var: "one", var2: "two"
+ end
end
diff --git a/test/adapters/sqlserver_test.rb b/test/adapters/sqlserver_test.rb
index d490a92d..7f1d389b 100644
--- a/test/adapters/sqlserver_test.rb
+++ b/test/adapters/sqlserver_test.rb
@@ -14,4 +14,50 @@ class SqlserverTest < ActionDispatch::IntegrationTest
def test_run
assert_result [{"hello" => "world"}], "SELECT 'world' AS hello"
end
+
+ def test_audit
+ assert_audit "SELECT @0 AS hello\n\n[\"world\"]", "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_string
+ assert_result [{"hello" => "world"}], "SELECT {var} AS hello", var: "world"
+ end
+
+ def test_integer
+ assert_result [{"hello" => "1"}], "SELECT {var} AS hello", var: "1"
+ end
+
+ # https://github.com/rails-sqlserver/activerecord-sqlserver-adapter/issues/643
+ # should be 1.5
+ def test_float
+ assert_result [{"hello" => "1"}], "SELECT {var} AS hello", var: "1.5"
+ end
+
+ def test_time
+ assert_result [{"hello" => "01-01-2022 08:00:00.0"}], "SELECT {created_at} AS hello", created_at: "2022-01-01 08:00:00"
+ end
+
+ def test_nil
+ assert_result [{"hello" => nil}], "SELECT {var} AS hello", var: ""
+ end
+
+ def test_single_quote
+ assert_result [{"hello" => "'"}], "SELECT {var} AS hello", var: "'"
+ end
+
+ def test_double_quote
+ assert_result [{"hello" => '"'}], "SELECT {var} AS hello", var: '"'
+ end
+
+ def test_backslash
+ assert_result [{"hello" => "\\"}], "SELECT {var} AS hello", var: "\\"
+ end
+
+ def test_bad_position
+ assert_bad_position "SELECT 'world' AS {var}", var: "hello"
+ end
+
+ def test_quoted
+ assert_result [{"hello"=>"@0 "}], "SELECT '{var}' AS hello", var: "world"
+ end
end
diff --git a/test/internal/config/blazer.yml b/test/internal/config/blazer.yml
index da4f6203..07c7b09d 100644
--- a/test/internal/config/blazer.yml
+++ b/test/internal/config/blazer.yml
@@ -75,7 +75,7 @@ data_sources:
url: mongodb://localhost:27017/blazer_test
mysql:
- url: mysql2://localhost:3306/blazer_test
+ url: mysql2://localhost:3306/blazer_test # ?prepared_statements=true
neo4j:
adapter: neo4j
diff --git a/test/support/adapter_test.rb b/test/support/adapter_test.rb
index 03667bea..c3251ad1 100644
--- a/test/support/adapter_test.rb
+++ b/test/support/adapter_test.rb
@@ -12,12 +12,28 @@ module AdapterTest
JSON.parse(response.body)
end
- def assert_result(expected, statement)
- assert_equal expected, run_statement(statement)
+ def assert_result(expected, statement, **params)
+ assert_equal expected, run_statement(statement, **params)
end
- def run_statement(statement)
- run_query statement, data_source: data_source, format: "csv"
+ def assert_audit(expected, statement, **params)
+ run_statement(statement, **params)
+ assert_equal expected, Blazer::Audit.last.statement
+ end
+
+ def assert_error(message, statement, **params)
+ error = assert_raises(Blazer::Error) do
+ run_statement(statement, **params)
+ end
+ assert_match message, error.message
+ end
+
+ def assert_bad_position(statement, **params)
+ assert_error "Variable cannot be used in this position", statement, **params
+ end
+
+ def run_statement(statement, **params)
+ run_query statement, **params, data_source: data_source, format: "csv"
CSV.parse(response.body, headers: true).map(&:to_h)
end
end