Merge branch 'dev/0.6.0' into date_spine_docs

tmastny · Sep 22, 2020 · 1cb3b01 · 1cb3b01
2 parents 4a5206a + 48567a6
commit 1cb3b01
Show file tree

Hide file tree

Showing 35 changed files with 180 additions and 122 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,18 +1,47 @@
-# dbt-utils v0.6.0 (unreleased)
+# dbt-utils v0.6.0
 
-## Fixes
+## Breaking changes
+- :rotating_light: dbt v0.18.0 or greater is required for this release. If you are not ready to upgrade, consider using a previous release of this package
+- :rotating_light: The `get_tables_by_prefix`, `union_tables` and `get_tables_by_pattern` macros have been removed
+
+## Migration instructions
+- Upgrade your dbt project to v0.18.0 using [these instructions](https://discourse.getdbt.com/t/prerelease-v0-18-0-marian-anderson/1545).
+- Upgrade your `packages.yml` file to use version `0.6.0` of this package. Run `dbt clean` and `dbt deps`.
+- If your project uses the `get_tables_by_prefix` macro, replace it with `get_relations_by_prefix`. All arguments have retained the same name.
+- If your project uses the `union_tables` macro, replace it with `union_relations`. While the order of arguments has stayed consistent, the `tables` argument has been renamed to `relations`. Further, the default value for the `source_column_name` argument has changed from `'_dbt_source_table'` to `'_dbt_source_relation'` — you may want to explicitly define this argument to avoid breaking changes.
+
+```
+-- before:
+{{ dbt_utils.union_tables(
+    tables=[ref('my_model'), source('my_source', 'my_table')],
+    exclude=["_loaded_at"]
+) }}
+
+-- after:
+{{ dbt_utils.union_relations(
+    relations=[ref('my_model'), source('my_source', 'my_table')],
+    exclude=["_loaded_at"],
+    source_column_name='_dbt_source_table'
+) }}
+```
+- If your project uses the `get_tables_by_pattern` macro, replace it with `get_tables_by_pattern_sql` — all arguments are consistent.
 
 ## Features
 
+* Switch usage of `adapter_macro` to `adapter.dispatch`, and define `dbt_utils_dispatch_list`,
+enabling users of community-supported database plugins to add or override macro implementations
+specific to their database ([#267](https://github.com/fishtown-analytics/dbt-utils/pull/267))
+* Use `add_ephemeral_prefix` instead of hard-coding a string literal, to support
+database adapters that use different prefixes ([#267](https://github.com/fishtown-analytics/dbt-utils/pull/267))
+* Implement a quote_columns argument in the unique_combination_of_columns schema test ([#270](https://github.com/fishtown-analytics/dbt-utils/pull/270) [@JoshuaHuntley](https://github.com/JoshuaHuntley))
+
 ## Quality of life
+* Remove deprecated macros `get_tables_by_prefix` and `union_tables` ([#268](https://github.com/fishtown-analytics/dbt-utils/pull/268))
+* Remove `get_tables_by_pattern` macro, which is equivalent to the `get_tables_by_pattern_sql` macro (the latter has a more logical name) ([#268](https://github.com/fishtown-analytics/dbt-utils/pull/268))
 
 # dbt-utils v0.5.1
 
-## Fixes
-
-## Features
-
 ## Quality of life
-* Improve release process, and fix tests (#251)
-* Make deprecation warnings more useful (#258 @tayloramurphy)
-* Add more docs for `date_spine` (#265 @calvingiles)
+* Improve release process, and fix tests ([#251](https://github.com/fishtown-analytics/dbt-utils/pull/251))
+* Make deprecation warnings more useful ([#258](https://github.com/fishtown-analytics/dbt-utils/pull/258) [@tayloramurphy](https://github.com/tayloramurphy))
+* Add more docs for `date_spine` ([#265](https://github.com/fishtown-analytics/dbt-utils/pull/265) [@calvingiles](https://github.com/calvingiles))
diff --git a/README.md b/README.md
@@ -406,6 +406,18 @@ case we recommend using this test instead.
           - product
 ```
 
+An optional `quote_columns` parameter (`default=false`) can also be used if a column name needs to be quoted.
+
+```yaml
+- name: revenue_by_product_by_month
+  tests:
+    - dbt_utils.unique_combination_of_columns:
+        combination_of_columns:
+          - month
+          - group
+        quote_columns: true
+```
+
 ---
 ### SQL helpers
 #### get_query_results_as_dict ([source](macros/sql/get_query_results_as_dict.sql))
@@ -442,9 +454,6 @@ Usage:
 ...
 ```
 #### get_relations_by_prefix
-> This replaces the `get_tables_by_prefix` macro. Note that the `get_tables_by_prefix` macro will
-be deprecated in a future release of this package.
-
 Returns a list of [Relations](https://docs.getdbt.com/docs/writing-code-in-dbt/class-reference/#relation)
 that match a given prefix, with an optional exclusion pattern. It's particularly
 handy paired with `union_relations`.
@@ -516,8 +525,6 @@ from {{ref('my_model')}}
 ```
 
 #### union_relations ([source](macros/sql/union.sql))
-> This replaces the `union_tables` macro. Note that the `union_tables` macro will
-be deprecated in a future release of this package.
 
 This macro unions together an array of [Relations](https://docs.getdbt.com/docs/writing-code-in-dbt/class-reference/#relation),
 even when columns have differing orders in each Relation, and/or some columns are
@@ -777,6 +784,42 @@ We welcome contributions to this repo! To contribute a new feature or a fix, ple
 
 ----
 
+### Dispatch macros
+
+**Note:** This is primarily relevant to users and maintainers of community-supported
+database plugins. If you use Postgres, Redshift, Snowflake, or Bigquery, this likely
+does not apply to you.
+
+dbt v0.18.0 introduces `adapter.dispatch()`, a reliable way to define different implementations of the same macro
+across different databases.
+
+All dispatched macros in `dbt_utils` have an override setting: a `var` named
+`dbt_utils_dispatch_list` that accepts a list of package names. If you set this
+variable in your project, when dbt searches for implementations of a dispatched
+`dbt_utils` macro, it will search through your listed packages _before_ using
+the implementations defined in `dbt_utils`.
+
+Set the variable:
+```yml
+vars:
+  dbt_utils_dispatch_list:
+    - first_package_to_search    # likely the name of your root project
+    - second_package_to_search   # likely an "add-on" package, such as spark_utils
+    # dbt_utils is always the last place searched
+```
+
+When running on Spark, if dbt needs to dispatch `dbt_utils.datediff`, it will search for the following in order:
+```
+first_package_to_search.spark__datediff
+first_package_to_search.default__datediff
+second_package_to_search.spark__datediff
+second_package_to_search.default__datediff
+dbt_utils.spark__datediff
+dbt_utils.default__datediff
+```
+
+----
+
 ### Getting started with dbt
 
 - [What is dbt]?

diff --git a/dbt_project.yml b/dbt_project.yml
@@ -1,7 +1,7 @@
 name: 'dbt_utils'
 version: '0.1.0'
 
-require-dbt-version: [">=0.17.0", "<0.18.0"]
+require-dbt-version: [">=0.18.0", "<0.19.0"]
 config-version: 2
 
 target-path: "target"

diff --git a/integration_tests/models/datetime/test_date_spine.sql b/integration_tests/models/datetime/test_date_spine.sql
@@ -9,8 +9,7 @@
 with date_spine as (
 
     {% if target.type == 'postgres' %}
-        {{ log("WARNING: Not testing - datediff macro is unsupported on Postgres", info=True) }}
-        select * from {{ ref('data_date_spine') }}
+        {{ dbt_utils.date_spine("day", "'2018-01-01'::date", "'2018-01-10'::date") }}
 
     {% elif target.type == 'bigquery' %}
         select cast(date_day as date) as date_day

diff --git a/integration_tests/models/sql/test_get_relations_by_pattern.sql b/integration_tests/models/sql/test_get_relations_by_pattern.sql
@@ -1,4 +1,4 @@
 {{ config(materialized = 'table') }}
 
-{% set relations = dbt_utils.get_relations_by_pattern(target.schema, 'data_events_') %}
+{% set relations = dbt_utils.get_relations_by_pattern(target.schema, 'data_events_%') %}
 {{ dbt_utils.union_relations(relations) }}
diff --git a/integration_tests/models/sql/test_get_tables_by_prefix_and_union.sql b/integration_tests/models/sql/test_get_tables_by_prefix_and_union.sql
diff --git a/macros/cross_db_utils/_get_utils_namespaces.sql b/macros/cross_db_utils/_get_utils_namespaces.sql
@@ -0,0 +1,4 @@
+{% macro _get_utils_namespaces() %}
+  {% set override_namespaces = var('dbt_utils_dispatch_list', []) %}
+  {% do return(override_namespaces + ['dbt_utils']) %}
+{% endmacro %}
diff --git a/macros/cross_db_utils/_is_ephemeral.sql b/macros/cross_db_utils/_is_ephemeral.sql
@@ -1,7 +1,8 @@
 {% macro _is_ephemeral(obj, macro) %}
     {%- if obj.is_cte -%}
-        {% if obj.name.startswith('__dbt__CTE__') %}
-            {% set model_name = obj.name[12:] %}
+        {% set ephemeral_prefix = api.Relation.add_ephemeral_prefix('') %}
+        {% if obj.name.startswith(ephemeral_prefix) %}
+            {% set model_name = obj.name[(ephemeral_prefix|length):] %}
         {% else %}
             {% set model_name = obj.name %}
         {%- endif -%}

diff --git a/macros/cross_db_utils/concat.sql b/macros/cross_db_utils/concat.sql
@@ -1,23 +1,23 @@
+
+
 {% macro concat(fields) -%}
-  {{ adapter_macro('dbt_utils.concat', fields) }}
+  {{ adapter.dispatch('concat', packages = dbt_utils._get_utils_namespaces())(fields) }}
 {%- endmacro %}
 
-
 {% macro default__concat(fields) -%}
     concat({{ fields|join(', ') }})
 {%- endmacro %}
 
-
 {% macro alternative_concat(fields) %}
     {{ fields|join(' || ') }}
 {% endmacro %}
 
 
 {% macro redshift__concat(fields) %}
-    {{dbt_utils.alternative_concat(fields)}}
+    {{ dbt_utils.alternative_concat(fields) }}
 {% endmacro %}
 
 
 {% macro snowflake__concat(fields) %}
-    {{dbt_utils.alternative_concat(fields)}}
+    {{ dbt_utils.alternative_concat(fields) }}
 {% endmacro %}
diff --git a/macros/cross_db_utils/current_timestamp.sql b/macros/cross_db_utils/current_timestamp.sql
@@ -1,5 +1,5 @@
 {% macro current_timestamp() -%}
-  {{ adapter_macro('dbt_utils.current_timestamp') }}
+  {{ adapter.dispatch('current_timestamp', packages = dbt_utils._get_utils_namespaces())() }}
 {%- endmacro %}
 
 {% macro default__current_timestamp() %}
@@ -17,7 +17,7 @@
 
 
 {% macro current_timestamp_in_utc() -%}
-  {{ adapter_macro('dbt_utils.current_timestamp_in_utc') }}
+  {{ adapter.dispatch('current_timestamp_in_utc', packages = dbt_utils._get_utils_namespaces())() }}
 {%- endmacro %}
 
 {% macro default__current_timestamp_in_utc() %}

diff --git a/macros/cross_db_utils/datatypes.sql b/macros/cross_db_utils/datatypes.sql
@@ -1,7 +1,7 @@
 {# string  -------------------------------------------------     #}
 
 {%- macro type_string() -%}
-  {{ adapter_macro('dbt_utils.type_string') }}
+  {{ adapter.dispatch('type_string', packages = dbt_utils._get_utils_namespaces())() }}
 {%- endmacro -%}
 
 {% macro default__type_string() %}
@@ -25,7 +25,7 @@
 {# timestamp  -------------------------------------------------     #}
 
 {%- macro type_timestamp() -%}
-  {{ adapter_macro('dbt_utils.type_timestamp') }}
+  {{ adapter.dispatch('type_timestamp', packages = dbt_utils._get_utils_namespaces())() }}
 {%- endmacro -%}
 
 {% macro default__type_timestamp() %}
@@ -40,7 +40,7 @@
 {# float  -------------------------------------------------     #}
 
 {%- macro type_float() -%}
-  {{ adapter_macro('dbt_utils.type_float') }}
+  {{ adapter.dispatch('type_float', packages = dbt_utils._get_utils_namespaces())() }}
 {%- endmacro -%}
 
 {% macro default__type_float() %}
@@ -54,7 +54,7 @@
 {# numeric  ------------------------------------------------     #}
 
 {%- macro type_numeric() -%}
-  {{ adapter_macro('dbt_utils.type_numeric') }}
+  {{ adapter.dispatch('type_numeric', packages = dbt_utils._get_utils_namespaces())() }}
 {%- endmacro -%}
 
 {% macro default__type_numeric() %}
@@ -69,7 +69,7 @@
 {# bigint  -------------------------------------------------     #}
 
 {%- macro type_bigint() -%}
-  {{ adapter_macro('dbt_utils.type_bigint') }}
+  {{ adapter.dispatch('type_bigint', packages = dbt_utils._get_utils_namespaces())() }}
 {%- endmacro -%}
 
 {% macro default__type_bigint() %}
@@ -83,7 +83,7 @@
 {# int  -------------------------------------------------     #}
 
 {%- macro type_int() -%}
-  {{ adapter_macro('dbt_utils.type_int') }}
+  {{ adapter.dispatch('type_int', packages = dbt_utils._get_utils_namespaces())() }}
 {%- endmacro -%}
 
 {% macro default__type_int() %}

diff --git a/macros/cross_db_utils/date_trunc.sql b/macros/cross_db_utils/date_trunc.sql
@@ -1,5 +1,5 @@
 {% macro date_trunc(datepart, date) -%}
-  {{ adapter_macro('dbt_utils.date_trunc', datepart, date) }}
+  {{ adapter.dispatch('date_trunc', packages = dbt_utils._get_utils_namespaces()) (datepart, date) }}
 {%- endmacro %}
 
 {% macro default__date_trunc(datepart, date) %}

diff --git a/macros/cross_db_utils/dateadd.sql b/macros/cross_db_utils/dateadd.sql
@@ -1,5 +1,5 @@
 {% macro dateadd(datepart, interval, from_date_or_timestamp) %}
-  {{ adapter_macro('dbt_utils.dateadd', datepart, interval, from_date_or_timestamp) }}
+  {{ adapter.dispatch('dateadd', packages = dbt_utils._get_utils_namespaces())(datepart, interval, from_date_or_timestamp) }}
 {% endmacro %}
 
 
@@ -23,7 +23,6 @@
 
 {% endmacro %}
 
-
 {% macro postgres__dateadd(datepart, interval, from_date_or_timestamp) %}
 
     {{ from_date_or_timestamp }} + ((interval '1 {{ datepart }}') * ({{ interval }}))

diff --git a/macros/cross_db_utils/datediff.sql b/macros/cross_db_utils/datediff.sql
@@ -1,5 +1,5 @@
 {% macro datediff(first_date, second_date, datepart) %}
-  {{ adapter_macro('dbt_utils.datediff', first_date, second_date, datepart) }}
+  {{ adapter.dispatch('datediff', packages = dbt_utils._get_utils_namespaces())(first_date, second_date, datepart) }}
 {% endmacro %}
 
 
@@ -24,7 +24,6 @@
 
 {% endmacro %}
 
-
 {% macro postgres__datediff(first_date, second_date, datepart) %}
 
     {% if datepart == 'year' %}

diff --git a/macros/cross_db_utils/except.sql b/macros/cross_db_utils/except.sql
@@ -1,5 +1,5 @@
 {% macro except() %}
-  {{ adapter_macro('dbt_utils.except') }}
+  {{ adapter.dispatch('except', packages = dbt_utils._get_utils_namespaces())() }}
 {% endmacro %}
 
 

diff --git a/macros/cross_db_utils/hash.sql b/macros/cross_db_utils/hash.sql
@@ -1,5 +1,5 @@
 {% macro hash(field) -%}
-  {{ adapter_macro('dbt_utils.hash', field) }}
+  {{ adapter.dispatch('hash', packages = dbt_utils._get_utils_namespaces()) (field) }}
 {%- endmacro %}
 
 

diff --git a/macros/cross_db_utils/identifier.sql b/macros/cross_db_utils/identifier.sql
@@ -4,7 +4,7 @@
     Use `adapter.quote` instead. The {}.{} model triggered this warning. \
     '.format(model.package_name, model.name) -%}
   {%- do exceptions.warn(error_message) -%}
-  {{ adapter_macro('dbt_utils.identifier', value) }}	
+  {{ adapter.dispatch('identifier', packages = dbt_utils._get_utils_namespaces()) (value) }}
 {% endmacro %}	
 
 {% macro default__identifier(value) -%}	

diff --git a/macros/cross_db_utils/intersect.sql b/macros/cross_db_utils/intersect.sql
@@ -1,5 +1,5 @@
 {% macro intersect() %}
-  {{ adapter_macro('dbt_utils.intersect') }}
+  {{ adapter.dispatch('intersect', packages = dbt_utils._get_utils_namespaces())() }}
 {% endmacro %}
 
 

diff --git a/macros/cross_db_utils/last_day.sql b/macros/cross_db_utils/last_day.sql
@@ -4,7 +4,7 @@ testing is required to validate that it will work on other dateparts.
 */
 
 {% macro last_day(date, datepart) %}
-  {{ adapter_macro('dbt_utils.last_day', date, datepart) }}
+  {{ adapter.dispatch('last_day', packages = dbt_utils._get_utils_namespaces()) (date, datepart) }}
 {% endmacro %}
 
 

diff --git a/macros/cross_db_utils/length.sql b/macros/cross_db_utils/length.sql
@@ -1,5 +1,5 @@
 {% macro length(expression) -%}
-    {{ adapter_macro('dbt_utils.length', expression) }}
+    {{ adapter.dispatch('length', packages = dbt_utils._get_utils_namespaces()) (expression) }}
 {% endmacro %}
 
 

diff --git a/macros/cross_db_utils/literal.sql b/macros/cross_db_utils/literal.sql
@@ -1,6 +1,6 @@
 
 {%- macro string_literal(value) -%}
-  {{ adapter_macro('dbt_utils.string_literal', value) }}
+  {{ adapter.dispatch('string_literal', packages = dbt_utils._get_utils_namespaces()) (value) }}
 {%- endmacro -%}
 
 {% macro default__string_literal(value) -%}

diff --git a/macros/cross_db_utils/position.sql b/macros/cross_db_utils/position.sql
@@ -1,5 +1,5 @@
 {% macro position(substring_text, string_text) -%}
-    {{ adapter_macro('dbt_utils.position', substring_text, string_text) }}
+    {{ adapter.dispatch('position', packages = dbt_utils._get_utils_namespaces()) (substring_text, string_text) }}
 {% endmacro %}
 
 

diff --git a/macros/cross_db_utils/replace.sql b/macros/cross_db_utils/replace.sql
@@ -1,5 +1,5 @@
 {% macro replace(field, old_chars, new_chars) -%}
-    {{ adapter_macro('dbt_utils.replace', field, old_chars, new_chars) }}
+    {{ adapter.dispatch('replace', packages = dbt_utils._get_utils_namespaces()) (field, old_chars, new_chars) }}
 {% endmacro %}
 
 

diff --git a/macros/cross_db_utils/right.sql b/macros/cross_db_utils/right.sql
@@ -1,5 +1,5 @@
 {% macro right(string_text, length_expression) -%}
-    {{ adapter_macro('dbt_utils.right', string_text, length_expression) }}
+    {{ adapter.dispatch('right', packages = dbt_utils._get_utils_namespaces()) (string_text, length_expression) }}
 {% endmacro %}
 
 {% macro default__right(string_text, length_expression) %}