Merge branch 'utils-v1' into main

danhphan · Oct 4, 2022 · 7eab492 · 7eab492
2 parents 0d1b202 + 2a1e586
commit 7eab492
Show file tree

Hide file tree

Showing 138 changed files with 580 additions and 2,004 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -19,9 +19,6 @@ jobs:
     steps:
       - checkout
       - run: pip install --pre dbt-postgres -r dev-requirements.txt
-      - run:
-          name: "Run Functional Tests - Postgres"
-          command: ./run_functional_test.sh postgres
       - run:
           name: "Run OG Tests - Postgres"
           command: ./run_test.sh postgres
@@ -34,9 +31,6 @@ jobs:
     steps:
       - checkout
       - run: pip install --pre dbt-redshift -r dev-requirements.txt
-      - run:
-          name: "Run Functional Tests - Redshift"
-          command: ./run_functional_test.sh redshift
       - run:
           name: "Run OG Tests - Redshift"
           command: ./run_test.sh redshift
@@ -49,9 +43,6 @@ jobs:
     steps:
       - checkout
       - run: pip install --pre dbt-snowflake -r dev-requirements.txt
-      - run:
-          name: "Run Functional Tests - Snowflake"
-          command: ./run_functional_test.sh snowflake
       - run:
           name: "Run OG Tests - Snowflake"
           command: ./run_test.sh snowflake
@@ -69,9 +60,6 @@ jobs:
       - run:
           name: "Set up credentials"
           command: echo $BIGQUERY_SERVICE_ACCOUNT_JSON > ${HOME}/bigquery-service-key.json
-      - run:
-          name: "Run Functional Tests - BigQuery"
-          command: ./run_functional_test.sh bigquery
       - run:
           name: "Run OG Tests - BigQuery"
           command: ./run_test.sh bigquery

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -11,7 +11,16 @@
 # Unreleased
 
 ## New features
+- Implemented an optional `group_by_columns` argument across many of the generic testing macros to test for properties that only pertain to group-level or are can be more rigorously conducted at the group level. Property available in `recency`, `at_least_one`, `equal_row_count`, `fewer_rows_than`, `not_constant`, `not_null_proportion`, and `sequential` tests [#633](https://github.com/dbt-labs/dbt-utils/pull/633)
 - New feature to omit the `source_column_name` column on the `union_relations` macro ([#331](https://github.com/dbt-labs/dbt-utils/issues/331), [#624](https://github.com/dbt-labs/dbt-utils/pull/624))
+- New feature to select fewer columns in `expression_is_true` ([#683](https://github.com/dbt-labs/dbt-utils/issues/683), [#686](https://github.com/dbt-labs/dbt-utils/pull/686))
+- Add `not_empty_string` generic test that asserts column values are not an empty string. ([#632](https://github.com/dbt-labs/dbt-utils/issues/632), [#634](https://github.com/dbt-labs/dbt-utils/pull/634))
+
+## Under the hood
+- Remove deprecated table argument from unpivot ([#671](https://github.com/dbt-labs/dbt-utils/pull/671))
+- Delete the deprecated identifier macro ([#672](https://github.com/dbt-labs/dbt-utils/pull/672))
+- Handle deprecations in deduplicate macro ([#673](https://github.com/dbt-labs/dbt-utils/pull/673))
+- Fully remove varargs usage in surrogate_key and safe_add ([#674](https://github.com/dbt-labs/dbt-utils/pull/674))
 
 ## Fixes
 - Better handling of whitespaces in the star macro ([#651](https://github.com/dbt-labs/dbt-utils/pull/651))
@@ -20,9 +29,12 @@
 
 ## Contributors:
 - [@christineberger](https://github.com/christineberger) (#624)
+- [@epapineau](https://github.com/epapineau) (#634)
 - [@courentin](https://github.com/courentin) (#651)
 - [@sfc-gh-ancoleman](https://github.com/sfc-gh-ancoleman) (#660)
 - [@zachoj10](https://github.com/zachoj10) (#692)
+- [@miles170](https://github.com/miles170)
+- [@emilyriederer](https://github.com/emilyriederer) 
 
 # dbt-utils v0.8.6
 
@@ -78,6 +90,7 @@ After:
 ## Fixes
 - Fix `union_relations` `source_column_name` none option.
 - Enable a negative part_number for `split_part()` ([#557](https://github.com/dbt-labs/dbt-utils/issues/557), [#559](https://github.com/dbt-labs/dbt-utils/pull/559))
+- Make `exclude` case insensitive for `union_relations()` ([#578](https://github.com/dbt-labs/dbt-utils/issues/557), [#587](https://github.com/dbt-labs/dbt-utils/issues/587))
 
 ## Quality of life
 - Documentation about listagg macro ([#544](https://github.com/dbt-labs/dbt-utils/issues/544), [#560](https://github.com/dbt-labs/dbt-utils/pull/560))
@@ -100,6 +113,7 @@ After:
 - [@LewisDavies](https://github.com/LewisDavies) (#554)
 - [@epapineau](https://github.com/epapineau) (#583)
 - [@b-per](https://github.com/b-per) (#559)
+- [@dbeatty10](https://github.com/dbeatty10), [@jeremyyeo](https://github.com/jeremyyeo) (#587)
 
 # dbt-utils v0.8.4
 ## Fixes

diff --git a/README.md b/README.md
@@ -10,22 +10,21 @@ Check [dbt Hub](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) for the lates
 
 **[Generic tests](#generic-tests)**
 
-- [equal_rowcount](#equal_rowcount-source)
-- [fewer_rows_than](#fewer_rows_than-source)
-- [equality](#equality-source)
-- [expression_is_true](#expression_is_true-source)
-- [recency](#recency-source)
-- [at_least_one](#at_least_one-source)
-- [not_constant](#not_constant-source)
-- [cardinality_equality](#cardinality_equality-source)
-- [unique_where](#unique_where-source)
-- [not_null_where](#not_null_where-source)
-- [not_null_proportion](#not_null_proportion-source)
-- [not_accepted_values](#not_accepted_values-source)
-- [relationships_where](#relationships_where-source)
-- [mutually_exclusive_ranges](#mutually_exclusive_ranges-source)
-- [unique_combination_of_columns](#unique_combination_of_columns-source)
-- [accepted_range](#accepted_range-source)
+  - [equal_rowcount](#equal_rowcount-source)
+  - [fewer_rows_than](#fewer_rows_than-source)
+  - [equality](#equality-source)
+  - [expression_is_true](#expression_is_true-source)
+  - [recency](#recency-source)
+  - [at_least_one](#at_least_one-source)
+  - [not_constant](#not_constant-source)
+  - [not_empty_string](#not_empty_string-source)
+  - [cardinality_equality](#cardinality_equality-source)
+  - [not_null_proportion](#not_null_proportion-source)
+  - [not_accepted_values](#not_accepted_values-source)
+  - [relationships_where](#relationships_where-source)
+  - [mutually_exclusive_ranges](#mutually_exclusive_ranges-source)
+  - [unique_combination_of_columns](#unique_combination_of_columns-source)
+  - [accepted_range](#accepted_range-source)
 
 **[Macros](#macros)**
 
@@ -56,7 +55,6 @@ Check [dbt Hub](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) for the lates
   - [get_url_path](#get_url_path-source)
 
 - [Cross-database macros](#cross-database-macros):
-  - [current_timestamp](#current_timestamp-source)
   - [dateadd](#dateadd-source)
   - [datediff](#datediff-source)
   - [split_part](#split_part-source)
@@ -98,6 +96,8 @@ models:
 
 ```
 
+This test supports the `group_by_columns` parameter; see [Grouping in tests](#grouping-in-tests) for details.
+
 #### fewer_rows_than ([source](macros/generic_tests/fewer_rows_than.sql))
 
 Asserts that the respective model has fewer rows than the model being compared.
@@ -114,6 +114,8 @@ models:
           compare_model: ref('other_table_name')
 ```
 
+This test supports the `group_by_columns` parameter; see [Grouping in tests](#grouping-in-tests) for details.
+
 #### equality ([source](macros/generic_tests/equality.sql))
 
 Asserts the equality of two relations. Optionally specify a subset of columns to compare.
@@ -205,6 +207,7 @@ models:
           field: created_at
           interval: 1
 ```
+This test supports the `group_by_columns` parameter; see [Grouping in tests](#grouping-in-tests) for details.
 
 #### at_least_one ([source](macros/generic_tests/at_least_one.sql))
 
@@ -223,6 +226,8 @@ models:
           - dbt_utils.at_least_one
 ```
 
+This test supports the `group_by_columns` parameter; see [Grouping in tests](#grouping-in-tests) for details.
+
 #### not_constant ([source](macros/generic_tests/not_constant.sql))
 
 Asserts that a column does not have the same value in all rows.
@@ -240,65 +245,59 @@ models:
           - dbt_utils.not_constant
 ```
 
-#### cardinality_equality ([source](macros/generic_tests/cardinality_equality.sql))
+This test supports the `group_by_columns` parameter; see [Grouping in tests](#grouping-in-tests) for details.
 
-Asserts that values in a given column have exactly the same cardinality as values from a different column in a different model.
+#### not_empty_string ([source](macros/generic_tests/not_empty_string.sql))
+Asserts that a column does not have any values equal to `''`. 
 
 **Usage:**
-
 ```yaml
 version: 2
 
 models:
   - name: model_name
     columns:
-      - name: from_column
+      - name: column_name
         tests:
-          - dbt_utils.cardinality_equality:
-              field: other_column_name
-              to: ref('other_model_name')
+          - dbt_utils.not_empty_string
 ```
 
-#### unique_where ([source](macros/generic_tests/test_unique_where.sql))
-
-Asserts that there are no duplicate values present in a field for a subset of rows by specifying a `where` clause.
-
-*Warning*: This test is no longer supported. Starting in dbt v0.20.0, the built-in `unique` test supports a `where` config. [See the dbt docs for more details](https://docs.getdbt.com/reference/resource-configs/where).
+The macro accepts an optional argument `trim_whitespace` that controls whether whitespace should be trimmed from the column when evaluating. The default is `true`. 
 
 **Usage:**
-
 ```yaml
 version: 2
 
 models:
-  - name: my_model
+  - name: model_name
     columns:
-      - name: id
+      - name: column_name
         tests:
-          - dbt_utils.unique_where:
-              where: "_deleted = false"
+          - dbt_utils.not_empty_string:
+              trim_whitespace: false
+              
 ```
 
-#### not_null_where ([source](macros/generic_tests/test_not_null_where.sql))
-
-Asserts that there are no null values present in a column for a subset of rows by specifying a `where` clause.
+#### cardinality_equality ([source](macros/generic_tests/cardinality_equality.sql))
 
-*Warning*: This test is no longer supported. Starting in dbt v0.20.0, the built-in `not_null` test supports a `where` config. [See the dbt docs for more details](https://docs.getdbt.com/reference/resource-configs/where).
+Asserts that values in a given column have exactly the same cardinality as values from a different column in a different model.
 
 **Usage:**
 
 ```yaml
 version: 2
 
 models:
-  - name: my_model
+  - name: model_name
     columns:
-      - name: id
+      - name: from_column
         tests:
-          - dbt_utils.not_null_where:
-              where: "_deleted = false"
+          - dbt_utils.cardinality_equality:
+              field: other_column_name
+              to: ref('other_model_name')
 ```
 
+
 #### not_null_proportion ([source](macros/generic_tests/not_null_proportion.sql))
 
 Asserts that the proportion of non-null values present in a column is between a specified range [`at_least`, `at_most`] where `at_most` is an optional argument (default: `1.0`).
@@ -317,6 +316,8 @@ models:
               at_least: 0.95
 ```
 
+This test supports the `group_by_columns` parameter; see [Grouping in tests](#grouping-in-tests) for details.
+
 #### not_accepted_values ([source](macros/generic_tests/not_accepted_values.sql))
 
 Asserts that there are no rows that match the given values.
@@ -511,6 +512,8 @@ seeds:
 - `interval` (default=1): The gap between two sequential values
 - `datepart` (default=None): Used when the gaps are a unit of time. If omitted, the test will check for a numeric gap.
 
+This test supports the `group_by_columns` parameter; see [Grouping in tests](#grouping-in-tests) for details.
+
 #### unique_combination_of_columns ([source](macros/generic_tests/unique_combination_of_columns.sql))
 
 Asserts that the combination of columns is unique. For example, the
@@ -592,6 +595,34 @@ models:
 
 ----
 
+#### Grouping in tests
+
+Certain tests support the optional `group_by_columns` argument to provide more granularity in performing tests. This can be useful when:
+
+- Some data checks can only be expressed within a group (e.g. ID values should be unique within a group but can be repeated between groups)
+- Some data checks are more precise when done by group (e.g. not only should table rowcounts be equal but the counts within each group should be equal)
+
+This feature is currently available for the following tests:
+
+- equal_rowcount()
+- fewer_rows_than()
+- recency()
+- at_least_one()
+- not_constant()
+- sequential_values()
+- non_null_proportion()
+
+To use this feature, the names of grouping variables can be passed as a list. For example, to test for at least one valid value by group, the `group_by_columns` argument could be used as follows:
+
+```
+  - name: data_test_at_least_one
+    columns:
+      - name: field
+        tests:
+          - dbt_utils.at_least_one:
+              group_by_columns: ['customer_segment']
+```
+
 ## Macros
 
 ### Introspective macros
@@ -1162,18 +1193,6 @@ Note that most of these macros moved to dbt Core as of dbt_utils v0.9.0 and dbt
 To access the version defined in dbt Core, remove the `dbt_utils.` prefix (see [https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros](https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros) for examples).
 As highlighted below, some of the cross-database macros are still in the process of being deprecated.
 
-#### current_timestamp ([source](macros/cross_db_utils/current_timestamp.sql))
-
-*DEPRECATED: This macro is deprecated and will be removed in a future version of the package, once equivalent functionality is implemented in dbt Core.*
-
-This macro returns the current timestamp.
-
-**Usage:**
-
-```
-{{ dbt_utils.current_timestamp() }}
-```
-
 #### dateadd ([source](macros/cross_db_utils/dateadd.sql))
 
 *DEPRECATED: This macro is now provided in dbt Core. It is no longer available in dbt_utils and backwards compatibility will be removed in a future version of the package.*

diff --git a/dbt_project.yml b/dbt_project.yml
@@ -1,7 +1,7 @@
 name: 'dbt_utils'
 version: '0.1.0'
 
-require-dbt-version: [">=1.2.0", "<2.0.0"]
+require-dbt-version: [">=1.3.0-rc1", "<2.0.0"]
 
 config-version: 2
 

diff --git a/integration_tests/data/cross_db/data_any_value_expected.csv b/integration_tests/data/cross_db/data_any_value_expected.csv
diff --git a/integration_tests/data/cross_db/data_array_append.csv b/integration_tests/data/cross_db/data_array_append.csv
diff --git a/integration_tests/data/cross_db/data_array_concat.csv b/integration_tests/data/cross_db/data_array_concat.csv
diff --git a/integration_tests/data/cross_db/data_array_construct.csv b/integration_tests/data/cross_db/data_array_construct.csv
diff --git a/integration_tests/data/cross_db/data_bool_or.csv b/integration_tests/data/cross_db/data_bool_or.csv
diff --git a/integration_tests/data/cross_db/data_bool_or_expected.csv b/integration_tests/data/cross_db/data_bool_or_expected.csv
diff --git a/integration_tests/data/cross_db/data_cast_bool_to_text b/integration_tests/data/cross_db/data_cast_bool_to_text
diff --git a/integration_tests/data/cross_db/data_concat.csv b/integration_tests/data/cross_db/data_concat.csv
diff --git a/integration_tests/data/cross_db/data_date_trunc.csv b/integration_tests/data/cross_db/data_date_trunc.csv
diff --git a/integration_tests/data/cross_db/data_dateadd.csv b/integration_tests/data/cross_db/data_dateadd.csv