Skip to content

Latest commit

 

History

History
543 lines (315 loc) · 54.3 KB

CHANGELOG.md

File metadata and controls

543 lines (315 loc) · 54.3 KB

Changelog

Untagged

3.1.0 (2023-04-21)

Features

  • add db2 hash and concat support (#800) (c16e2f7)
  • add Impala connection optional parameters (#743) (#790) (414d7f8)
  • added source_type in output while listing connections list (#803) (056275b)
  • Adding Custom-Query support for DB2. (#807) (a8085d3)
  • Option for simpler report output grid (#802) (b92eb91)

Bug Fixes

  • Mysql fix to support row hash validations, random row validation, and filter (#812) (ae07fa4)
  • schema validation fixes for Oracle/SQL Server float64 and SQL Server datetimeoffset (#796) (ad0e64f)

Documentation

  • add README for Airflow DAG sample, update code formatting in other docs (#722) (f4c3241)
  • score-cutoff changed to 1 (#779) (d3aabca)

3.0.0 (2023-03-28)

⚠ BREAKING CHANGES

  • issue673 optimize CLI tools arg parser (#701)

Features

  • ✨ Add support for source/target inline sql queries for validate custom-query command (#734) (c5e7a37)
  • gcp secret manger support for DVT (#704) (d6c40f1)
  • ibis_bigquery strftime support for DATETIME columns (#737) (b1141de)

Bug Fixes

  • Add support for numeric and precision with length and precision in Postgres Custom Query (#723) (742b77e)
  • Adding Decimal datatype support for MSSQL custom query validation (#771) (0d5c5eb)
  • Better detection of Oracle client (#736) (efce0b8)
  • Cater for query driven comparisons in date format override code (#733) (0a22643)
  • issue 740 teradata strftime function (#747) (9fd102a)
  • issue673 optimize CLI tools arg parser (#701) (26bb8e9)
  • Protect column and row validation calculated column names from Oracle 30 character identifier limit (#749) (89413c1)
  • remove secret manager warnings (#781) (7e72bfd)

Documentation

2.9.0 (2023-02-16)

Features

  • Added Partition support to generate multiple YAML config files (#653) (Issue #619,#662) (f79c308)
  • added run_id to output (#708) (17720f2)
  • Divert cast of PostgreSQL decimal with scale>0 to to_char (#721) (3542851)
  • Use centralized date/time format in order to compare row data across engines (#720) (0de823b)

Bug Fixes

  • Error handling for batch processing of config files (#663) (21a26af)
  • Protect non-date columns from astype(str) date workaround (#726) (489ee27)
  • schema validation fix for different base names of source and destination data types (#710) (d7b44b0)

Documentation

  • updated Oracle parameter from user_name to user and changed underscores to hypens across the document (#689) (8777e00)

2.8.0 (2023-01-19)

Features

  • Logic to add allow-list to support datatype matching with a provided list in case of mismatched datatypes between source and target (#643) (269f8dc)

Bug Fixes

  • making logmech as optional for TD connection (#665) (500caa3)

2.7.0 (2023-01-06)

Features

  • Add AlloyDB support (#645) (cfedc22)
  • Add Integration test for Oracle (#651) (de3bbcc)
  • Added custom query support for Oracle (#646) (3f8771a)
  • Added custom query support for PostgreSQL (#644) (88dcfd3)
  • extend TO_CHAR to cover date, time and timestamp types (#641) (e0c184f)
  • SQL Server custom query support (#640) (98ab010)
  • Support config directory for running validations and add multithreading for DB queries (#654) (c67b51a)
  • Support custom calculated fields (#637) (14b506b)

2.6.0 (2022-11-28)

Features

Bug Fixes

  • bare data-validation command throws exception (#627) (7595c50)
  • column validation casing to allow for case-insensitive match (#626) (c694357)

2.5.0 (2022-10-18)

Features

  • adding scaffold for concatenate as a cli operation (#566) (ec4ef33)

Bug Fixes

  • Custom query validation throwing error with sql files ending with semicolon(;) (#591) (16a89ac)
  • Row validation optimization to avoid select all columns (#599) (de3758e)
  • update function to return non-unicode string (#615) (e334c65)

2.4.0 (2022-10-06)

⚠ BREAKING CHANGES

  • Add Python 3.10 support (#564)

Features

Miscellaneous Chores

2.3.0 (2022-09-15)

Features

  • Addition of log level as an argument for DVT logging and replac… (#577) (dbd9bc3)
  • Oracle row level validation support (#583) (489654c)

Bug Fixes

  • Add RawSQL support for Postgres and SQL Server (#576) (0693782)
  • fixing String to varchar for teradata (a979931)
  • random rows with filter option (#582) (da4faaf)
  • support NUMBER with no precision/scale (#572) (03219ba)
  • Teradata limit on column name, bug when casting to VARCHAR (#580) (c8700be)

Documentation

  • remove snowflake, add row supported DBs (#587) (1d923f5)

2.2.0 (2022-08-29)

⚠ BREAKING CHANGES

  • Added teradata custom query support (#547)

Features

  • Added teradata custom query support (#547) (97c3203)
  • Improve schema validation debugging, Support DATE for Hive validations (#558) (e67de5b)
  • Support for MSSQL row validation (#570) (61dabe0)

Bug Fixes

Miscellaneous Chores

2.1.0 (2022-07-14)

Features

  • new flag to exclude columns from schema validation (#507) (53ac41a)
  • Remove dependency on tables list for custom query (#541) (7dca5bd)

Bug Fixes

Documentation

2.0.1 (2022-06-10)

Bug Fixes

  • Schema validation to make case insensitive column name comparision (#500) (ee8c542)

2.0.0 (2022-05-26)

⚠ BREAKING CHANGES

  • Add 'primary_keys' and 'num_random_rows' fields to result handler (#372)

Features

  • Add 'primary_keys' and 'num_random_rows' fields to result handler (#372) (b123279)
  • add a new DAG example to run DVT (#485) (e3dd7ed)
  • adding impala random function (#483) (93d2072)
  • Enable sum/avg/bit_xor for BigQuery datetime type (#488) (083de07)

Documentation

1.7.2 (2022-05-12)

⚠ BREAKING CHANGES

  • Adds custom query row level hash validation feature. (#440)

Features

  • Add example of BigQuery cast to NUMERIC, update chore release version (#476) (50fac28)
  • Adds custom query row level hash validation feature. (#440) (f057fe8)
  • Issue356 db2 test (#383) (70fb7bc)
  • Support cast to BIGINT before aggregation (#461) (ca598a0)
  • support float and decimal types in Hive (#470) (5936f60)

Bug Fixes

  • add get_ibis_table_schema (#410) (#411) (4093625)
  • only replaces datatypes and not column names (#453) (6143794)
  • supports NULL datetime/timestamps, fixes bug with validation_status in PR 455 (#460) (57896f4)
  • Updated schema validation logic to column as 'validation_status' (#455) (e30c337)
  • updating teradata docs for sha256 UDF and swapping string_join for concat (#457) (23dbf56)

1.7.1 (2022-04-14)

⚠ BREAKING CHANGES

  • Changed result schema 'status' column to 'validation_status' (#420)

Features

Bug Fixes

  • bug introduced with new pr (#429) (a6cf3f0)
  • Hash all bug, noxfile updates (#413) (fc73e21)
  • Hive boolean nan to None, Unsupported ibis data types in structs and arrays (#444) (e94a1da)
  • ibis default sql option limits query results at 10k rows (#418) (7539efe)
  • Impala strings/objects now return None instead of NaN (#406) (9d3c5ec)
  • issue 265 add cloud spanner functionality (#394) (783cdf8)
  • support labels for schema validation (#260) (#381) (f787701)
  • Treat both source and target values being NULL as a success (#437) (c4da5ca)

Miscellaneous Chores

1.7.0 (2022-03-23)

Features

Bug Fixes

  • add to_hex for bigquery hash (#400) (e5c7ded)
  • Comparison fields Key Error fix (#396) (a597b56)
  • ensure all statuses are success or fail, particularly after _join_pivots (#329) (#370) (310747d)
  • make status values consistent across validation types (#377) (#378) (5c08463)
  • Multiple updates (#359) (6b2614d)
  • revert change from #345 that causes filters, threshold and labels to be ignored for column validations (#376) (#379) (8b295cf)
  • Status when source and target agg values are 0 (#393) (6a41f68)
  • support schema validation for more clients (#355) (#380) (ed46295)
  • supporting non default schemas for mssql (#365) (100b3ea)
  • test for nan when calculating fail/success in combiner (#341) (#366) (a9720c2)
  • use an appropriate column filter list for schema validation (#350) (#371) (806151a)

Documentation

  • Add Hive as a supported data source to docs (#354) (be2a49d)

1.6.0 (2021-12-01)

Features

Bug Fixes

Documentation

1.5.0 (2021-10-19)

Features

  • added kerberos service name flag for Impala connections, fixed bug in row validation with YAML (#320) (351994c)
  • Track DVT GCS connections (#326) (b384b1f)

Bug Fixes

Documentation

1.4.0 (2021-09-30)

Features

  • add state manager client (#311) (e893ea5)
  • Allow user to specify a format for stdout (#242) (#293) (f0a9fa1)
  • Allow user to specify a format for stdout T2 (#242) (#296) (ec1af22)
  • cast aggregates (#306) (e3da4c3)
  • Issue262 impala connect (#281) (eaa052f)
  • logic to deploy dvt on Cloud Run (#280) (9076286)
  • promote 3.9 to main version (as it is in Cloudtops now for local testing) and add a small unit test for persoanl use (#292) (eb0f21a)
  • Refactor CLI to fit Command Pattern (#303) (f6d2b9d)
  • Updated Cloud Functions sample (#297) (923413d)

Bug Fixes

  • updated code so that BQ target schema would not set to None for FileSystem to BQ validations (#309) (5016d65)

1.3.2 (2021-06-29)

Documentation

1.3.1 (2021-06-28)

Documentation

1.3.0 (2021-06-28)

Features

  • add table matching score as a param incase adjusted is needed (#267) (b02aed5)
  • CI/CD Release to PyPi via Cloud Build (#258) (0870fc7)

Bug Fixes

  • correct issues blocking impala and hive (#266) (5110d1f)

1.2.0 (2021-05-27)

Features

Bug Fixes

1.1.8

  • Adding and documenting find-tables CLI feature with schema filter
  • Correct filter errors caused by SQL Alchemy errors
  • Adding beta calculated fields logic

1.1.7

  • Adding tests to validate BIGNUMERIC BQ type behavior

1.1.6

  • Minor fix for Teradata client from breaking IBis changes

1.1.5

  • Add support for running raw queries against a connection
  • Upgraded Ibis to v1.4 with large client organizational and design changes
  • Added support for "use_no_lock_tables" Teradata config to optionally avoid table locking

1.1.4

  • Added an options to add key:value labels to validation runs
  • Oracle and SQL Alchemy now support RawSql filters
  • Add support for Cloud Functions in samples
  • Added schema information to result set

1.1.3

  • Release find-tables logic too help build table lists
  • Teradata client improvements
  • Remove rarely used dependencies into extras

1.1.2

  • Teradata numeric column and general bug fixes
  • Fix Ibis query compliation order causing cross join

1.1.1

  • Bug fixes to support case insensitivity
  • Allow null values to be handled in grouped columns
  • Oracle client improvements

1.1.0

  • Added Row validations for cell level validation with primary keys
  • Client support for Oracle, SQL Server, Postgres, and GCS files

1.0

  • Support for Column and GroupedColumn validations
  • Allow custom filter via YAML config
  • BigQuery result handlers supported
  • Client support for BigQuery, MySQL, and Teradata

0.1.1 (release date TBD)

Bug Fixes

  • update BigQuery dependencies to fix group-by results handler #64

Documentation

  • remove references to unsupported validations from README #63
  • includes wheel file installation steps in README #57
  • add filters and data sources to README #56

Internal / Testing Changes

  • move ibis addons to third-party directory #61

0.1.0 (2020-07-16)

Initial alpha release.

Features

  • Add data-validation CLI, which can run from CLI arguments, store a configuration YAML file, or run from a run-config YAML file.
  • Add support for querying Teradata.
  • Add support for querying BigQuery.
  • Write report output to BigQuery.

Dependencies

  • To use Teradata support, you must manually install the teradatasql PIP package.

Documentation

  • See the README.md file for getting started instructions.