Skip to content

Latest commit

 

History

History
73 lines (50 loc) · 2.59 KB

release-notes.md

File metadata and controls

73 lines (50 loc) · 2.59 KB
Raw

Data Prep Kit Release notes

Release 0.2.1 - 9/24/2024

General

  1. Bug fixes across the repo
  2. Added AI Alliance RAG demo, tutorials and notebooks and tips for running on google colab
  3. Added new transforms and single package for transforms published to pypi
  4. Improved CI/CD with targeted workflow triggered on specific changes to specific modules
  5. New enhancements for cutting a release

data-prep-toolkit libraries (python, ray, spark)

  1. Restructure the repository to distinguish/separate runtime libraries
  2. Split data-processing-lib/ray into python and ray
  3. Spark runtime
  4. Updated pyarrow version
  5. Define required transform() method as abstract to AbstractTableTransform
  6. Enables configuration of makefile to use src or pypi for data-prep-kit library dependencies

KFP Workloads

  1. Add a configurable timeout before destroying the deployed Ray cluster.

Transforms

  1. Added 7 new transdforms including: language identification, profiler, repo level ordering, doc quality, pdf2parquet, HTML2Parquet and PII Transform
  2. Added ededup python implementation and incremental ededup
  3. Added fuzzy floating point comparison

Release 0.2.0 - 6/27/2024

General

  1. Many bug fixes across the repo, plus the following specifics.
  2. Enhanced CI/CD and makefile improvements include definition of top-level targets (clean, set-verions, build, publish, test)
  3. Automation of release process branch/tag management
  4. Documentation improvements

data-prep-toolkit libraries (python, ray, spark)

  1. Split libraries into 3 runtime-specific implementations
  2. Fix missing final count of processed and add percentages
  3. Improved fault tolerance in python and ray runtimes
  4. Report global DataAccess retry metric
  5. Support for binary data transforms
  6. Updated to Ray version to 2.24
  7. Updated to PyArrow version 16.1.0

KFP Workloads

  1. Add KFP V2 support
  2. Create a distinct (timestamped) execution.log file for each retry
  3. Support for multiple inputs/outputs

Transforms

  1. Added language/lang_id - detects language in documents
  2. Added universal/profiler - counts works/tokens in documents
  3. Converted ingest2parquet tool to transform named code2parquet
  4. Split transforms, as appropriate, into python, ray and/or spark.
  5. Added spark implementations of filter, doc_id and noop transforms.
  6. Switch from using requirements.txt to pyproject.toml file for each transform runtime
  7. Repository restructured to move kfp workflow definitions to associated transform project directory

Release 0.1.1 - 5/24/2024

Release 0.1.0 - 5/15/2024

Release 0.1.0 - 5/08/2024