Various fixes to workflows, especially kfp (IBM#664)

* disable test workflow when none code files change Signed-off-by: David Wood <[email protected]> * one more path-ignore in test.yml Signed-off-by: David Wood <[email protected]> * one more fix for path-ignore in test.yml Signed-off-by: David Wood <[email protected]> * test universal transform separately Signed-off-by: David Wood <[email protected]> * rename test universal workflow Signed-off-by: David Wood <[email protected]> * add comments to noop src to trigger new universal test workflow Signed-off-by: David Wood <[email protected]> * fix paths in test universal workflow Signed-off-by: David Wood <[email protected]> * addj back ignore paths in test universal workflow Signed-off-by: David Wood <[email protected]> * another noop comment Signed-off-by: David Wood <[email protected]> * move ignored paths to paths in univesal test workflow Signed-off-by: David Wood <[email protected]> * test-universal workflow name changes Signed-off-by: David Wood <[email protected]> * noop comments Signed-off-by: David Wood <[email protected]> * noop readme change' Signed-off-by: David Wood <[email protected]> * change test universal not paths Signed-off-by: David Wood <[email protected]> * disable all but new noop and doc_id test workflows Signed-off-by: David Wood <[email protected]> * code change in noop Signed-off-by: David Wood <[email protected]> * remake test transforms Signed-off-by: David Wood <[email protected]> * add individual test transform workflows Signed-off-by: David Wood <[email protected]> * noop README change Signed-off-by: David Wood <[email protected]> * better ignore of .md on test transform workflows Signed-off-by: David Wood <[email protected]> * noop readme change Signed-off-by: David Wood <[email protected]> * noop test transform worklow 1 ignore Signed-off-by: David Wood <[email protected]> * noop readme Signed-off-by: David Wood <[email protected]> * split out the tests into test-kfp/lib/misc and remove test.yml, add readme Signed-off-by: David Wood <[email protected]> * test-kfp only on kfp/** Signed-off-by: David Wood <[email protected]> * noop code change to trigger build Signed-off-by: David Wood <[email protected]> * comments in workflows Signed-off-by: David Wood <[email protected]> * updated workflow readme Signed-off-by: David Wood <[email protected]> * only run build-library workflow on data-processing-lib changes Signed-off-by: David Wood <[email protected]> * try and ignore docs in build-library, test-kfp/lib Signed-off-by: David Wood <[email protected]> * workflow title changes for consistency Signed-off-by: David Wood <[email protected]> * test change on filter source Signed-off-by: David Wood <[email protected]> * change to lib readme Signed-off-by: David Wood <[email protected]> * change to lib source Signed-off-by: David Wood <[email protected]> * minor job name changes in transform workflows Signed-off-by: David Wood <[email protected]> * noop readme Signed-off-by: David Wood <[email protected]> * test-lib workflow ignores Signed-off-by: David Wood <[email protected]> * top level readme Signed-off-by: David Wood <[email protected]> * noop test source Signed-off-by: David Wood <[email protected]> * filter source change' Signed-off-by: David Wood <[email protected]> * updated all transform tets workflows Signed-off-by: David Wood <[email protected]> * fix typo in test template on check_images Signed-off-by: David Wood <[email protected]> * noop src change Signed-off-by: David Wood <[email protected]> * check for makefile in test transform workflow Signed-off-by: David Wood <[email protected]> * automatically determine transforms in transforms directory for which to generate test workflows Signed-off-by: David Wood <[email protected]> * worklow readme, transform existence verification, disable build-library, and tools tests Signed-off-by: David Wood <[email protected]> * workflow readme details on kfp and misc tests Signed-off-by: David Wood <[email protected]> * backing out change to dpk lib code Signed-off-by: David Wood <[email protected]> * restore filter code Signed-off-by: David Wood <[email protected]> * restore noop code Signed-off-by: David Wood <[email protected]> * workflow readme Signed-off-by: David Wood <[email protected]> * really restore noop code Signed-off-by: David Wood <[email protected]> * check for makefile in transform test-src testing Signed-off-by: David Wood <[email protected]> * don't include lib test dependencies in transform test workflows Signed-off-by: David Wood <[email protected]> * noop code change Signed-off-by: David Wood <[email protected]> * disable noop, don't include lib test-data in transform dependencies Signed-off-by: David Wood <[email protected]> * use job.id.if on Makefile to enable transform test job Signed-off-by: David Wood <[email protected]> * use job.id.if on Makefile to enable transform test job Signed-off-by: David Wood <[email protected]> * restore noop Makefile Signed-off-by: David Wood <[email protected]> * exclude kfp_ray from transfor test workflow and change noop code Signed-off-by: David Wood <[email protected]> * remove if: from test workflows Signed-off-by: David Wood <[email protected]> * backout noop code change Signed-off-by: David Wood <[email protected]> * backout noop code change Signed-off-by: David Wood <[email protected]> * only build spark image for transform image tests that need it Signed-off-by: David Wood <[email protected]> * header_cleanser code change Signed-off-by: David Wood <[email protected]> * only build spark image for transform image tests that need it Signed-off-by: David Wood <[email protected]> * update workflows to trigger on .make.* changes Signed-off-by: David Wood <[email protected]> * blank line added to .make.versions to test new workflows Signed-off-by: David Wood <[email protected]> * remove change to .make.versions Signed-off-by: David Wood <[email protected]> * remove test-universal-html2parquet.yml since it is now moved to language Signed-off-by: David Wood <[email protected]> * mv kfp blacklist definition to check-workflows.sh script and have it check for kfp workflows Signed-off-by: David Wood <[email protected]> * license_select_wf.py comment to trigger kfp tests here Signed-off-by: David Wood <[email protected]> * have transform top level makefile check for kfp_ray directory before recursing into Signed-off-by: David Wood <[email protected]> * fix doc_quality Makefile kfp rules Signed-off-by: David Wood <[email protected]> * fix tabbing in recent Makefile updates Signed-off-by: David Wood <[email protected]> * fix kfp workflows to only build the target transform Signed-off-by: David Wood <[email protected]> * switch workflows from ubuntu-22.04 to ubuntu-latest to try and avoid excessive job queuing Signed-off-by: David Wood <[email protected]> * remove non-change from license_select_wf.py to trigger new ci/cd Signed-off-by: David Wood <[email protected]> * one more fix to kfp transform workflows to not workflow-build in all transforms Signed-off-by: David Wood <[email protected]> * fix repo_level_ordering/ray/Makefile to support kind/kfp testing targets Signed-off-by: David Wood <[email protected]> * change kfp test workflows to not test if transform's Makefile or kfp_ray dir is not present Signed-off-by: David Wood <[email protected]> * set cancel-in-progress=true in workflows Signed-off-by: David Wood <[email protected]> --------- Signed-off-by: David Wood <[email protected]>
pankajskku · Oct 3, 2024 · a5f86da · a5f86da
1 parent 0a9fe37
commit a5f86da
Show file tree

Hide file tree

Showing 78 changed files with 1,924 additions and 1,019 deletions.
diff --git a/.github/workflows/Makefile b/.github/workflows/Makefile
@@ -8,7 +8,6 @@ LANG_TRANSFORMS=doc_chunk doc_quality lang_id pdf2parquet pii_redactor text_enco
 
 
 # A list that holds transforms that should not be tested with KFP
-KFP_BLACK_LIST="doc_chunk,pdf2parquet,pii_redactor"
 
 transform-tests:
  $(MAKE) TRANSFORM_SUBDIR=universal .transform-tests 
@@ -29,13 +28,19 @@ transform-tests:
  done
 
 .transform-kfp-tests:
- @for i in $$(find ../../transforms/$(TRANSFORM_SUBDIR) -mindepth 1 -maxdepth 1 -type d); do \
+ @KFP_BLACK_LIST=$$(cd ../..; bash scripts/check-workflows.sh -show-kfp-black-list); \
+ for i in $$(find ../../transforms/$(TRANSFORM_SUBDIR) -mindepth 1 -maxdepth 1 -type d); do \
  dir=$$(basename $$i); \
- z=$$(echo ${KFP_BLACK_LIST} | grep -v $$dir); \
- if [ ! -d ../../transforms/$(TRANSFORM_SUBDIR)/$$dir/kfp_ray ] || [ -z "$$z" ]; then \
+ yml=test-$(TRANSFORM_SUBDIR)-$$dir-kfp.yml; \
+ if [ ! -d ../../transforms/$(TRANSFORM_SUBDIR)/$$dir/kfp_ray ]; then \
+ echo No kfp_ray directory for $$dir. Skipping generation of $$yml; \
+ continue; \
+ fi; \
+ z=$$(echo $${KFP_BLACK_LIST} | grep $$dir); \
+ if [ ! -z "$$z" ]; then \
+ echo $$dir is black listed. Skipping generation of $$yml; \
  continue; \
  fi; \
- yml=test-$(TRANSFORM_SUBDIR)-$$dir-kfp.yml; \
  echo Generating $$yml; \
  cat test-kfp-transform.template | sed -e "s?@TARGET_TRANSFORM_DIR@?transforms/$${TRANSFORM_SUBDIR}/$$dir?g" > $$yml; \
  done

diff --git a/.github/workflows/deploy-docs.yml b/.github/workflows/deploy-docs.yml
@@ -8,7 +8,7 @@ on:
  - "releases/**"
 jobs:
  deploy:
- runs-on: ubuntu-22.04
+ runs-on: ubuntu-latest
  env:
  REPO_URL: "https://github.com/${{ github.repository }}"
  REPO_BRANCH: "dev"

diff --git a/.github/workflows/deploy-library.yml b/.github/workflows/deploy-library.yml
@@ -14,7 +14,7 @@ permissions:
 jobs:
  build-package:
  name: Build Ray data processing libraries
- runs-on: ubuntu-22.04
+ runs-on: ubuntu-latest
  steps:
  - name: Checkout
  uses: actions/checkout@v4
@@ -30,7 +30,7 @@ jobs:
  name: Publish packages to test.pypi.org
  # disabled
  if: false
- runs-on: ubuntu-22.04
+ runs-on: ubuntu-latest
  needs: build-package
 
  steps:
@@ -47,7 +47,7 @@ jobs:
 
  publish-pypi:
  name: Publish release to pypi.org
- runs-on: ubuntu-22.04
+ runs-on: ubuntu-latest
  needs: build-package
  # disabled as of now
  if: false

diff --git a/.github/workflows/deploy-transforms.yml b/.github/workflows/deploy-transforms.yml
@@ -9,7 +9,7 @@ on:
 jobs:
  build-images:
  name: Build and check images
- runs-on: ubuntu-22.04
+ runs-on: ubuntu-latest
  steps:
  - name: Checkout
  uses: actions/checkout@v4
@@ -23,7 +23,7 @@ jobs:
  name: Publish packages to quay.io
  # disabled
  if: false
- runs-on: ubuntu-22.04
+ runs-on: ubuntu-latest
  needs: build-images
 
  steps:

diff --git a/.github/workflows/test-code-code2parquet-kfp.yml b/.github/workflows/test-code-code2parquet-kfp.yml
@@ -12,6 +12,8 @@ on:
  tags:
  - "*"
  paths:
+ - ".make.*"
+ - "transforms/.make.workflow"
  - "transforms/code/code2parquet/**"
  - "!kfp/**" # This is tested in separate workflow
  - "!data-processing-lib/**" # This is tested in separate workflow
@@ -24,6 +26,8 @@ on:
  - "dev"
  - "releases/**"
  paths:
+ - ".make.*"
+ - "transforms/.make.workflow"
  - "transforms/code/code2parquet/**"
  - "!data-processing-lib/**" # This is tested in separate workflow
  - "!kfp/**" # This is tested in separate workflow
@@ -32,10 +36,14 @@ on:
  - "!**/images/**"
  - "!**.gitignore"
 
+# taken from https://stackoverflow.com/questions/66335225/how-to-cancel-previous-runs-in-the-pr-when-you-push-new-commitsupdate-the-curre
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+ cancel-in-progress: true
 
 jobs:
  test-kfp-v1:
- runs-on: ubuntu-22.04
+ runs-on: ubuntu-latest
  steps:
  - name: Checkout
  uses: actions/checkout@v4
@@ -51,30 +59,34 @@ jobs:
  - name: Test KFP libs (shared and v1) and run a workflow
  timeout-minutes: 120
  run: |
- export REPOROOT=$PWD
- export K8S_SETUP_SCRIPTS=$PWD/scripts/k8s-setup
- source $K8S_SETUP_SCRIPTS/requirements.env
- export PATH=$PATH:/tmp/
- curl -Lo /tmp/kind https://kind.sigs.k8s.io/dl/v${KIND_VERSION}/kind-linux-amd64
- chmod 777 /tmp/kind
- curl -fsSL -o /tmp/get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
- chmod 700 /tmp/get_helm.sh
- HELM_INSTALL_DIR=/tmp/ /tmp/get_helm.sh -v v${HELM_VERSION} --no-sudo
- chmod 777 /tmp/helm
- curl -L https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl -o /tmp/kubectl
- chmod 777 /tmp/kubectl
- curl https://dl.min.io/client/mc/release/linux-amd64/mc --create-dirs -o /tmp/mc
- chmod +x /tmp/mc
- export DEPLOY_KUBEFLOW=1
- make -C $K8S_SETUP_SCRIPTS setup
- make -C kfp/kfp_support_lib test
- make -C transforms workflow-build
- source $K8S_SETUP_SCRIPTS/common.sh
- make -C transforms/code/code2parquet workflow-test
- echo "Run transforms/code/code2parquet completed"
+ if [ -e "@TARGET_TRANSFORM_DIR/Makefile" -a -d "@TARGET_TRANSFORM_DIR/kfp_ray" ]; then
+ export REPOROOT=$PWD
+ export K8S_SETUP_SCRIPTS=$PWD/scripts/k8s-setup
+ source $K8S_SETUP_SCRIPTS/requirements.env
+ export PATH=$PATH:/tmp/
+ curl -Lo /tmp/kind https://kind.sigs.k8s.io/dl/v${KIND_VERSION}/kind-linux-amd64
+ chmod 777 /tmp/kind
+ curl -fsSL -o /tmp/get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
+ chmod 700 /tmp/get_helm.sh
+ HELM_INSTALL_DIR=/tmp/ /tmp/get_helm.sh -v v${HELM_VERSION} --no-sudo
+ chmod 777 /tmp/helm
+ curl -L https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl -o /tmp/kubectl
+ chmod 777 /tmp/kubectl
+ curl https://dl.min.io/client/mc/release/linux-amd64/mc --create-dirs -o /tmp/mc
+ chmod +x /tmp/mc
+ export DEPLOY_KUBEFLOW=1
+ make -C $K8S_SETUP_SCRIPTS setup
+ make -C kfp/kfp_support_lib test
+ make -C transforms/code/code2parquet workflow-build
+ source $K8S_SETUP_SCRIPTS/common.sh
+ make -C transforms/code/code2parquet workflow-test
+ echo "Run transforms/code/code2parquet completed"
+ else
+ echo "Skipping transforms/code/code2parquet kfp test for lack of Makefile and/or kfp_ray"
+ fi
 
  test-kfp-v2:
- runs-on: ubuntu-22.04
+ runs-on: ubuntu-latest
  steps:
  - name: Checkout
  uses: actions/checkout@v4
@@ -90,25 +102,29 @@ jobs:
  - name: Test KFP libs (shared and v2) and run a workflow
  timeout-minutes: 120
  run: |
- export REPOROOT=$PWD
- export K8S_SETUP_SCRIPTS=$PWD/scripts/k8s-setup
- source $K8S_SETUP_SCRIPTS/requirements.env
- export PATH=$PATH:/tmp/
- curl -Lo /tmp/kind https://kind.sigs.k8s.io/dl/v${KIND_VERSION}/kind-linux-amd64
- chmod 777 /tmp/kind
- curl -fsSL -o /tmp/get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
- chmod 700 /tmp/get_helm.sh
- HELM_INSTALL_DIR=/tmp/ /tmp/get_helm.sh -v v${HELM_VERSION} --no-sudo
- chmod 777 /tmp/helm
- curl -L https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl -o /tmp/kubectl
- chmod 777 /tmp/kubectl
- curl https://dl.min.io/client/mc/release/linux-amd64/mc --create-dirs -o /tmp/mc
- chmod +x /tmp/mc
- export DEPLOY_KUBEFLOW=1
- export KFPv2=1
- make -C $K8S_SETUP_SCRIPTS setup
- make -C kfp/kfp_support_lib test
- make -C transforms workflow-build
- source $K8S_SETUP_SCRIPTS/common.sh
- make -C transforms/code/code2parquet workflow-test
- header_text "Run transforms/code/code2parquet completed"
+ if [ -e "@TARGET_TRANSFORM_DIR/Makefile" -a -d "@TARGET_TRANSFORM_DIR/kfp_ray" ]; then
+ export REPOROOT=$PWD
+ export K8S_SETUP_SCRIPTS=$PWD/scripts/k8s-setup
+ source $K8S_SETUP_SCRIPTS/requirements.env
+ export PATH=$PATH:/tmp/
+ curl -Lo /tmp/kind https://kind.sigs.k8s.io/dl/v${KIND_VERSION}/kind-linux-amd64
+ chmod 777 /tmp/kind
+ curl -fsSL -o /tmp/get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
+ chmod 700 /tmp/get_helm.sh
+ HELM_INSTALL_DIR=/tmp/ /tmp/get_helm.sh -v v${HELM_VERSION} --no-sudo
+ chmod 777 /tmp/helm
+ curl -L https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl -o /tmp/kubectl
+ chmod 777 /tmp/kubectl
+ curl https://dl.min.io/client/mc/release/linux-amd64/mc --create-dirs -o /tmp/mc
+ chmod +x /tmp/mc
+ export DEPLOY_KUBEFLOW=1
+ export KFPv2=1
+ make -C $K8S_SETUP_SCRIPTS setup
+ make -C kfp/kfp_support_lib test
+ make -C transforms/code/code2parquet workflow-build
+ source $K8S_SETUP_SCRIPTS/common.sh
+ make -C transforms/code/code2parquet workflow-test
+ echo "Run transforms/code/code2parquet completed"
+ else
+ echo "Skipping transforms/code/code2parquet kfp test for lack of Makefile and/or kfp_ray"
+ fi
diff --git a/.github/workflows/test-code-code2parquet.yml b/.github/workflows/test-code-code2parquet.yml
@@ -12,6 +12,8 @@ on:
  tags:
  - "*"
  paths:
+ - ".make.*"
+ - "transforms/.make.transforms"
  - "transforms/code/code2parquet/**"
  - "data-processing-lib/**"
  - "!transforms/code/code2parquet/**/kfp_ray/**" # This is/will be tested in separate workflow
@@ -26,6 +28,8 @@ on:
  - "dev"
  - "releases/**"
  paths:
+ - ".make.*"
+ - "transforms/.make.transforms"
  - "transforms/code/code2parquet/**"
  - "data-processing-lib/**"
  - "!transforms/code/code2parquet/**/kfp_ray/**" # This is/will be tested in separate workflow
@@ -36,13 +40,18 @@ on:
  - "!**/images/**"
  - "!**.gitignore"
 
+# Taken from https://stackoverflow.com/questions/66335225/how-to-cancel-previous-runs-in-the-pr-when-you-push-new-commitsupdate-the-curre
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+ cancel-in-progress: true
+
 jobs:
  check_if_push_image:
  # check whether the Docker images should be pushed to the remote repository
  # The images are pushed if it is a merge to dev branch or a new tag is created.
  # The latter being part of the release process.
  # The images tag is derived from the value of the DOCKER_IMAGE_VERSION variable set in the .make.versions file.
- runs-on: ubuntu-22.04
+ runs-on: ubuntu-latest
  outputs:
  publish_images: ${{ steps.version.outputs.publish_images }}
  steps:
@@ -59,7 +68,7 @@ jobs:
  fi
  echo "publish_images=$publish_images" >> "$GITHUB_OUTPUT"
  test-src:
- runs-on: ubuntu-22.04
+ runs-on: ubuntu-latest
  steps:
  - name: Checkout
  uses: actions/checkout@v4
@@ -81,7 +90,7 @@ jobs:
  fi
  test-image:
  needs: [check_if_push_image]
- runs-on: ubuntu-22.04
+ runs-on: ubuntu-latest
  timeout-minutes: 120
  env:
  DOCKER_REGISTRY_USER: ${{ secrets.DOCKER_REGISTRY_USER }}