Automation with Kubeflow Pipelines

Map betweens transforms and KFP pipelines

Transform	KFP pipeline
language/lang_id	lang_id_wf.py
language/html2parquet	html2parquet_wf.py
code/malware	malware_wf.py
code/code2parquet	code2parquet_wf.py
code/code_quality	code_quality_wf.py
code/proglang_select	proglang_select_wf.py
code/license_select	license_select_wf.py
universal/doc_id	doc_id_wf.py
universal/ededup	ededup_wf.py
universal/fdedup	fdedup_wf.py
universal/filtering	filter_wf.py
universal/noop	noop_wf.py
universal/profiler	profiler_wf.py
universal/tokenization	tokenization_wf.py
universal/hap	hap_wf.py

Set up and working steps

• Set up a Kubernetes clusters for KFP execution
• Simple Transform pipeline tutorial
• Execution several transformers
• Clean up the cluster