Transform | KFP pipeline |
---|---|
language/lang_id | lang_id_wf.py |
language/html2parquet | html2parquet_wf.py |
code/malware | malware_wf.py |
code/code2parquet | code2parquet_wf.py |
code/code_quality | code_quality_wf.py |
code/proglang_select | proglang_select_wf.py |
code/license_select | license_select_wf.py |
universal/doc_id | doc_id_wf.py |
universal/ededup | ededup_wf.py |
universal/fdedup | fdedup_wf.py |
universal/filtering | filter_wf.py |
universal/noop | noop_wf.py |
universal/profiler | profiler_wf.py |
universal/tokenization | tokenization_wf.py |
universal/hap | hap_wf.py |