ToG-Tools

Some of the tool functions that may be used are kept here, including:

1. Convert jsonl to json files in jsonl2json.py
2. Remove duplicate elements from json file based on 'question' key in de_duplicate.py
3. Random sampling n datasets from json file and save to the new json file in split_dataset.py

This folder will be updated frequently, and users can define new functions according to their needs.