Name		Name	Last commit message	Last commit date
Latest commit History 13 Commits
NER_Tagging/HashtagSeg		NER_Tagging/HashtagSeg
README.md		README.md
annotation_analysis.ipynb		annotation_analysis.ipynb
annotations.json		annotations.json
dataset_prep.ipynb		dataset_prep.ipynb
full_pipeline.ipynb		full_pipeline.ipynb
loadDataset.py		loadDataset.py
multi_token_hashtags_with_source.tsv		multi_token_hashtags_with_source.tsv
test.conll		test.conll

Repository files navigation

hashtagseg

the training samples as shared by @akshala, slight change to that
- for ex: "German" - "BIIIII"
- changed to : "German" - "BBBBBB"
those samples are saved in conll format for ease in loading. fields are [id,form,tag]
test.conll : doesnt mean that it is test file. temporary name for the file.
test.conll is referenced in loadDataset.py
loadDataset.py : used to load the conll format file in HuggingFace's Datasets library format. for ease of training.
full_pipeline.ipynb : used for performing hashtag segmentation