[BUG] NER prediction : mismatch between number of input tokens and output labels

[fadhleryani]

Howdy owo!

check this out:
Given a string such as the following (note the misplaced shadda after ان):
'أوهاشي، أن ّ "لأجل تحقيق التنمية المرجوّة'
simple_word_tokenize returns 9 tokens. These are then passed to the ner predictor, which returns 8 labels.

Desktop (please complete the following information):

• OS: macOS 11.5.2
• Python version: 3.9.13
• CAMeL Tools version as well as installation source (pip, conda, source). If installed from source, specify which branch [e.g. master] and/or commit hash: 1.4.1 (from pip)