Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fixed perplexity_color extension #6485

Open
wants to merge 4 commits into
base: dev
Choose a base branch
from
Open

fixed perplexity_color extension #6485

wants to merge 4 commits into from

Conversation

KT313
Copy link

@KT313 KT313 commented Oct 22, 2024

Checklist:

non-hoverable mode:

  • improved colors
  • improved / fixed formatting

hoverable mode:

  • fixed (the worst of) formatting problems

@KT313 KT313 marked this pull request as draft October 23, 2024 06:16
@KT313 KT313 marked this pull request as ready for review October 23, 2024 07:09
@KT313
Copy link
Author

KT313 commented Oct 23, 2024

so as it turns out, for some tokenizers like transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast from TinyMistral-248M, the tokenizer acts up when you try to decode single id's; it will always strip whitespaces from the tokens. However it seems to work as intended for most models like Hermes, which uses transformers.tokenization_utils_fast.PreTrainedTokenizerFast.

@oobabooga
Copy link
Owner

@KT313 thanks for the many fixes, this looks good. Could you merge the dev branch and give it an additional test? I have made a test with google/gemma-2-9b-it and I am not sure if the extension is working or not (the colors look homogeneous, maybe it's my settings).

@KT313
Copy link
Author

KT313 commented Dec 19, 2024

@oobabooga i can't test it right now since i'm busy, but it worked for me when i made the pull request. Depending on which options you check, the colors either have a red hue depending on perplexity, or between green and red iirc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants