4 Pull requests merged by 1 person

• Fix test_hf_tokenizer

  #95 merged Jul 2, 2025

• Add support for more behavior in Split pretokenizer

  #93 merged Jul 2, 2025

• Use a small tokenizer.json for unit test (#92)

  #94 merged Jul 2, 2025

• Use a small tokenizer.json for unit test

  #92 merged Jul 2, 2025

2 Pull requests opened by 1 person

• Support gemma3 HF tokenizer.json

  #96 opened Jul 3, 2025

• Add Package.swift

  #97 opened Jul 7, 2025

1 Unresolved conversation

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• Fix CQS signal performance-faster-string-find in fbcode/pytorch/tokenizers/src

  #86 commented on Jul 5, 2025 • 0 new comments