Tags: chonkie-ai/autotiktokenizer
Toggle v0.2.2's commit message
Merge pull request #17 from chonkie-ai/development
Bump version to 0.2.2 and update __init__.py to reflect the new version
Toggle v0.2.1's commit message
Merge pull request #9 from bhavnicksm/development
Add support for WordPiece Tokenizers + Aya Family
Toggle v0.2.0's commit message
Merge pull request #8 from bhavnicksm/development
Remove tokenizers dependency from AutoTikTokenizers
Toggle v0.1.2's commit message
Merge pull request #7 from bhavnicksm/development
Bump version to 0.1.2 and update package metadata
Toggle v0.1.1's commit message
Merge pull request #4 from bhavnicksm/development
Bump version to 0.1.1 and enhance _AutoTikTokenizer functionality
Toggle v0.1.0's commit message
Merge pull request #3 from bhavnicksm/development
Add logo + Bump version to v0.1.0
Toggle v0.0.1a0's commit message
Merge pull request #2 from bhavnicksm/development
Add version and author metadata to AutoTikTokenizer
You can’t perform that action at this time.