Skip to content

Tags: chonkie-ai/autotiktokenizer

Tags

v0.2.2

Toggle v0.2.2's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Merge pull request #17 from chonkie-ai/development

Bump version to 0.2.2 and update __init__.py to reflect the new version

v0.2.1

Toggle v0.2.1's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Merge pull request #9 from bhavnicksm/development

Add support for WordPiece Tokenizers + Aya Family

v0.2.0

Toggle v0.2.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Merge pull request #8 from bhavnicksm/development

Remove tokenizers dependency from AutoTikTokenizers

v0.1.2

Toggle v0.1.2's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Merge pull request #7 from bhavnicksm/development

Bump version to 0.1.2 and update package metadata

v0.1.1

Toggle v0.1.1's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Merge pull request #4 from bhavnicksm/development

Bump version to 0.1.1 and enhance _AutoTikTokenizer functionality

v0.1.0

Toggle v0.1.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Merge pull request #3 from bhavnicksm/development

Add logo + Bump version to v0.1.0

v0.0.1a0

Toggle v0.0.1a0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Merge pull request #2 from bhavnicksm/development

Add version and author metadata to AutoTikTokenizer