README.md: minor improvements

knpubscripts · Dec 16, 2022 · ab3688a · ab3688a
1 parent 1f098ca
commit ab3688a
Showing 1 changed file with 3 additions and 2 deletions.
diff --git a/README.md b/README.md
@@ -1,11 +1,12 @@
 # ⏳ tiktoken
 
-tiktoken is a fast tokeniser.
+tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with
+OpenAI's models.
 
 ```python
 import tiktoken
 enc = tiktoken.get_encoding("gpt2")
-print(enc.encode("hello world"))
+assert enc.decode(enc.encode("hello world")) == "hello world"
 ```
 
 The open source version of `tiktoken` can be installed from PyPI: