Benchmarks

This directory contains several benchmarks, measuring in particular encoding-decoding speeds, training speeds and sequence lengths. They are intended to give you reference points and help you to choose your tokenization parameters.