Tokenizer index
Tokenizers
Byte-pair encoding vocabularies with token counts, special tokens, and per-token lookup.
Available tokenizers
OpenAI
cl100k_base 100,261 tokensOpenAI
o200k_base 200,000 tokensOpenAI
r50k_base 50,257 tokensOpenAI
p50k_base 50,281 tokensMistral
Mistral Tekken v3 131,072 tokensAlibaba
Qwen2 151,646 tokensAlibaba
Qwen2.5 151,657 tokensDeepSeek
DeepSeek V3 128,804 tokens