Tokenizer index

Tokenizers

Byte-pair encoding vocabularies with token counts, special tokens, and per-token lookup.

Available tokenizers

OpenAI

cl100k_base 100,261 tokens

OpenAI

o200k_base 200,000 tokens

OpenAI

r50k_base 50,257 tokens

OpenAI

p50k_base 50,281 tokens

Mistral

Mistral Tekken v3 131,072 tokens

Alibaba

Qwen2 151,646 tokens

Alibaba

Qwen2.5 151,657 tokens

DeepSeek

DeepSeek V3 128,804 tokens