Tokenizer alias

GPT-2 tokenizer

GPT-2 tokenizer reference using the r50k_base vocabulary, with token counts and token ID lookup.

OpenAI

Creator
OpenAI
Vocabulary
r50k_base
Total known tokens
50,257
Mergeable tokens
50,256
Token ID range
0-50,255
Data
r50k_base
Open tokenizer Open token index