Tokenizer alias

GPT-2 tokenizer

GPT-2 tokenizer alias for r50k_base.

OpenAI

Creator
OpenAI
Vocabulary
r50k_base
Total known tokens
50,257
Mergeable tokens
50,256
Token ID range
0-50,255
Data
r50k_base
Open tokenizer Open token index