Tokenizer vocabulary

Qwen2

Qwen2 Hugging Face ByteLevel BPE tokenizer with NFC normalization and chat special tokens. Vocabulary size, token ranges, and special-token IDs are listed here.

Alibaba

Creator

Alibaba

Mergeable tokens

151,643

Total known tokens

151,646

Browse by type

PunctuationSymbols and marks WhitespaceSpaces, tabs, newlines EmojiPictographs and fragments MultilingualExamples across scripts CodeProgramming tokens Control/special tokensReserved tokenizer values

Open token index