Tokenizer vocabulary

Qwen2.5

Qwen2.5 Hugging Face ByteLevel BPE tokenizer with NFC normalization and expanded added-token metadata. Vocabulary size, token ranges, and special-token IDs are listed here.

Alibaba

Creator

Alibaba

Mergeable tokens

151,643

Total known tokens

151,657

Browse by type

PunctuationSymbols and marks WhitespaceSpaces, tabs, newlines EmojiPictographs and fragments MultilingualExamples across scripts CodeProgramming tokens Control/special tokensReserved tokenizer values

Open token index