Tokenizer vocabulary

Gemma 1 & 2 SentencePiece

Google's shared Gemma 1 and Gemma 2 SentencePiece BPE tokenizer, with a 256k vocabulary and byte fallback pieces. Vocabulary size, token ranges, and special-token IDs are listed here.