Token category

p50k_base Emoji

Emoji often tokenize as multiple UTF-8 byte fragments or mixed symbol pieces. Skin tones, zero-width joiners, and flags can span several tokens.

Loading tokens...