Tokenization is an essential preprocessing and decoding stage in language models. A tokenizer applies a vocabulary and a set of segmentation rules to convert text into token units. Depending on the tokenizer, a token may represent a complete word, a common word fragment, punctuation, whitespace, or a byte-level sequence. Subword tokenization methods such as Byte Pair Encoding (BPE) and WordPiece help balance vocabulary size and the ability to represent unfamiliar words. After segmentation, each token is mapped to an integer ID. These IDs are then converted into embedding vectors before being processed by the neural network. During generation, the model predicts token IDs, which the tokenizer decodes into text. Tokenization affects context length because models generally process a limited number of tokens at a time. It also influences processing cost and how efficiently different languages, symbols, and technical terms are represented. The same visible word may be represented by one token or several tokens depending on the tokenizer. Therefore, token count should not be assumed to equal word count.
Tokenization in Large Language Models
Tokenization is the process of converting text into smaller units called tokens so that a language model can process it numerically. Tokens may represent words, subwords, individual characters, or other byte-level units.
T = Tokenize(text); IDs = Map(T, Vocabulary)
Explanation
Variables
| Symbol | Meaning |
|---|---|
T | Sequence of tokens produced by tokenization |
text | Original input text |
IDs | Numerical identifiers assigned to tokens |
Vocabulary | Collection of token units recognized by the tokenizer |
Example or application
The word 'unhappiness' may be divided into smaller units such as 'un', 'happi', and 'ness' by a subword tokenizer. The exact segmentation depends on the tokenizer's vocabulary and rules.
Text
LLMs do not usually process raw text directly. A tokenizer divides text into tokens and maps each token to a numerical ID that the model can process. The model's output token IDs are converted back into readable text.
Key Points
Tokenization converts text into model-readable units.
Tokens are mapped to numerical IDs.
Tokens may be words, subwords, characters, or byte-level units.
BPE and WordPiece are examples of subword tokenization methods.
Tokenization affects context length and processing cost.
Applications
LLM input preprocessing
Text generation and decoding
Context-window management
Multilingual language processing
Code language modeling
Text data preparation
Learning Outcomes
Define tokenization in the context of LLMs.
Distinguish words from tokens.
Explain how token IDs are used by language models.
Identify why tokenization affects model performance.
Practice Questions
What is a token in a language model?
Why are numerical token IDs needed?
What is subword tokenization?
How can tokenization affect a model's context window?
Related Concepts
Large Language Models
Byte Pair Encoding
WordPiece
Vocabulary
Embeddings
Context Window
Common Mistakes
Assuming one word always equals one token.
Assuming all tokenizers split text in the same way.
Confusing token IDs with embedding vectors.
Ignoring how tokenization affects multilingual text.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Sennrich, R., Haddow, B., and Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. Proceedings of ACL.
- Kudo, T., and Richardson, J. (2018). SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer. Proceedings of EMNLP.
- Jurafsky, D., and Martin, J. H. Speech and Language Processing. Draft edition.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning