✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-144

Tokenization in Large Language Models

Tokenization is the process of converting text into smaller units called tokens so that a language model can process it numerically. Tokens may represent words, subwords, individual characters, or other byte-level units.

T = Tokenize(text); IDs = Map(T, Vocabulary)

Explanation

Tokenization is an essential preprocessing and decoding stage in language models. A tokenizer applies a vocabulary and a set of segmentation rules to convert text into token units. Depending on the tokenizer, a token may represent a complete word, a common word fragment, punctuation, whitespace, or a byte-level sequence. Subword tokenization methods such as Byte Pair Encoding (BPE) and WordPiece help balance vocabulary size and the ability to represent unfamiliar words. After segmentation, each token is mapped to an integer ID. These IDs are then converted into embedding vectors before being processed by the neural network. During generation, the model predicts token IDs, which the tokenizer decodes into text. Tokenization affects context length because models generally process a limited number of tokens at a time. It also influences processing cost and how efficiently different languages, symbols, and technical terms are represented. The same visible word may be represented by one token or several tokens depending on the tokenizer. Therefore, token count should not be assumed to equal word count.

Variables

SymbolMeaning

T

Sequence of tokens produced by tokenization

text

Original input text

IDs

Numerical identifiers assigned to tokens

Vocabulary

Collection of token units recognized by the tokenizer

Example or application

The word 'unhappiness' may be divided into smaller units such as 'un', 'happi', and 'ness' by a subword tokenizer. The exact segmentation depends on the tokenizer's vocabulary and rules.

Text

LLMs do not usually process raw text directly. A tokenizer divides text into tokens and maps each token to a numerical ID that the model can process. The model's output token IDs are converted back into readable text.

Key Points

  • Tokenization converts text into model-readable units.

  • Tokens are mapped to numerical IDs.

  • Tokens may be words, subwords, characters, or byte-level units.

  • BPE and WordPiece are examples of subword tokenization methods.

  • Tokenization affects context length and processing cost.

Applications

  • LLM input preprocessing

  • Text generation and decoding

  • Context-window management

  • Multilingual language processing

  • Code language modeling

  • Text data preparation

Learning Outcomes

  • Define tokenization in the context of LLMs.

  • Distinguish words from tokens.

  • Explain how token IDs are used by language models.

  • Identify why tokenization affects model performance.

Practice Questions

  • What is a token in a language model?

  • Why are numerical token IDs needed?

  • What is subword tokenization?

  • How can tokenization affect a model's context window?

Related Concepts

  • Large Language Models

  • Byte Pair Encoding

  • WordPiece

  • Vocabulary

  • Embeddings

  • Context Window

Common Mistakes

  • Assuming one word always equals one token.

  • Assuming all tokenizers split text in the same way.

  • Confusing token IDs with embedding vectors.

  • Ignoring how tokenization affects multilingual text.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Sennrich, R., Haddow, B., and Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. Proceedings of ACL.
  • Kudo, T., and Richardson, J. (2018). SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer. Proceedings of EMNLP.
  • Jurafsky, D., and Martin, J. H. Speech and Language Processing. Draft edition.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning