✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-122

Text Processing and Tokenization

Text processing prepares raw language data for computational tasks, while tokenization divides text into smaller units that can be represented and analyzed by NLP systems.

Explanation

Text processing refers to the operations used to prepare and transform raw text before or during natural language processing. Real-world text may contain inconsistent spacing, punctuation, special characters, spelling variations, HTML tags, and other elements that require attention.

Tokenization is one of the most important text-processing operations. It divides a sequence of text into smaller units called tokens. Depending on the task and model, a token may represent a complete word, part of a word, a character, or even a punctuation mark.

Common tokenization approaches include:

1. Sentence tokenization: Divides a document into individual sentences.
2. Word tokenization: Splits sentences into word-like units.
3. Character tokenization: Treats individual characters as tokens.
4. Subword tokenization: Divides words into smaller frequently occurring units. Methods such as Byte Pair Encoding (BPE) and WordPiece use this general approach.

For example, the sentence 'Natural language processing is useful.' may be divided into the tokens 'Natural', 'language', 'processing', 'is', 'useful', and '.'. A subword tokenizer may divide an unfamiliar word into several smaller units.

Modern Transformer models commonly use subword tokenization because it provides a balance between vocabulary size and the ability to represent unfamiliar words. However, tokenization rules differ across models, and the same sentence may produce different token sequences under different tokenizers.

Text processing can also include removing unwanted markup, handling Unicode characters, identifying language, and managing special tokens. These operations must be selected carefully because aggressive cleaning can remove meaningful information, such as punctuation, capitalization, or special symbols.

Example or application

The sentence 'I love learning Python!' can be divided into word-level tokens: 'I', 'love', 'learning', 'Python', and '!'. A subword tokenizer may divide 'learning' into smaller units depending on its vocabulary.

Text

Tokenization is a fundamental NLP operation that splits text into sentences, words, subwords, or other tokens. The selected tokenization method affects vocabulary size, model input, and how language is represented.

Key Points

  • Text processing prepares raw text for NLP tasks.

  • Tokenization divides text into smaller units called tokens.

  • Sentence, word, character, and subword tokenization are common approaches.

  • Subword tokenization helps represent unfamiliar words.

  • Different models can use different tokenization methods.

  • Unnecessary text cleaning can remove useful information.

Applications

  • Search engines

  • Language modeling

  • Machine translation

  • Text classification

  • Chatbots

  • Information extraction

  • Document processing

Learning Outcomes

  • Explain the purpose of text processing.

  • Define tokens and tokenization.

  • Differentiate word-level and subword tokenization.

  • Understand why modern language models use tokenizers.

Practice Questions

  • What is tokenization?

  • How does sentence tokenization differ from word tokenization?

  • Why is subword tokenization useful?

  • Can two NLP models tokenize the same sentence differently?

Related Concepts

  • Text Normalization

  • Vocabulary

  • Byte Pair Encoding

  • WordPiece

  • Language Modeling

  • Text Vectorization

Common Mistakes

  • Assuming every token represents one complete word.

  • Assuming all NLP models use the same tokenizer.

  • Removing punctuation without considering whether it carries meaning.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Jurafsky, D., and Martin, J. H., Speech and Language Processing, 3rd edition draft.
  • Sennrich, R., Haddow, B., and Birch, A., Neural Machine Translation of Rare Words with Subword Units, ACL, 2016.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning