Text processing refers to the operations used to prepare and transform raw text before or during natural language processing. Real-world text may contain inconsistent spacing, punctuation, special characters, spelling variations, HTML tags, and other elements that require attention.
Tokenization is one of the most important text-processing operations. It divides a sequence of text into smaller units called tokens. Depending on the task and model, a token may represent a complete word, part of a word, a character, or even a punctuation mark.
Common tokenization approaches include:
1. Sentence tokenization: Divides a document into individual sentences.
2. Word tokenization: Splits sentences into word-like units.
3. Character tokenization: Treats individual characters as tokens.
4. Subword tokenization: Divides words into smaller frequently occurring units. Methods such as Byte Pair Encoding (BPE) and WordPiece use this general approach.
For example, the sentence 'Natural language processing is useful.' may be divided into the tokens 'Natural', 'language', 'processing', 'is', 'useful', and '.'. A subword tokenizer may divide an unfamiliar word into several smaller units.
Modern Transformer models commonly use subword tokenization because it provides a balance between vocabulary size and the ability to represent unfamiliar words. However, tokenization rules differ across models, and the same sentence may produce different token sequences under different tokenizers.
Text processing can also include removing unwanted markup, handling Unicode characters, identifying language, and managing special tokens. These operations must be selected carefully because aggressive cleaning can remove meaningful information, such as punctuation, capitalization, or special symbols.
Text Processing and Tokenization
Text processing prepares raw language data for computational tasks, while tokenization divides text into smaller units that can be represented and analyzed by NLP systems.
Explanation
Example or application
The sentence 'I love learning Python!' can be divided into word-level tokens: 'I', 'love', 'learning', 'Python', and '!'. A subword tokenizer may divide 'learning' into smaller units depending on its vocabulary.
Text
Tokenization is a fundamental NLP operation that splits text into sentences, words, subwords, or other tokens. The selected tokenization method affects vocabulary size, model input, and how language is represented.
Key Points
Text processing prepares raw text for NLP tasks.
Tokenization divides text into smaller units called tokens.
Sentence, word, character, and subword tokenization are common approaches.
Subword tokenization helps represent unfamiliar words.
Different models can use different tokenization methods.
Unnecessary text cleaning can remove useful information.
Applications
Search engines
Language modeling
Machine translation
Text classification
Chatbots
Information extraction
Document processing
Learning Outcomes
Explain the purpose of text processing.
Define tokens and tokenization.
Differentiate word-level and subword tokenization.
Understand why modern language models use tokenizers.
Practice Questions
What is tokenization?
How does sentence tokenization differ from word tokenization?
Why is subword tokenization useful?
Can two NLP models tokenize the same sentence differently?
Related Concepts
Text Normalization
Vocabulary
Byte Pair Encoding
WordPiece
Language Modeling
Text Vectorization
Common Mistakes
Assuming every token represents one complete word.
Assuming all NLP models use the same tokenizer.
Removing punctuation without considering whether it carries meaning.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Jurafsky, D., and Martin, J. H., Speech and Language Processing, 3rd edition draft.
- Sennrich, R., Haddow, B., and Birch, A., Neural Machine Translation of Rare Words with Subword Units, ACL, 2016.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning