Stemming and lemmatization are text preprocessing techniques used to reduce morphological variations of words. They help NLP systems treat related forms as connected when their grammatical differences are not important for a particular task.
Stemming generally applies simple rules to remove prefixes or suffixes. It does not necessarily produce a valid dictionary word. For example, a stemmer may reduce 'connected', 'connecting', and 'connection' to a shortened form, depending on the algorithm.
Lemmatization is more linguistically informed. It attempts to identify the dictionary base form of a word, known as its lemma. It may use vocabulary, morphological analysis, part-of-speech information, and language-specific rules.
Consider the following examples:
- 'am', 'is', and 'are' can be mapped to the lemma 'be'.
- 'running' can be mapped to 'run' when used as a verb.
- 'better' can be mapped to 'good' when it is used as the comparative form of the adjective.
The correct lemma may depend on grammatical context. For example, the word 'saw' can represent a form of the verb 'see' or a noun referring to a cutting tool. A lemmatizer may need part-of-speech information to choose the appropriate interpretation.
Stemming is often faster and simpler, making it useful for large-scale information retrieval. Lemmatization is generally more linguistically accurate but may require more computational resources and language-specific tools.
Neither technique is appropriate for every NLP task. In some modern language-modeling pipelines, the original word forms are retained because tokenizers and neural models can learn useful morphological patterns directly.
Stemming and Lemmatization
Stemming and lemmatization are NLP techniques that reduce related word forms to simpler representations to support text analysis and information retrieval.
Explanation
Example or application
In a search engine, a query containing 'running shoes' may be matched with a document containing 'run' or 'runs' after applying a suitable stemming or lemmatization process.
Text
Stemming removes word endings or applies heuristic rules to produce a stem, while lemmatization uses linguistic information to identify a word's dictionary base form, called its lemma.
Key Points
Both techniques reduce morphological variation.
Stemming usually uses rule-based word reduction.
A stem may not be a valid dictionary word.
Lemmatization aims to produce a meaningful dictionary base form.
Lemmatization may use grammatical context and part-of-speech information.
The choice depends on the task, language, and accuracy requirements.
Applications
Search engines
Information retrieval
Document classification
Text clustering
Keyword extraction
Text analytics
Learning Outcomes
Define stemming and lemmatization.
Explain the difference between a stem and a lemma.
Identify cases where grammatical context matters.
Choose an appropriate technique for a basic NLP task.
Practice Questions
What is stemming?
How does lemmatization differ from stemming?
Why might a stemmer produce a non-dictionary word?
Why can part-of-speech information improve lemmatization?
Related Concepts
Text Normalization
Morphology
Tokenization
Part-of-Speech Tagging
Information Retrieval
Common Mistakes
Assuming stemming always produces valid dictionary words.
Treating stemming and lemmatization as identical methods.
Ignoring the grammatical role of a word during lemmatization.
Applying stemming automatically to every modern NLP task.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Porter, M. F., An Algorithm for Suffix Stripping, Program, 1980.
- Jurafsky, D., and Martin, J. H., Speech and Language Processing, 3rd edition draft.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning