✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-126

TF-IDF and Text Vectorization

TF-IDF is a statistical weighting method that represents text using numerical values based on term frequency and inverse document frequency.

TF-IDF(t,d) = TF(t,d) × IDF(t)
IDF(t) = log(N / DF(t))

Explanation

Text vectorization is the process of converting text into numerical representations that machine learning algorithms can use. TF-IDF, short for Term Frequency–Inverse Document Frequency, is a widely used statistical method for assigning importance weights to terms.

TF-IDF combines two ideas:

1. Term Frequency (TF): Measures how frequently a term appears in a particular document. Different implementations may use raw counts, normalized counts, or logarithmic scaling.
2. Inverse Document Frequency (IDF): Measures how rare or informative a term is across the document collection. Terms appearing in many documents receive lower IDF values.

A common formulation is:

TF-IDF(t,d) = TF(t,d) × IDF(t)

One frequently used IDF definition is:

IDF(t) = log(N / DF(t))

Here, N is the total number of documents and DF(t) is the number of documents containing term t. Practical libraries may use smoothing or alternative normalization methods.

A term that appears many times in one document but rarely across the collection can receive a relatively high TF-IDF weight. A very common term may receive a lower weight because it provides less distinguishing information.

TF-IDF vectors are commonly sparse because most documents contain only a small portion of the complete vocabulary. They can be used with traditional machine learning algorithms and similarity measures such as cosine similarity.

TF-IDF is useful for document search and classification, but it does not directly capture word meaning, word order, or contextual relationships. Modern neural embeddings provide alternative representations that can capture richer semantic information.

Variables

SymbolMeaning

TF(t,d)

Term frequency of term t in document d

IDF(t)

Inverse document frequency of term t

N

Total number of documents in the collection

DF(t)

Number of documents containing term t

t

Term or token

d

Document

Example or application

In a collection of news articles, the word 'the' appears in almost every document and usually receives a low IDF value. A specialized term such as 'photosynthesis' may receive a higher IDF value if it appears in relatively few articles.

Text

TF-IDF gives higher importance to terms that occur frequently in a particular document but appear in relatively few documents across a collection. Text vectorization converts textual information into numerical features.

Key Points

  • TF measures term frequency within a document.

  • IDF reduces the weight of terms common across many documents.

  • TF-IDF combines local frequency and collection-wide rarity.

  • Different implementations may use different TF and IDF variations.

  • TF-IDF usually produces sparse numerical vectors.

  • TF-IDF does not directly understand semantic similarity or word order.

Applications

  • Document retrieval

  • Search engines

  • Keyword extraction

  • Document classification

  • Text similarity

  • Information retrieval

Learning Outcomes

  • Explain term frequency and inverse document frequency.

  • Calculate a basic TF-IDF value.

  • Understand why common terms receive lower IDF weights.

  • Compare TF-IDF with Bag-of-Words and word embeddings.

Practice Questions

  • What does TF measure?

  • Why does IDF reduce the weight of common terms?

  • What happens to IDF when a term appears in every document under the basic formula?

  • How does TF-IDF differ from a semantic word embedding?

Related Concepts

  • Bag-of-Words

  • Text Vectorization

  • Cosine Similarity

  • Information Retrieval

  • Word Embeddings

Common Mistakes

  • Assuming TF-IDF directly measures semantic meaning.

  • Confusing document frequency with total term frequency.

  • Assuming every software library uses the same IDF formula.

  • Ignoring smoothing and normalization choices in practical implementations.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Salton, G., and Buckley, C., Term-Weighting Approaches in Automatic Text Retrieval, Information Processing and Management, 1988.
  • Manning, C. D., Raghavan, P., and Schütze, H., Introduction to Information Retrieval, Cambridge University Press, 2008.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning