✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-127

Word Embeddings

Word embeddings are dense numerical vectors that represent words in a continuous space where learned patterns can reflect semantic and syntactic relationships.

cos(θ) = (A · B) / (||A|| ||B||)

Explanation

Word embeddings are numerical representations of words or tokens in which each item is mapped to a dense vector of real-valued numbers. These vectors are learned from data and can capture useful patterns about how words are used in language.

Traditional representations such as one-hot encoding and Bag-of-Words treat vocabulary items as separate features. They do not naturally express that words such as 'king' and 'queen' may have related meanings. Word embeddings provide a continuous representation in which related words can have similar vector patterns.

Popular embedding methods include:

1. Word2Vec: Learns word representations using predictive objectives such as Continuous Bag-of-Words (CBOW) and Skip-gram.
2. GloVe: Learns word vectors using global word co-occurrence statistics.
3. FastText: Represents words using character n-gram information, helping it handle morphological variations and some rare words.

Embeddings can be evaluated using measures such as cosine similarity. A higher cosine similarity generally indicates that two vectors point in more similar directions, although this does not guarantee that the words are interchangeable in every context.

Traditional static embeddings assign one vector to a word regardless of its context. For example, the word 'bank' may have the same vector in 'river bank' and 'bank account'. Contextual language models such as BERT generate representations that depend on surrounding words, helping distinguish different meanings.

Word embeddings are widely used as inputs to NLP models and as tools for studying semantic relationships. However, they can also inherit social or cultural biases present in their training data.

Variables

SymbolMeaning

A

First word embedding vector

B

Second word embedding vector

A · B

Dot product of the two vectors

||A||

Magnitude of vector A

||B||

Magnitude of vector B

θ

Angle between the vectors

Example or application

A search system can use word embeddings to recognize that a query containing 'car' may be related to documents containing 'automobile', even though the two words are written differently.

Text

Word embeddings map words or tokens to dense vectors. Unlike simple word-count representations, embeddings can encode similarities and relationships learned from language data.

Key Points

  • Embeddings represent words as dense numerical vectors.

  • Word2Vec uses predictive learning objectives.

  • GloVe uses global co-occurrence statistics.

  • FastText uses character n-gram information.

  • Cosine similarity can compare embedding directions.

  • Static embeddings assign one vector per word.

  • Contextual embeddings depend on surrounding language.

Applications

  • Semantic search

  • Text classification

  • Machine translation

  • Question answering

  • Recommendation systems

  • Information retrieval

  • Document similarity

Learning Outcomes

  • Explain the purpose of word embeddings.

  • Differentiate dense embeddings from one-hot vectors.

  • Identify major static embedding methods.

  • Understand the difference between static and contextual representations.

Practice Questions

  • What is a word embedding?

  • How does an embedding differ from a Bag-of-Words representation?

  • What is the difference between Word2Vec and GloVe?

  • Why can contextual embeddings represent different meanings of the same word?

Related Concepts

  • Word2Vec

  • GloVe

  • FastText

  • Cosine Similarity

  • Language Modeling

  • Transformer Architecture

Common Mistakes

  • Assuming similar vectors always mean two words are interchangeable.

  • Confusing static embeddings with contextual embeddings.

  • Assuming embedding dimensions have universally fixed human-readable meanings.

  • Ignoring bias inherited from training data.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Mikolov, T. et al., Efficient Estimation of Word Representations in Vector Space, 2013.
  • Pennington, J., Socher, R., and Manning, C. D., GloVe: Global Vectors for Word Representation, EMNLP, 2014.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning