✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-145

Embeddings and Vector Representations

Embeddings are dense numerical vectors that represent tokens, words, sentences, images, or other data in a mathematical space. They allow machine learning models to represent and compare patterns and relationships between information.

cos(θ) = (A · B) / (||A|| ||B||)

Explanation

Embeddings are learned or constructed numerical representations used throughout modern machine learning. In language models, token IDs are mapped to dense vectors that become inputs to neural network layers. These vectors encode useful statistical and contextual patterns learned during training. Some embedding systems produce static representations, where a word has the same vector in different contexts. Contextual embedding systems produce representations that depend on surrounding tokens, allowing the same word to have different representations in different sentences. Embeddings can also represent entire sentences, documents, images, audio segments, and other modalities. Similarity between vectors can be measured using cosine similarity, dot product, or distance metrics such as Euclidean distance. In information retrieval, a query and a document can be embedded into a shared vector space so that semantically related content can be retrieved. Embeddings are central to semantic search, recommendation systems, clustering, and retrieval-augmented generation. However, vector similarity is not proof that two items are factually equivalent. Embeddings may also reflect biases and limitations in their training data.

Variables

SymbolMeaning

A

First embedding vector

B

Second embedding vector

A · B

Dot product of the two vectors

||A||

Magnitude of vector A

||B||

Magnitude of vector B

θ

Angle between the vectors

Example or application

A semantic search system converts the query 'How do plants make food?' and relevant document passages into vectors. It can retrieve passages about photosynthesis even when they do not contain the exact same words.

Text

An embedding converts an item into a list of numerical values. Items with similar learned characteristics may have vectors that are close to one another according to a chosen similarity or distance measure.

Key Points

  • Embeddings represent data as numerical vectors.

  • Token embeddings are important inputs to language models.

  • Contextual embeddings depend on surrounding information.

  • Cosine similarity is commonly used to compare vector directions.

  • Embeddings support semantic search and retrieval systems.

Applications

  • Semantic search

  • Document retrieval

  • Recommendation systems

  • Text classification

  • Clustering

  • Image-text matching

  • Retrieval-Augmented Generation

Learning Outcomes

  • Define an embedding.

  • Explain how token IDs become vector representations.

  • Understand the difference between static and contextual embeddings.

  • Describe how vector similarity supports semantic retrieval.

Practice Questions

  • What is an embedding?

  • Why do language models use vector representations?

  • What is the difference between static and contextual embeddings?

  • How does cosine similarity compare two vectors?

Related Concepts

  • Vector Space

  • Tokenization

  • Neural Networks

  • Cosine Similarity

  • Semantic Search

  • Vector Databases

Common Mistakes

  • Confusing token IDs with embedding vectors.

  • Assuming similar vectors always represent identical meanings.

  • Assuming every embedding model produces contextual vectors.

  • Ignoring the effect of embedding model choice on retrieval quality.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Mikolov, T. et al. (2013). Efficient Estimation of Word Representations in Vector Space.
  • Devlin, J. et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT.
  • Reimers, N., and Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP-IJCNLP.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning