✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-128

Language Modeling

Language modeling is the task of learning probability distributions over sequences of tokens to estimate or generate plausible language.

P(w_1, w_2, ..., w_n) = ∏_(t=1)^n P(w_t | w_1, ..., w_(t-1))

Explanation

Language modeling is a central task in natural language processing. A language model learns statistical or neural patterns in language and uses them to estimate how likely a sequence of words or tokens is.

An autoregressive language model predicts the next token based on the tokens that came before it. It repeatedly generates a probability distribution over possible next tokens and selects one according to its decoding strategy.

For example, given the context 'The sun rises in the', a language model may assign relatively high probability to the token 'east'. It does not simply memorize a fixed sentence; it uses learned patterns from training data to estimate likely continuations.

A language model can be trained using different approaches:

1. N-gram models: Estimate token probabilities using a limited number of preceding tokens.
2. Recurrent language models: Use recurrent hidden states to represent previous sequence information.
3. Transformer language models: Use attention mechanisms to process contextual relationships between tokens.
4. Masked language models: Predict tokens that have been hidden within an input sequence, using surrounding context.

Autoregressive and masked language modeling have different training objectives. Autoregressive models generally predict the next token from preceding context, while masked language models predict selected masked tokens using context from both directions.

Language models are evaluated using measures such as perplexity for suitable probabilistic prediction tasks, as well as task-specific metrics and human evaluation for generated text. A lower perplexity generally indicates better predictive performance on the evaluated data, but it does not guarantee factual accuracy, reasoning ability, or overall usefulness.

Modern large language models are trained on large datasets and may use additional stages such as instruction tuning and preference optimization. Their generated responses can still contain incorrect or unsupported information.

Variables

SymbolMeaning

P

Probability assigned to a token sequence

w_t

Token at position t

n

Number of tokens in the sequence

P(w_t | w_1, ..., w_(t-1))

Conditional probability of the current token given preceding tokens

Example or application

A predictive keyboard uses a language model to estimate which word or token is likely to follow a partially typed sentence and suggests possible continuations.

Text

A language model assigns probabilities to token sequences or predicts the next token based on preceding context. It is a foundation of applications such as text generation, translation, and speech recognition.

Key Points

  • Language models learn probability distributions over token sequences.

  • Autoregressive models predict the next token from previous context.

  • N-gram, recurrent, and Transformer models are different language-modeling approaches.

  • Masked language models use surrounding context to predict hidden tokens.

  • Perplexity is a common measure for probabilistic language-model evaluation.

  • Good predictive performance does not guarantee factual correctness.

Applications

  • Predictive text

  • Text generation

  • Machine translation

  • Speech recognition

  • Question answering

  • Text summarization

  • Conversational assistants

Learning Outcomes

  • Define language modeling.

  • Explain next-token prediction.

  • Differentiate autoregressive and masked language modeling.

  • Understand the purpose and limitations of perplexity.

Practice Questions

  • What does a language model learn?

  • How does autoregressive next-token prediction work?

  • How does masked language modeling differ from autoregressive modeling?

  • Does low perplexity guarantee factual accuracy?

Related Concepts

  • N-gram Models

  • Recurrent Neural Networks

  • Transformer Architecture

  • Word Embeddings

  • Perplexity

  • Large Language Models

Common Mistakes

  • Assuming every language model generates text.

  • Confusing masked language modeling with next-token prediction.

  • Treating perplexity as a complete measure of language quality.

  • Assuming fluent generated text is necessarily factually correct.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Jurafsky, D., and Martin, J. H., Speech and Language Processing, 3rd edition draft.
  • Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C., A Neural Probabilistic Language Model, Journal of Machine Learning Research, 2003.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning