✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-143

Large Language Models (LLMs)

Large Language Models (LLMs) are neural networks trained on large collections of text and other data to understand and generate language. Modern LLMs commonly use Transformer architectures and predict sequences of tokens.

P(x_1, x_2, ..., x_n) = ∏_{t=1}^{n} P(x_t | x_1, ..., x_{t-1})

Explanation

Large Language Models are a major development in natural language processing. They are trained using large datasets and deep neural network architectures, especially Transformer-based architectures. Many modern LLMs use self-attention mechanisms to model relationships between tokens across a sequence. During pretraining, an autoregressive language model commonly learns to predict the next token from previous tokens. Other training objectives, including masked-token prediction, are also used in some model families. After pretraining, models may undergo supervised fine-tuning, instruction tuning, and preference-based optimization to improve their ability to follow instructions and produce useful responses. LLMs operate on tokens rather than directly processing words as humans see them. Their internal representations encode statistical and semantic relationships that support many language tasks. Despite their capabilities, LLMs do not guarantee factual correctness, complete understanding, or reliable reasoning. Their responses can contain hallucinations, outdated information, and biases. Performance also depends on model architecture, training data, context length, and inference settings.

Variables

SymbolMeaning

x_t

Token at sequence position t

n

Total number of tokens in the sequence

P(x_t | x_1, ..., x_{t-1})

Probability of the current token given preceding tokens

Example or application

When a user asks an LLM to summarize a scientific article, the model processes the input tokens and generates a shorter sequence that attempts to preserve the article's main ideas.

Text

An LLM is a machine learning model with a large number of parameters that learns statistical relationships between language units. It can perform tasks such as answering questions, summarizing documents, translating text, and generating code.

Key Points

  • LLMs are trained on large datasets using neural networks.

  • Transformer architectures are widely used in modern LLMs.

  • Token prediction is a common language-modeling objective.

  • LLMs can perform many tasks through instructions and context.

  • Their outputs require verification when accuracy is important.

Applications

  • Conversational assistants

  • Document summarization

  • Language translation

  • Code generation

  • Question answering

  • Information extraction

  • Writing assistance

Learning Outcomes

  • Define a Large Language Model.

  • Explain the role of tokens and model parameters.

  • Describe the basic next-token prediction process.

  • Identify common applications and limitations of LLMs.

Practice Questions

  • What makes a language model large?

  • Why do LLMs use tokenization?

  • What is next-token prediction?

  • Why can an LLM produce incorrect information?

Related Concepts

  • Natural Language Processing

  • Transformers

  • Self-Attention

  • Tokenization

  • Pretraining

  • Fine-Tuning

Common Mistakes

  • Assuming that every LLM uses an identical architecture.

  • Treating tokens and words as exactly the same thing.

  • Assuming that fluent responses are necessarily correct.

  • Believing that an LLM stores and retrieves every training document exactly.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Vaswani, A. et al. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems.
  • Brown, T. B. et al. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems.
  • Jurafsky, D., and Martin, J. H. Speech and Language Processing. Draft edition.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning