Large Language Models are a major development in natural language processing. They are trained using large datasets and deep neural network architectures, especially Transformer-based architectures. Many modern LLMs use self-attention mechanisms to model relationships between tokens across a sequence. During pretraining, an autoregressive language model commonly learns to predict the next token from previous tokens. Other training objectives, including masked-token prediction, are also used in some model families. After pretraining, models may undergo supervised fine-tuning, instruction tuning, and preference-based optimization to improve their ability to follow instructions and produce useful responses. LLMs operate on tokens rather than directly processing words as humans see them. Their internal representations encode statistical and semantic relationships that support many language tasks. Despite their capabilities, LLMs do not guarantee factual correctness, complete understanding, or reliable reasoning. Their responses can contain hallucinations, outdated information, and biases. Performance also depends on model architecture, training data, context length, and inference settings.
Large Language Models (LLMs)
Large Language Models (LLMs) are neural networks trained on large collections of text and other data to understand and generate language. Modern LLMs commonly use Transformer architectures and predict sequences of tokens.
P(x_1, x_2, ..., x_n) = ∏_{t=1}^{n} P(x_t | x_1, ..., x_{t-1})
Explanation
Variables
| Symbol | Meaning |
|---|---|
x_t | Token at sequence position t |
n | Total number of tokens in the sequence |
P(x_t | x_1, ..., x_{t-1}) | Probability of the current token given preceding tokens |
Example or application
When a user asks an LLM to summarize a scientific article, the model processes the input tokens and generates a shorter sequence that attempts to preserve the article's main ideas.
Text
An LLM is a machine learning model with a large number of parameters that learns statistical relationships between language units. It can perform tasks such as answering questions, summarizing documents, translating text, and generating code.
Key Points
LLMs are trained on large datasets using neural networks.
Transformer architectures are widely used in modern LLMs.
Token prediction is a common language-modeling objective.
LLMs can perform many tasks through instructions and context.
Their outputs require verification when accuracy is important.
Applications
Conversational assistants
Document summarization
Language translation
Code generation
Question answering
Information extraction
Writing assistance
Learning Outcomes
Define a Large Language Model.
Explain the role of tokens and model parameters.
Describe the basic next-token prediction process.
Identify common applications and limitations of LLMs.
Practice Questions
What makes a language model large?
Why do LLMs use tokenization?
What is next-token prediction?
Why can an LLM produce incorrect information?
Related Concepts
Natural Language Processing
Transformers
Self-Attention
Tokenization
Pretraining
Fine-Tuning
Common Mistakes
Assuming that every LLM uses an identical architecture.
Treating tokens and words as exactly the same thing.
Assuming that fluent responses are necessarily correct.
Believing that an LLM stores and retrieves every training document exactly.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Vaswani, A. et al. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems.
- Brown, T. B. et al. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems.
- Jurafsky, D., and Martin, J. H. Speech and Language Processing. Draft edition.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning