An evasion attack attempts to mislead a trained model during inference by providing specially crafted inputs. A data-poisoning attack introduces harmful or misleading examples into training data to influence the learned model. Model extraction attempts to reproduce a model by observing its responses to queries. Membership-inference attacks attempt to determine whether a particular record was included in a model's training dataset. These threats affect different stages of the machine learning lifecycle and require different defensive approaches. Security testing should consider realistic attacker capabilities and the consequences of successful attacks. Defenses may include validating training data, limiting query access, monitoring unusual activity, applying adversarial training where appropriate, protecting model files, and testing against known threat scenarios. No single defense prevents every attack, and security must be reviewed as systems and threats evolve.
AI Security and Adversarial Attacks
AI security protects machine learning systems against threats that manipulate inputs, training data, models, or infrastructure.
Explanation
Example or application
An image-classification system is designed to recognize road signs. During security testing, engineers evaluate whether small, carefully designed input changes can cause incorrect classifications and then improve the system's robustness and monitoring.
Text
AI systems can be attacked through weaknesses in their data, algorithms, software, or deployment environments. Adversarial attacks deliberately create inputs or conditions that cause a model to make incorrect predictions or behave unexpectedly. Common threat categories include evasion attacks, data poisoning, model extraction, and privacy-related attacks. Defending AI systems requires secure data pipelines, robust model evaluation, access controls, monitoring, and careful system design.
Key Points
AI systems can be attacked during training and inference.
Evasion attacks target model inputs at prediction time.
Data poisoning targets the training process.
Model extraction attempts to reproduce a model through its outputs.
Membership inference investigates whether data was used during training.
Security requires layered defenses and continuous testing.
Applications
Securing autonomous-system perception
Protecting AI-powered financial services
Defending machine learning APIs
Securing training-data pipelines
Testing the robustness of computer vision models
Learning Outcomes
Define adversarial attacks in machine learning.
Distinguish evasion attacks from data poisoning.
Recognize model extraction and membership inference.
Explain why AI security requires multiple defensive layers.
Practice Questions
What is an adversarial attack?
How does an evasion attack differ from a poisoning attack?
What is model extraction?
Why is continuous security testing necessary?
Related Concepts
Adversarial Machine Learning
AI Safety
Cybersecurity
Robustness Testing
Data Integrity
Common Mistakes
Assuming that a model with high test accuracy is secure.
Treating all adversarial attacks as input-manipulation attacks.
Ignoring threats to training data and model infrastructure.
Assuming that one defensive technique prevents every attack.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Biggio, B. and Roli, F., Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning, Pattern Recognition, 2018.
- Goodfellow, I., Shlens, J., and Szegedy, C., Explaining and Harnessing Adversarial Examples, ICLR, 2015.
- NIST, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, 2024.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning