✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-176

AI Security and Adversarial Attacks

AI security protects machine learning systems against threats that manipulate inputs, training data, models, or infrastructure.

Explanation

An evasion attack attempts to mislead a trained model during inference by providing specially crafted inputs. A data-poisoning attack introduces harmful or misleading examples into training data to influence the learned model. Model extraction attempts to reproduce a model by observing its responses to queries. Membership-inference attacks attempt to determine whether a particular record was included in a model's training dataset. These threats affect different stages of the machine learning lifecycle and require different defensive approaches. Security testing should consider realistic attacker capabilities and the consequences of successful attacks. Defenses may include validating training data, limiting query access, monitoring unusual activity, applying adversarial training where appropriate, protecting model files, and testing against known threat scenarios. No single defense prevents every attack, and security must be reviewed as systems and threats evolve.

Example or application

An image-classification system is designed to recognize road signs. During security testing, engineers evaluate whether small, carefully designed input changes can cause incorrect classifications and then improve the system's robustness and monitoring.

Text

AI systems can be attacked through weaknesses in their data, algorithms, software, or deployment environments. Adversarial attacks deliberately create inputs or conditions that cause a model to make incorrect predictions or behave unexpectedly. Common threat categories include evasion attacks, data poisoning, model extraction, and privacy-related attacks. Defending AI systems requires secure data pipelines, robust model evaluation, access controls, monitoring, and careful system design.

Key Points

  • AI systems can be attacked during training and inference.

  • Evasion attacks target model inputs at prediction time.

  • Data poisoning targets the training process.

  • Model extraction attempts to reproduce a model through its outputs.

  • Membership inference investigates whether data was used during training.

  • Security requires layered defenses and continuous testing.

Applications

  • Securing autonomous-system perception

  • Protecting AI-powered financial services

  • Defending machine learning APIs

  • Securing training-data pipelines

  • Testing the robustness of computer vision models

Learning Outcomes

  • Define adversarial attacks in machine learning.

  • Distinguish evasion attacks from data poisoning.

  • Recognize model extraction and membership inference.

  • Explain why AI security requires multiple defensive layers.

Practice Questions

  • What is an adversarial attack?

  • How does an evasion attack differ from a poisoning attack?

  • What is model extraction?

  • Why is continuous security testing necessary?

Related Concepts

  • Adversarial Machine Learning

  • AI Safety

  • Cybersecurity

  • Robustness Testing

  • Data Integrity

Common Mistakes

  • Assuming that a model with high test accuracy is secure.

  • Treating all adversarial attacks as input-manipulation attacks.

  • Ignoring threats to training data and model infrastructure.

  • Assuming that one defensive technique prevents every attack.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Biggio, B. and Roli, F., Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning, Pattern Recognition, 2018.
  • Goodfellow, I., Shlens, J., and Szegedy, C., Explaining and Harnessing Adversarial Examples, ICLR, 2015.
  • NIST, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, 2024.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning