Security fundamentals

Before we embark on securing our adversarial AI playground, let’s cover some foundational concepts and approaches in security.

Security evolves around the triad known as CIA and can be described as follows:

• Confidentiality: Protecting data from unauthorized access. For instance, restricting access and encrypting sensitive data ensures that only those possessing the correct decryption key can access it.
• Integrity: Guaranteeing that data remains unchanged during storage or transmission, except by authorized entities. Implementing cryptographic hash functions is one method to verify the integrity of data.
• Availability: Assuring that systems, data, and resources remain accessible to users when required. Load balancing and redundancy are techniques that are often used to uphold system availability during demand surges or system failures.

Frameworks such as the NIST Cybersecurity Framework offer a standardized approach to handling cybersecurity risks. Comprising five core functions – Identify, Protect, Detect, Respond, and Recover – this framework guides organizations through the entire life cycle of cybersecurity risk management. Other frameworks include the ISACA Control Objectives for Information and Related Technologies (COBIT), CIS Critical Security Controls, System and Organization Controls 2 (SOC2), Federal Risk and Authorization Management Program (FedRAMP), and Payment Card Industry Data Security Standard (PCI DSS). Organizations adopt these frameworks for compliance and adapt to their own needs to apply the principles of the CIA triad more effectively. These frameworks are enterprise-wide and outside the scope of this book. However, they help us understand the context of how security works in a given organization.

Threat modeling

Threat modeling is a structured approach that’s used to identify, prioritize, and manage potential threats in a system. It is often a collaborative exercise that starts with identifying critical assets, processes, and the data flows that are used to identify potential threats. A pivotal exercise element is agreeing on trust boundaries demarcating areas of trust and concern.

There are many approaches to identifying threats, but two popular ones stand out:

• STRIDE: This stands for Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, and Elevation of Privilege. This approach identifies threats that are relevant to each of its categories.
• Attack trees: These trees visualize paths an attacker might take to compromise a system. Each node in the tree represents a specific action or event that contributes to the overarching goal of the attacker. The MITRE Adversarial Tactics, Techniques, and Common Knowledge (ATT&CK) framework provides a comprehensive standardized vocabulary and various techniques to capture threats with attack trees. MITRE ATLAS is a more recent MITRE attack framework dedicated to AI. We will cover it in more detail later in Chapter 15 when we discuss MLSecOps.

You can find more information on threat modeling and the MITRE ATT&CK framework at https://insights.sei.cmu.edu/blog/threat-modeling-12-available-methods/ and https://attack.mitre.org/, respectively.

Risks and mitigations

Once we have identified threats, we assign risk. We usually estimate risk based on the likelihood of it happening and the impact of successful exploitation. This helps us prioritize threats and mitigations. Mitigations are how we defend against these threats; we usually discuss security controls.

Industry-standard security controls provide a repertoire of defenses we can deploy, test, and verify. The CIS Benchmarks are a widely accepted set of standard controls for platform and infrastructure. The OWASP Top 10 and the more detailed OWASP Application Security Verification Standard (ASVS) are application-specific standards.

We use security testing, especially penetration tests (or pen tests) with external testers, ensuring that these controls have been implemented. Pen tests involve simulating cyberattacks to identify vulnerabilities before going live.

DevSecOps

However, security testing and pen testing can sometimes come too late in the development cycle. This is where DevSecOps enters. By integrating security within the DevOps process, we shift left security, introducing security cycle checks within the system life cycle much earlier. Tools such as continuous integration (CI), static application security testing (SAST), dynamic application security testing (DAST), and vulnerability scanning are employed to detect vulnerabilities in the code base or the running application.

DevSecOps deals with traditional application artifacts, such as code, libraries, packages, containers, environments, and applications. In the context of machine learning (ML), the introduction of MLOps has elevated models and data to first-class citizens.

Note

MLOps is built on DevOps and adds capabilities for models, data, experiment tracking, and governance. This is important because, unlike other applications, AI development and ML depend on live data, which can often be sensitive. This dependency makes it crucial to apply security controls as early as possible, including development environments accessing sensitive data for model training and testing.

With that, we’ve provided a broad overview of cybersecurity and how it is being applied. The depth and breadth of its application depend on the project and organization. In the next section, we will apply security concepts in practice and add traditional security to our adversarial AI playground. We will also demonstrate the limitations of conventional security defenses when it comes to adversarial AI.