Secure AI System Development

Scientific progress in AI and downstream innovation to solve concrete real-world problems is part of a greater movement toward inventing Artificial General Intelligence (AGI). Broadly speaking, AGI is defined as an intelligent agent that can emulate and surpass human intelligence.

Today, we are already familiar with incomplete forms of AGI:

• Large Language Models (LLMs) including ChatGPT and other generative AIs.
• Partly autonomous vehicles (up to Level 3 for public consumption) such as Tesla.

Despite these promising innovations moving from the scientific domain to consumer marketplaces, we are still far from achieving AGI.

While we make our way to developing highly intelligent systems, a primary consideration is the safety and security of such a system. That’s a big umbrella of topics including:

• How any AI interacts with users
• Its robustness against a variety of threat vectors
• Our dependence on intelligent machines to solve critical tasks

In November 2023, CISA in the US and UK NCSC released joint guidelines for the development of secure AI systems. Let’s take a look at what these guidelines mean, and why two governments joined together for this effort.

Key characteristics of AI systems

Already, our exposure to AI-related security risks has motivated us to embed cybersecurity as a necessary precondition for a variety of factors:

• Safety: against threats from external adversaries as well as malicious and uncontrollable threats from intelligent systems.
• Resilience: continued functionality and high dependability, especially in critical situations.
• Privacy: protection against exposure of sensitive information or ability to access information without authority.
• Fairness: to remove bias against specific groups and attributes.
• Efficacy: enable cost-effective and efficient performance that enables scalable usage and global adoption.
• Reliability: to deliver accurate, true and optimal solutions.

These are the key characteristics that any secure AI system should demonstrate.

(Related reading: AI-augmented software engineering, the AI bill of rights & AI TRiSM: trust, risk and security management.)

AI systems + threats = security risk

In order to develop such a system, we need to understand how the common AI systems operate and the type of threats that can risk a security exposure.

The main security risk comes from the underlying mechanism used to develop AI systems. AI tools are based on mechanisms such as deep neural networks that are trained and tuned on data — instead of specifying the mathematical formulation of its parameters exactly. (Using math is not possible, anyhow, given the vast number of parameters involved in an AI system.)

For example, the AI systems underlying ChatGPT have billions of parameters, which must be updated and tuned every time the system has to incorporate new and learn from information as input. The exact understanding of how these parameters values evolve and update are too complex — and therefore practically unexplainable.

This so-called black-box characteristic of the AI system exposes it to security risks such as training the AI models with adversarial examples, manipulated data and incorrect information to generate output that violates the security preconditions and characteristics outlined earlier.

(Understand adaptive AI and multimodal AI, both in early stages of the AI hype cycles.)

Secure AI development: guidelines

So, what are the key considerations and guidelines for secure AI development? When developing AI systems to solve critical problems, the following guidelines relevant to AI capabilities should be considered:

Adversarial robustness

attacks that guide the training process such that the models generate incorrect predictions from a perceived correct input.

Bias & fairness

since the AI algorithms can guide the training process in favor of biased and unfair outcomes, including fairness related to acceptable societal standards. Fairness may not be an inherent trait of AI systems and may be modeled explicitly within the model training algorithms and guided by the variety of input data used to train the AI models.

Model explainability & interpretability

The explainability and interpretability means the AI’s ability to validate model outcomes. This validation enhances both our trust and our transparency into the decision-making process that is driven by an AI model output.

(Put the other way: if we cannot trust the outputs and outcomes of an AI, what value does the AI deliver?)

Discover how to use explainability with SDLC

Transfer learning

Learning from pre-trained models may be essential as the training process of billion-parameter models is time consuming and resource-intensive. Embedding security into the transfer learning process means that malicious patterns from previous training should be identified and removed when using pre-trained for down-stream tasks.

Privacy-preserving AI

Following data privacy regulations such as GDPR and prompting user consent to train AI models is necessary. What’s different for developing secure AI systems, however, is that AI mechanisms such as generative AI should not be enabled to generate person-specific information that violates the established privacy and security rights of the subject.

Common strategies to develop privacy-preserving AI include:

• Homomorphic encryption, which allows for training on encrypted information.
• Federated learning, which decentralizes the model training process.

Indeed, federated learning allows users to train AI models using their personal data on their own devices, and then send the trained parameter update to a centralized AI system. This is significantly better than the alternative: sending over sensitive and personally identifiable information itself.

AI governance

Finally, any AI governance framework you use or establish must adopted as part of it three key pieces:

• Ethical standards
• Accountability
• Compliance to established data security and privacy regulations

This can be a guideline playbook that is unique to every organization based on the applicable security risks and available best practices that can be adopted efficiently by any user of the AI system.

The responsibility is with developers and users

Lastly, it is the responsibility of developers and business organizations to report any security flows and limitations of the AI systems.

Again, the first step to developing secure AI systems is to acknowledge and adopt the associated ethical responsibilities and guidelines, especially when AI models, mechanisms and processes can only demonstrate limited accountability and interpretability.