Addressing Bias in AI Systems: A Holistic Approach to Detection, Reporting, and Mitigation

Summary

This blog post summarizes insights from a Future of Privacy Forum workshop on Privacy Preserving Machine Learning (PPML) held on May 25, 2023, where experts from Holistic AI presented on performing bias assessments in AI systems. The post explores practical approaches to identifying and addressing unfair treatment in machine learning applications used in recruitment, judicial systems, and other critical domains.

The content covers five key questions for bias assessment, a three-phase cyclical framework (assessment, reporting, and mitigation), and the challenges posed by data collection regulations like GDPR and NYC Local Law 144. Special attention is given to intersectionality considerations and the trade-offs between performance, privacy, and fairness in AI system development.

On 25 May 2023, the Future of Privacy Forum organized a workshop on the state-of-play of Privacy Preserving Machine Learning (PPML). Lindsay Levine Carignan , Head of Customer Success at Holistic AI, and Nigel Kingsman , AI Audit and Assurance Officer at Holistic AI, took the floor with a presentation on the question: how to perform a bias assessment? This post summarizes what I learned from the presenters while preparing and moderating the workshop. The presenters were tasked to explain a complex technical topic to a non-technical audience. In passing, I will elaborate on some of the key concepts presented.

What is Bias?

Bias, as defined in this write-up, is looking at the risk that the system treats individuals or groups unfairly. For example, when using machine learning in applications like recruitment or the judicial system. In these cases, it is especially important to ensure that algorithms do not discriminate and, instead, treat everyone equally. Artificial Intelligence (AI) systems can exhibit bias due to, e.g., inappropriate training data, a poor training pipeline, or a discriminatory interface.

Key Questions to Ask a Data Scientist about Potential Bias

To better understand potential bias, Carignan suggested learning about algorithmic aspects, e.g., (1) data provenance (origin), (2) representativeness (does it reflect the population), and (3) balance (are score categories representative). Also, it is essential to know (4) whether intersectionality was considered and (5) whether sensitive attributes were present in the data at all. These aspects help people without a technical background in machine learning to initiate a conversation with data scientists about algorithms. We arrived at the following five example questions to start a discussion on bias:

• What is the provenance of data?
• How representative is the data?
• Is the data balanced?
• Has intersectionality been considered?
• Are sensitive attributes present in the data?

Below, we will explore how to perform a bias assessment with these example questions in mind.

Performing a Bias Assessment

Kingsman offered an insightful view of addressing bias in AI systems and the trade-offs involved in this process. He argued that tackling bias is not only essential but also challenging and multifaceted, as it can potentially affect system performance, privacy, and even other bias measures. He explored the various strategies employed by Holistic AI in addressing biases in AI systems. The procedure was divided into three primary phases: (1) assessment, (2) reporting, and (3) mitigation. These phases are part of a cyclic process that aims to analyze and enhance AI systems' fairness continually, from its detection to reporting and mitigation. The holistic cycle ensures constant evaluation and enhancement of AI systems to reduce bias and improve fairness.

Phase 1: Assessment

In the assessment phase, Kingsman explained how he discovered potential biases by leveraging expert reviews and technical analysis. Here, experts meticulously study the system's function, application context, and stakeholders involved, while technical analysis includes simulating datasets to highlight biased behaviors in the AI system.

Kingsman stressed that, in some cases, direct access to the client's model is advantageous for testing across a more comprehensive dataset.

He delved into two main paradigms of bias: (1) equality of outcome and (2) equality of opportunity. The former looks at prediction outcomes, whereas the latter considers conditioning assessments based on some features, such as qualifications in a recruitment context.

More specifically, equality of outcome focuses on ensuring that the distribution of positive outcomes among different groups aligned with their respective application rates, irrespective of qualifications. On the other hand, equality of opportunity conditions bias assessment based on specific candidate attributes or qualifications.

Kingsman emphasized that reducing bias did not come without trade-offs. He noted that system performance or efficacy often had to be compromised to minimize bias. Furthermore, debiasing could potentially lead to increased privacy risks because of the need to collect sensitive attributes to measure and reduce bias. He also warned that efforts to improve one bias measure might worsen another due to the many different bias measures that exist.

He highlighted standard metrics such as Disparate Impact and Statistical Parity (for Equality of Outcome) and Equal Opportunity Difference and Average Odds Difference (for Equality of Opportunity). Let me remark on the difference between the two terms in each set of metrics. Disparate Impact identifies unintentional discrimination from neutral data processing, while Statistical Parity aims to equalize outcomes across demographic groups—however, neither guarantees complete fairness. Equal Opportunity Difference focuses on favorable outcomes (correct positive predictions), while Average Odds Difference considers both favorable and unfavorable outcomes (both correct and incorrect predictions).

Phase 2: Reporting

When discussing the reporting phase, Kingsman introduced two risk concepts: inherent and residual risks. The inherent risk represented risks the system might have without any technical mitigations or robust procedures, while residual risks were those that remained after implementing mitigations and procedures. An important goal was to lower each identified risk as much as possible by implementing appropriate procedures and technical mitigations.

Phase 3: Mitigation

Kingsman addressed mitigation strategies, focusing on how they can modify the machine learning pipeline in three phases: (a) pre-processing, (b) in-processing, and (c) post-processing. Pre-processing, such as oversampling data, is useful when the dataset is already biased. In-processing involves changing the model-building algorithm, which could be expensive as it requires model retraining. Post-processing is usually the cheapest method, as it involves changes after the model is trained, but it might be less effective.

Training Pipeline and System Accessibility

Kingsman also highlighted the significance of employing a robust training pipeline. He mentioned the importance of collecting samples accurately, using appropriate targets during training, and employing methods to decrease bias along the way. Additionally, he emphasized the often overlooked aspect of system accessibility, where assumptions about users' resources or language proficiency could result in discrimination against specific individuals.

How Does Bias Manifest in AI Systems?

Lindsay Carignan's intervention focused on understanding bias in AI and the challenges posed by data collection methods and regulations. She (further) explored the limitations, challenges, and five key questions to ask when conducting a bias assessment.

Sensitive Data Collection

Carignan observed that sensitive attributes like race, gender, and age were often not readily available. This was due to restrictions like the General Data Protection Regulation (GDPR) or individuals' unwillingness to provide such data. When such data was collected, it usually represented a small subset of the total population, sometimes less than 1%.

Data Set Size and Intersectionality

Carignan explained that demographic data, like gender, was easier to collect, but intersectional attributes, e.g., a Black woman over 40, were more challenging due to reduced sample sizes. If you're looking at male-female, you start seeing a drop-off when it comes to ethnicity. For example, when using EEOC categories (classifications used in an Equal Employment Opportunity Compliance report of employees' race, gender, and job classifications). She also highlighted underrepresentation in data sets, e.g., data on Native Americans, Pacific Islanders, or Hawaiian often accounted for less than 2% of the total.

Limitations of Bias Assessments

Bias assessments often have a narrow focus, meaning they could overlook other crucial aspects of AI systems such as efficacy (how well the model performed), privacy (how well the model protected user data), and robustness (how well the model handled different conditions or adversities). Additionally, optimizing a model for one bias metric may have negatively affected others, demonstrating the complexity and trade-offs involved in minimizing bias.

Furthermore, Carignan pointed out that not collecting sensitive attribute data didn't necessarily remove bias since biased outcomes could still occur in their absence. She specifically highlighted the challenges posed by the availability of sensitive attributes and the impact of regulations such as (a) in the United States, the New York City Local Law 144 (NYC Local Law 144) as well as proposed legislation regarding automated employment decision tools in places like California, New Jersey, New York, and (b) (proposed) legislation regarding automated decisions in Europe under the GDPR and the proposed AI Act. In these cases, it's especially important to ensure that algorithms do not discriminate and instead treat everyone equally.

The NYC Local Law 144 was mentioned as an example that affects the data sets obtained from vendors. She explained that, under this legislation, data sets often only contained sensitive attribute information for a small portion, sometimes as low as 20%, of the population. But often, it can even be under 1% of the total population that is going through their AI system containing sensitive attributes. Additionally, Carignan noted that NYC Local Law 144 initially focused on separate categories such as male, female, and different ethnicities but later recognized the importance of considering intersectionality to address bias effectively. In fact, a version of disparate impact is used by the NYC Local Law 144 (see Phase 1: Assessment – Equality of Outcome).

In closing, I am grateful that CPDP Conferences facilitated the recording of the workshop. You can watch the workshop recording .