Identifying and Measuring Algorithmic Bias

Overview

Fairness is key component of ethical AI. In a technical context, fairness means ensuring that an AI system does not produce systematically different or discriminatory outcomes across groups based on sensitive attributes such as race, gender, religion, or disability.

Algorithmic bias is a repeatable error in a system that creates unfair outcomes, such as generating predictions that are unbalanced across different inputs, potentially favoring one or more groups over others.

For the developer, the first step is to understand where bias originates and how to measure its presence.

Sources of algorithmic bias

Bias is rarely introduced maliciously; it is typically an unintended consequence of design choices and data limitations. Developers must be vigilant about the three primary sources of bias:

·     Historical bias (societal bias): This bias is inherent in real-world data and reflects past and present societal prejudices. For example, if a hiring model is trained on historical data where men predominantly held high-level technical roles, the model will learn to associate "male" with "successful technical candidate," even if gender is not an explicit feature.

·     Measurement bias (data collection bias): This occurs when the data collected does not accurately reflect the real-world construct being measured. For instance, using zip code as a feature can act as a proxy for socioeconomic status, effectively reintroducing a sensitive attribute that was explicitly removed.

·     Algorithmic bias (model bias): This arises during model training. It can be caused by the choice of algorithm (e.g., an algorithm that optimizes overall accuracy may ignore poor performance among a small, minority group) or by imbalanced training data, in which the model learns less about the underrepresented group.

Key fairness metrics for developers

Fairness is not a single, monolithic concept; it is a family of mathematical definitions. The choice of metric depends heavily on the application's context and the potential harm being mitigated.

Developers should consider moving beyond simple accuracy and adopt metrics that quantify the disparity between groups.

Let's explore some key metrics:

·     Disparate Impact Ratio is the ratio of the favorable outcome rate for the unprivileged group to the favorable outcome rate for the privileged group. DIR should be close to 1.0 (typically between 0.8 and 1.25).

o     Used in high-stakes decisions like hiring or loan approval, where the rate of selection should be similar across groups.

·     Equal Opportunity Difference is the difference in the true positive rate (TPR) between the privileged and unprivileged groups. EOD should be close to 0.0.

o     Used when ensuring that qualified individuals from all groups have an equal chance of being correctly classified (such as correctly identifying a disease).

·     Average Odds Difference is the average of the difference in true positive rate (TPR) and false positive rate (FPR) between the privileged and unprivileged groups. AOD should be close to 0.0.

This is a comprehensive metric that balances both false positives and false negatives across groups.

Ethical coding practice: Calculating disparate impact

The disparate impact ratio (DIR) is one of the most common and legally relevant fairness metrics, derived from the Four-Fifths Rule in US employment law. It's important to understand that a DIR below 0.8 is often arbitrarily considered evidence of adverse impact.

The following Python code demonstrates how to calculate the DIR using a synthetic credit risk model output.

Code Listing 4-a: Simple Python DIR example (disparate-impact.py)

This code demonstrates how to calculate and interpret the DIR to assess whether a model's predictions disproportionately disadvantage an unprivileged group.

First, it simulates a dataset containing a protected attribute (Race), the true outcome (Actual_Risk), and the model's prediction (Prediction).

Two groups are: Group A, the privileged_group, and Group B, the unprivileged_group.

A favorable outcome is explicitly defined as credit approval (Prediction == 0), which makes the fairness criterion transparent and auditable.

Next, the code computes the favorable outcome rate for each group by calculating the proportion of predictions that are approvals within that group. The data is filtered by race, and the approval rate is computed separately for the privileged and unprivileged groups.

Then it calculates the DIR as the approval rate for the unprivileged group divided by the approval rate for the privileged group. A guard is included to avoid division by zero, which reflects good defensive coding practice.

Finally, the results are printed along with an ethical interpretation based on the commonly used 80% rule. If the DIR is below 0.8, the code flags a potential adverse impact against the unprivileged group; otherwise, it notes that the outcome rates are within an acceptable range. This makes the fairness assessment explicit, transparent, and easy to interpret.

With that said, let's run the code from the terminal within VS Code using the following command: py disparate-impact.py.

Code Listing 4-b: Simple Python DIR example execution

Favorable Outcome Rate (Privileged Group A): 0.7000
This means that 70% of individuals in the privileged group received a favorable outcome (for example, approval, acceptance, or an optimistic prediction).

Favorable Outcome Rate (Unprivileged Group B): 0.7000
This shows that the unprivileged group also received favorable outcomes at the same 70% rate.

Disparate Impact Ratio (DIR): 1.0000
The DIR is calculated as the favorable rate for the unprivileged group divided by the favorable rate for the privileged group.

A value of 1.0000 indicates that the two groups are treated identically with respect to outcome rates—there is no disparity in this measure.

The note explains that the DIR is within the commonly used 80% rule threshold (DIR ≥ 0.8).

Since 1.0000 is well above 0.8, this check indicates no evidence of disparate impact between the groups based on outcome rates. This result only shows that the rates of favorable outcomes are equal. It does not guarantee that the model is entirely fair. Other issues (such as differences in error rates, calibration differences, or harms associated with false positives versus false negatives) may still exist and should be evaluated separately.

Ethical coding practice: Equal opportunity difference

Now, let's look at an equal opportunity difference Python code. This is a fairness metric that compares how well a model correctly identifies true positives across different groups.

Code Listing 4-c: Equal opportunity difference example (equal-opp-diff.py)

First, it defines the test data: y_true contains the true labels (where 1 indicates a qualified or positive case), and y_pred includes the model's predictions.

A protected attribute (group) labels each data point as belonging to either a Privileged or Unprivileged group.

Next, the function true_positive_rate computes the true positive rate (TPR)—also known as recall or sensitivity—for a given subset of data. It does this by building a confusion matrix, extracting the true positives (tp) and false negatives (fn), and returning tp/(tp+fn). A safety check avoids division by zero when no positive cases are present. The code then calculates TPR separately for each group by filtering the true labels and predictions according to group membership. This yields one TPR value for the privileged group and one for the unprivileged group.

After that, it computes the equal opportunity difference (EOD) as the difference between the unprivileged group's TPR and the privileged group's TPR.

Finally, the code prints the results and applies a simple ethical interpretation rule:

If the absolute EOD is within a small tolerance (around 0.05), the equal opportunity is considered satisfied; otherwise, it flags a warning that individuals may be more often missed in one group, signaling a potential fairness issue.

Now, let's run this code by invoking this command from the terminal in VS Code: py equal-opp-diff.py.

Code Listing 4-d: Equal opportunity difference example execution

TPR (Privileged Group): 1.000

The model correctly identified 100% of truly positive cases in the privileged group. No qualified individuals in this group were missed.

TPR (Unprivileged Group): 0.333

The model correctly identified only 33.3% of truly positive cases in the unprivileged group. This means that two out of three qualified individuals in this group were incorrectly classified as negative (false negatives).

A value of -0.667 indicates a significant gap in favor of the privileged group. The negative sign shows that the unprivileged group is being missed much more often.

The warning highlights a serious fairness concern: although the model performs perfectly for the privileged group, it fails to give qualified individuals in the unprivileged group an equal chance of being correctly identified. In contexts such as medical diagnosis, credit approval, or safety screening, this could lead to systematic harm, in which deserving individuals from the unprivileged group are denied benefits or incorrectly given unwarranted benefits or protection at a much higher rate.

In short, this result indicates a violation of equal opportunity, and the model would require investigation, mitigation, or redesign before it could be considered ethically acceptable.

Ethical coding practice: Average odds difference

Now, let's have a look at another Python example code. This code demonstrates how to calculate and interpret average odds difference (AOD), a fairness metric that checks whether a model's error rates are balanced across protected groups.

Code Listing 4-e: Average odds difference example (avg-odds-diff.py)

First, it defines the test data: y_true contains the true labels (where 1 indicates a positive or qualified case), and y_pred includes the model's predictions. A protected attribute (group) labels each data point as belonging to either a Privileged or Unprivileged group.

Next, the helper function tpr_fpr computes two key error rates from the confusion matrix:

·     True positive rate (TPR), which measures how often truly positive cases are correctly identified (sensitivity/recall).

·     False positive rate (FPR), which measures how often truly negative cases are incorrectly predicted as positive.

The code then calculates TPR and FPR separately for each group by filtering the data according to group membership. This yields one pair of rates for the privileged group and one pair for the unprivileged group.

After that, it computes the average odds difference (AOD) as the average of the TPR and FPR differences between the two groups. This single value indicates whether the model treats groups similarly with respect to both false negatives and false positives.

Finally, the script prints the per-group rates and the AOD value, then applies a simple ethical interpretation rule:

If the absolute AOD is within a small tolerance (here, ±0.05), the model's error rates are considered approximately equal across groups; otherwise, it issues a warning that one group is experiencing systematically different error rates, indicating a potential fairness issue.

Now, let's run this command from the terminal within VS Code to see the results: py avg-odds-diff.py.

Code Listing 4-f: Average odds difference example execution

TPR (Privileged): 1.000
The model correctly identifies 100% of truly positive cases in the privileged group. No qualified individuals in this group are missing.

TPR (Unprivileged): 0.333
The model correctly identifies only 33.3% of truly positive cases in the unprivileged group. Most qualified individuals in this group are incorrectly rejected (false negatives).

FPR (Privileged): 0.333
About 33.3% of truly negative cases in the privileged group are incorrectly predicted as positive (false positives).

FPR (Unprivileged): 0.500
In the unprivileged group, 50% of truly negative cases are misclassified as positive, leading to more false alarms.

AOD: -0.250
Average odds difference is the average of the differences in TPR and FPR between groups.

The negative value indicates that, on average, the unprivileged group experiences significantly different outcomes: they are both missed more often when they qualify and incorrectly flagged more often when they do not.

The warning highlights a meaningful fairness problem. Even though the model performs perfectly for the privileged group on positives, it treats the unprivileged group significantly worse for both types of errors.

In real-world contexts (such as healthcare, credit, or risk assessment), this combination can lead to systematic harm and strongly suggests the model needs mitigation, redesign, or tighter governance before deployment.