Public Input Could Make AI Fairer, Glasgow Study Finds

Involving people without AI expertise in the development and evaluation of artificial intelligence applications could help create better, fairer and more trustworthy automated decision-making systems, new research from the University of Glasgow suggests.

After enlisting members of the public to evaluate the potential impacts of two real-world applications, researchers from across the UK, led by Glasgow University, found that ‘participatory AI auditing’ could improve AI decision-making.

Responsibility for ensuring the applications make fair and impartial decisions usually lies with the engineers and data scientists who develop them, but when developers fail to consider the full social or economic conditions of people affected by the tools’ outcomes, unexpected problems can arise.

For example, in 2019, a healthcare algorithm trained to predict patients’ health risk score was found to demonstrate bias in underrating the severity of Black patients’ health conditions relative to their white peers, while an early Amazon AI tool to rank job applicants discriminated against women, after being trained on ten years of resumes predominantly from men.

By involving a wider group of people in the early stages of AI development, participatory audits might prevent those problems before they occur. Although participants may lack specific technical knowledge of how the systems work, the research found they can offer unexpected insight into social and ethical considerations that traditional audits overlook.

However, for this to work, the researchers argue that participants need significant support to help them provide useful feedback and new tools to guide them through the audit process.

Setting up co-design workshops, the team tasked seventeen people without AI expertise with auditing two real-world AI tools designed for use in healthcare and education.

The auditors were tasked with identifying the applications’ potential impacts, determining how those impacts should be measured, and suggesting how tools to support audits might work.

Although the study initially focused on identifying harms, participants were keen to ensure that the benefits of the applications for the groups were captured, saying they felt constrained by having to mark aspects of the systems as having passed or failed, suggesting a third option is needed when an impact defies binary categorisation.

Recommended reading

• KPMG UK Interview: Why Trust is AI’s Most Underrated Asset

• How will AI Change Where Software Engineers Add Value?

• AI Progress Fatigue Is Starting to Show, Finds Bosch

The research team are now working to build a framework for responsible AI auditing employing a collaborative approach to building AI applications, which will help improve public trust as well as benefit the organisations which develop them.

“Regulations like the European Union’s AI Act, introduced in 2024, are seeking to limit the harms that badly-designed AI applications could inflict on the people affected by their decisions,” said Professor Simone Stumpf, of Glasgow’s School of Computing Science, the project’s lead investigator.

“Our research aims to provide a systematic framework and tools to help people without AI expertise use their lived experience to identify and report those harms through participatory audits, and ultimately be more involved in creating more trustworthy AI systems.”