Shedding light into AI’s black boxes

When large language models, or generative AI, make mistakes, they often do so with incredible confidence. But how does a user really know why a recommendation was made or where it came from? 

It’s a reality that has created challenges — and opportunities — for researchers like Dr. Samira Ebrahimi Kahou, PhD, an associate professor at the Schulich School of Engineering at the University of Calgary. 

“Artificial intelligence makes headlines for its risks, but I’ve always focused on its potential for good. AI has enabled incredible progress in software development, where we can test and verify code before it causes harm,” says Ebrahimi Kahou, director of UCalgary’s Transdisciplinary AI Hub and affiliated with Mila, one of Canada’s three national AI institutes. She also previously held a CIFAR AI Chair researching how AI models generalize and reason through decisions.

Ebrahimi Kahou recently turned to a bigger question: how to make AI’s reasoning understandable. Her work advances explainability and generalization — two key concepts that show how models reach their predictions and make them reliable with new data.

Understanding the ‘why’ behind an AI suggestion

By combining unclear, or “opaque,” neural networks with simpler methods such as decision trees, Ebrahimi Kahou aims to make AI “black boxes” more transparent, helping users and developers understand how a model reached its recommendation. A neural network is a computer system that learns to recognize patterns and solve complex problems, while a black box is defined by IBM as, “an AI system whose internal workings are a mystery to its users.” 

One real-world application or this work is in health care. Ebrahimi Kahou is collaborating with faculty at UCalgary’s Cumming School of Medicine to develop models that can help detect brain tumours, uncovering biomarkers that the human eye might have missed.

“My research targets situations where we need to understand a model’s reasoning to catch mistakes, assign accountability and earn trust, like using AI for clinical decision support,” she says. “If you know why your model is broken, you can fix the problems and keep the AI from causing harm to users.”  

That focus on fixing models before they can cause harm drives her latest project. With a grant from Coefficient Giving, Ebrahimi Kahou and her collaborators are studying emerging harmful behaviours in AI, using tools to spot them early.

Redefining tech through diversity

Ebrahimi Kahou was once one of only a handful of women in the lab, sometimes the only one. She has worked to clear a path for those coming after, and challenges her students to keep that path open.

“If you want biases removed, you need to be the one who designs the system, especially for women. If you think there are too many obstacles, don’t wait for them to disappear — focus on the impact you want to bring to the world,” she says. “Think about what you are building and what it will do 10 years from now.”

This outlook for lasting impact recently earned her the Ten-Year Technical Impact Award from the International Conference on Multimodal Interaction, a leading forum for multimodal AI and social interaction research that advances innovative, human-centred interactive systems. 

The award recognized her paper, “Recurrent neural networks for emotion recognition in video,” which was an early demonstration of how deep learning could capture how visual cues change over time, and fuse that information effectively with other signals like audio. 

“Research requires curiosity, persistence and tolerance for uncertainty. Sometimes, the impact of a project is not clear for a while, it only emerges over time,” says Ebrahimi Kahou. “It’s amazing to see how that original research evolved over 10 years and is now used in many applications.”