Opening the Black Box of AI

​With the inevitable incorporation and reliance on artificial intelligence (AI) in professional settings, especially those with high stakes, many are getting familiar with the “black box” of AI, where the internal decision-making process of models is complex and vast, making it hard to understand the path to the AI’s solution. This creates problems with trust and cooperation, for example, a teacher is not able to give a student full marks if they merely provide their answer without showing their work. Just like teachers, professionals struggle to use AI as a helpful tool for their work due to the unclear process that results in a trust gap, leaving room for hidden biases or hallucinations where the system will make up solutions. Explainable Artificial Intelligence (XAI) is a set of methods and techniques for AI models that describe how these systems arrive at their own conclusions. It acts as a bridge, translating a thorough and deep collection of information into a digestible flow of thought that builds trust and cooperation between AI and humans in real-life scenarios.

A study in Norway and Egypt by Hassan et al. (2025) analyzed clinicians’ interactions with XAI techniques, using a learning model called Spatio-Temporal Graph Convolutional Networks (ST-GCN) that maps joints and movement over time. By producing a graph of these results, scientists are able to identify indicators of cerebral palsy in infants (Hassan et al., 2025). This AI was given videotapes of infants lying down and moving, and it produced a prediction of risk for cerebral palsy. When initially presented with the “black box” AI results, clinicians whose answers conflicted with the AI's found it hard to trust the output due to a lack of justification from the AI, thus adhering to their own initial diagnosis. Then, once presented with the output from XAI methods, such as highlighting specific times that indicated high risk versus low risk and the AI’s certainty level, the clinicians began to engage and question the output as well as their own diagnosis. In addition to its conclusion, the AI also provides justification and its understanding, allowing for more interaction and acceptance from professionals, which is both beneficial and crucial for critical moments that require multiple perspectives. This study found that clinicians using XAI were more likely to reconsider and update their initial diagnosis, which was less likely in scenarios where the “black box” AI was used (Hasan et al., 2025). Improved trust and explanation from XAI methods create learning opportunities where professionals can use AI techniques for important decisions, such as the ones made in this study, and apply those insights to their future medical analyses and identifications. By citing their reasoning and evidence behind each output, AI models reduce a lack of trust and weaknesses like hallucinations, where systems make up data or potential biases (Hasan et al., 2025). 

The lack of transparency in AI creates gaps in ideas and solutions, decreasing efficiency and trust, which is especially harmful in crucial healthcare settings. Explainable AI connects and involves humans in the decision-making process behind outputs. This allows for more applications of AI assistance, including broader perspectives on decision-making and improved accuracy in healthcare. XAI not only builds stronger relationships between machines and humans, but it also creates an environment for curiosity and cooperation in professional settings, further raising AI capabilities in healthcare and other critical fields. As AI continues to become more complex, the role of human understanding becomes even more important, making it crucial to foster successful AI integration in all professional settings.


References

Hassan, R., Nguyen, N., Finserås, S. R., Adde, L., Strümke, I., & Støen, R. (2025). Unlocking the black box: Enhancing human-AI collaboration in high-stakes healthcare scenarios through explainable AI, Technological Forecasting and Social Change, 219. https://doi.org/10.1016/j.techfore.2025.124265.
Turri, V. (2022, January 17). What is Explainable AI?. Retrieved May 6, 2026, from https://www.sei.cmu.edu/blog/what-is-explainable-ai/.
Gilson, D. (2022, October 6). Trust but Verify: Peeking Inside the “Black Box” of Machine Learning. Retrieved May 6, 2026, from https://www.gsb.stanford.edu/insights/trust-verify-peeking-inside-black-box-machine-learning#off-canvas