Artificial intelligence (AI) is rapidly revolutionizing healthcare, offering huge potential for improved diagnosis, personalized treatment, and operational efficiency. However, for AI to be widely adopted and truly beneficial in such a high-stake area, two connected concepts are fundamental: Explainability and Accuracy. In practice, there is usually a trade-off regarding these concepts in AI. The most complex models are the most accurate and least explainable, while the simplest models are the least accurate and most explainable.

For the purpose of this publication, we present the necessity for AI models to be explainable & accurate and the challenges to do so.

– Explainable AI in Healthcare

Explainable AI refers to the ability of an AI system to clarify how it arrived at a particular prediction, decision, or recommendation. In healthcare, where decisions can directly impact human lives, explainability is not just a technical requirement but an ethical and practical necessity.

Why is Explainability Crucial in Healthcare?

• Trust and Acceptance: Clinicians and patients need to trust AI systems. If an AI gives a diagnosis or recommends a treatment without any clear reasoning, it’s unlikely to be adopted, regardless of its accuracy. Explanations foster trust.
• Clinical Validation and Safety: Understanding why an AI made a decision allows medical professionals to critically evaluate its output, identify potential errors or biases, and ensure patient safety. It helps in validating the AI’s efficacy in real-world clinical settings.
• Accountability and Liability: When an AI system makes a mistake, explainability is crucial for determining accountability. Was it a flawed algorithm, biased data, or improper human oversight? Without explanations, attributing responsibility becomes incredibly difficult.
• Bias Detection and Mitigation: AI models, especially deep learning “black boxes,” can inadvertently pick up biases present in their training data (e.g., underrepresentation of certain demographics). Explainability techniques can help uncover these biases, allowing for their mitigation and promoting fairness and equity in healthcare.
• Learning and Improvement: By understanding the factors an AI considers, clinicians can gain new insights into diseases or treatment responses, potentially leading to new medical discoveries. It also helps developers improve the AI model itself.
• Informed Consent: Patients have a right to understand why certain medical decisions are being made. If an AI plays a role, its reasoning should be communicated in a comprehensible way.

Types of Explainability Methods:

• Inherently Explainable methods: These are for “white box” models designed to be transparent from the beginning. Examples include linear regression, logistic regression, decision trees, and rule-based systems. While often simpler, they may not achieve the same level of accuracy as more complex models for certain tasks.
• Post-hoc Explainable methods: These techniques are applied after a “black box” model (like a deep neural network) has been trained to provide insights into its decisions. Common examples include:
  • LIME (Locally Interpretable Model-agnostic Explanations): Explains individual predictions by approximating the black-box model locally with an interpretable model.
  • SHAP (SHapley Additive exPlanations): Assigns an importance value to each feature for a particular prediction, based on game theory.
  • Feature Importance Maps (Saliency Maps): For image-based AI, these highlight the regions of an image that were most influential in the AI’s decision.
• Global vs. Local Explanations:
  • Global: Explains the overall behavior of the model. “How does the model generally predict outcomes?”
  • Local: Explains a specific prediction for a single patient. It is very challenging. “Why did the model predict this patient has a certain condition?”

Key challenges in Implementing Explainable AI:

• Complexity of AI Models: Highly accurate AI models, especially deep neural networks, are inherently complex and non-linear, making their internal workings difficult to decipher.
• Data Complexity and Privacy: Healthcare data is voluminous, varied, and often sensitive. Explaining decisions sometimes requires access to this sensitive data, raising privacy concerns.
• Lack of Standardization: There’s no universal agreement on what constitutes a “good” explanation or standardized metrics for evaluating explainable AI methods.
• Integration into Clinical Workflows: Explanations need to be presented in a way that is intuitive and actionable for clinicians, fitting seamlessly into existing clinical workflows.
• User Understanding: Even with explanations, some users may lack the necessary background knowledge to fully comprehend them. Explanations need to be tailored to different user groups.

– Accurate AI in Healthcare

Accuracy refers to how well an AI system’s predictions or classifications match the true outcomes. In healthcare, high accuracy is paramount, as misdiagnosis or incorrect treatment recommendations can have severe consequences.

Factors Contributing to AI Accuracy in Healthcare:

• High-Quality, Diverse Data: AI models learn from data. The quantity, quality, and diversity of training data are critical. Data must be representative of the target population to avoid biases and ensure generalizability. This includes diverse demographics, clinical presentations, and disease stages.
• Robust Algorithms and Model Selection: Choosing the right AI algorithm for a specific healthcare task is crucial. Advanced machine learning and deep learning models can identify complex patterns in large datasets.
• Feature Engineering: Identifying and preparing the most relevant features (patient symptoms, lab results, imaging features, genetic markers) from raw data significantly impacts accuracy.
• Validation and Testing: Rigorous validation using independent datasets is essential to ensure the AI performs accurately on unseen data. This includes internal testing, external validation studies, and peer review.
• Continuous Monitoring and Iteration: AI models can degrade over time due to changes in patient populations, medical practices, or data input. Continuous monitoring and regular updates are necessary to maintain accuracy.
• Human-in-the-Loop: While AI can augment human capabilities, human oversight and collaboration are vital. Clinicians provide critical context and can identify AI errors that might otherwise go unnoticed.
• Addressing Data Challenges: Healthcare data often has missing values, noise, and heterogeneity. Effective data preprocessing techniques are needed to handle these issues and improve accuracy.

Areas where AI shows high accuracy potential:

• Medical Imaging Analysis: AI, particularly deep learning, excels at analyzing medical images (X-rays, MRIs, CT scans) for detecting subtle anomalies indicative of diseases like cancer, diabetic retinopathy, or neurological disorders.
• Pathology: AI can assist in analyzing histopathology slides for cancer diagnosis and grading.
• Early Disease Detection: By analyzing vast amounts of patient data (EHRs, wearables), AI can identify risk factors and predict disease onset earlier than traditional methods.
• Drug Discovery and Development: AI can accelerate the identification of new drug candidates, predict their efficacy and toxicity, and optimize drug design.
• Personalized Medicine: AI can analyze an individual’s genetic makeup, lifestyle, and clinical data to tailor treatment plans for maximum effectiveness.
• Predictive Analytics: Forecasting patient deterioration, hospital readmissions, or disease outbreaks.

Challenges in Achieving and Maintaining Accuracy:

• Data Availability and Quality: Obtaining sufficiently large, diverse, and well-annotated healthcare datasets can be challenging due to privacy concerns, data silos, and the cost of annotation.
• Data Bias: If training data is biased (e.g., predominantly from one ethnic group or socioeconomic status), the AI model may perform poorly or perpetuate disparities when applied to underrepresented groups.
• Generalizability: An AI model trained on data from one hospital system or population may not perform as accurately when deployed in a different setting.
• Dynamic Nature of Healthcare: Medical knowledge and practices evolve, requiring continuous updates and retraining of AI models.
• Rare Diseases: AI accuracy can be limited for rare conditions due to insufficient training data.
• Regulatory Hurdles: Demonstrating and validating AI accuracy to regulatory bodies is a complex and evolving process.

In summary, both explainability and accuracy are fundamental pillars for the successful and ethical integration of AI in healthcare. Achieving high accuracy is crucial for effective outcomes, but without explainability, AI systems will struggle to gain the trust and adoption necessary to truly revolutionize patient care. The ongoing research and development in Explainable AI aim to bridge the gap between highly accurate “black box” models and the need for transparent, trustworthy AI in critical domains like healthcare.

References:

Ennab M, Mcheick H. Enhancing interpretability and accuracy of AI models in healthcare: a comprehensive review on challenges and future directions. Front Robot AI. 2024 Nov 28; 11:1444763. doi: 10.3389/frobt.2024.1444763. PMID: 39677978; PMCID: PMC11638409.

Marzyeh Ghassemi, Luke Oakden-Rayner, Andrew L Beam. The false hope of current approaches to explainable artificial intelligence in health care. http://www.thelancet.com/digital-health Vol 3 November 2021