Abstract

Artificial Intelligence (AI) holds substantial promise for augmenting clinical decision-making, yet barriers to interpretability and real-world deployment remain. In this work, we introduce AICare, an AI–clinician interaction system designed to enhance transparency, encourage expert feedback, and ultimately improve the deployment potential of deep learning models in healthcare. AICare integrates four key components (Figure 1): (1) visualization of complex patient electronic health record (EHR) data, (2) visualization of model predictions with both granular and longitudinal interpretability, as well as patient clustering and similarity assessments, (3) large language model (LLM)-based summarization for comprehensive clinical reports, and (4) a questionnaire module that captures expert feedback along with fine-grained behavioral data on how clinicians review and interpret patient information. By adopting emerging standards like OMOP for data structuring, AICare can be deployed offline in diverse healthcare environments.

One of the primary challenges in clinical AI is effectively visualizing heterogeneous, longitudinal EHR data that span laboratory results, procedures, and clinical events. AICare addresses this by consolidating these elements into an interactive interface that is both intelligible and actionable. This design aligns with clinical workflows requiring quick, real-time assessment of large volumes of patient historical data. Building on this visualization framework, we incorporate state-of-the-art deep learning models as well as conventional clinical prediction models that generate risk trajectories such as mortality or re-admission risk, while also employing interpretable models to highlight the most influential clinical features at both the feature (e.g., vital signs, lab trends) and visit (e.g., episodes of care) levels. By clarifying how individual model features contribute to predictions, we promote transparency and enable clinicians to more confidently calibrate their reliance on AI-generated recommendations.

AICare goes beyond prediction to provide advanced analysis such as patient clustering based on latent model embeddings, helping clinicians contextualize typical disease progressions and validate a model’s risk estimates. This capability is complemented by an LLM-based reporting mechanism that synthesizes complex patient data, model outputs, and cluster-level insights into clinically oriented summaries. By bridging the gap between raw machine learning outputs and actionable clinical intelligence, we strive to make AI-driven insights more accessible and meaningful in real-world settings.

To further support the robust deployment of AI, AICare includes a questionnaire system that presents the same EHR-derived features the model considers, prompting clinicians to complete identical prediction tasks. By capturing the order and duration of clinicians’ feature inspection, we gain insights into the reasoning processes involved in diagnosis. Comparing these human interpretability pathways with the model’s rationales highlights areas of divergence or alignment. This dual interpretability framework can guide the refinement of machine learning models through reinforcement learning from human feedback (RLHF), ultimately contributing to the development of more trustworthy, generalizable “medical foundation models.”

Overall, AICare aims to advance the deployment of AI models by prioritizing interpretability, feedback capture, and seamless integration into clinical practice. Through rigorous evaluation and iterative refinement, we envision this platform as a critical enabler for safe, transparent and constructive AI solutions in healthcare.