Medical AI Pinpoints Key Factors for More Reliable Diagnoses

Researchers are tackling the persistent challenge of feature selection in medical prediction, where current methods like LASSO frequently suffer from limited robustness and interpretability. Yuheng Luo from the Chinese Academy of Medical Sciences & Peking Union Medical College, Shuyan Li from Queen’s University, Belfast, United Kingdom, and Zhong Cao from Heidelberg University, working in collaboration, present GRASP, a new framework combining Shapley value attribution with group regularisation to identify concise and non-redundant feature sets. GRASP initially determines group-level feature importance using SHAP values from a pre-trained tree model, subsequently imposing structured sparsity via group regularised logistic regression, resulting in stable and readily interpretable selections. Comprehensive comparisons against LASSO, SHAP, and related techniques demonstrate that GRASP consistently achieves equivalent or improved predictive accuracy, whilst pinpointing fewer, less correlated, and more reliable features.

Scientists have developed a new method for selecting the most relevant features in complex medical datasets, addressing longstanding challenges in predictive accuracy and interpretability. This work introduces GRASP, an acronym for GRoup-SHAPley feature Selection for Patients, a framework that combines Shapley value attribution with group regularization to identify compact and meaningful sets of variables for medical prediction. GRASP distinguishes itself by simultaneously optimising for both predictive performance and the clinical relevance of selected features, a crucial requirement often overlooked in existing approaches. The research demonstrates that GRASP consistently matches or exceeds the accuracy of established techniques like LASSO, while delivering significantly more stable and interpretable results. GRASP tackles the problem of high-dimensional medical data, where the sheer volume of potential variables can hinder effective knowledge discovery. The method begins by leveraging a pretrained tree model to calculate Shapley values, a concept from game theory used to quantify the contribution of each feature to the model’s predictions. These values are then used to guide a group-based regularization process, encouraging the selection of cohesive and clinically meaningful feature sets. Unlike traditional methods that often produce unstable and redundant feature lists, GRASP prioritises structured sparsity, resulting in models that are both simpler and more robust. The core of GRASP lies in its ability to integrate interpretability directly into the feature selection process. By coupling Shapley values with a group L21 regularization penalty, the algorithm actively seeks features that are not only predictive but also align with established clinical understanding. This is achieved through a proximal-gradient optimisation algorithm, carefully tuned to balance predictive accuracy with the desire for concise and interpretable models. Extensive evaluations across multiple datasets reveal that GRASP consistently identifies fewer features than competing methods, while maintaining comparable or superior predictive power. The resulting feature sets are demonstrably less redundant and exhibit greater stability, offering a significant advantage for real-world clinical applications. By explicitly evaluating both performance and interpretability, the study highlights the importance of considering clinical relevance alongside statistical metrics. The framework’s ability to simplify complex models and enhance stability positions GRASP as a promising tool for precision medicine, enabling clinicians to gain deeper insights from patient data and make more informed decisions. The method’s design facilitates the extraction of the most informative variables, reducing computational costs and paving the way for more efficient and effective healthcare solutions. A 72-qubit superconducting processor forms the foundation of the GRASP methodology, enabling the distillation of group-level feature importance scores. Initially, a pretrained tree model, specifically an XGBoost classifier, underwent training on the training fold of the dataset to establish a baseline predictive capability. Subsequently, SHAP values were computed on a held-out validation fold, quantifying the contribution of each feature to the model’s output. These Shapley values, representing each feature’s marginal impact on prediction, were then averaged across all samples in the validation set to derive a single importance score, φj, for each feature j. To facilitate the selection of cohesive and interpretable feature sets, features were partitioned into G disjoint groups. An aggregated group importance, sg, was calculated for each group by averaging the individual feature importance scores of its constituent members. This grouping strategy allows for correlated features to be considered collectively, promoting stability and reducing redundancy in the final selection. The resulting group importance scores were then integrated into a group-L21 regularized logistic regression framework. This framework employs a proximal-gradient algorithm with Armijo backtracking to optimise the selection process. The loss function combines cross-entropy, measuring the difference between predicted and actual outcomes, with a penalty term enforcing structured sparsity. Specifically, the L21 norm of the feature weights encourages the selection of entire groups of features, rather than individual features in isolation, further enhancing interpretability and generalisation performance. This approach unifies prediction and interpretability within a single optimisation objective. Across both the NHANES and UK Biobank datasets, GRASP consistently identified feature sets exhibiting high predictive performance. Logistic Regression models trained on GRASP-selected features achieved an accuracy of 0.783 on NHANES and 0.755 on UKB, alongside F1 scores of 0.483 and 0.197 respectively. Random Forest models demonstrated even stronger performance, reaching accuracies of 0.890 on NHANES and 0.946 on UKB, though F1 scores were lower at 0.226 and 0.016. XGBoost models mirrored these results, attaining accuracies of 0.897 and 0.942, with F1 scores of 0.437 and 0.143 on the respective datasets. Notably, GRASP achieved these results while selecting an average of only 23 features, significantly fewer than the 44, 43, and 59 features identified by LASSO, SHAP, and AFS respectively. This parsimony translated to improved stability, as measured by an Adjusted Stability Measure of 0.593 for GRASP, compared to 0.382 for LASSO, 0.398 for SHAP, and 0.258 for AFS. Redundancy, assessed using Variance Inflation Factor, was also lowest for GRASP at 2.942, indicating minimal multicollinearity among selected features. Further analysis revealed a Shapley Value of 0.101 for GRASP, and a Pearson correlation coefficient of 0.070. Calibration curves demonstrated that GRASP’s predicted probabilities aligned more closely with observed risks, particularly within high-risk groups, while Kaplan-Meier survival curves showed comparable or superior discrimination compared to LASSO and SHAP. Scientists developing predictive models in medicine face a persistent paradox; while increasingly sophisticated algorithms promise improved accuracy, understanding why a model makes a particular prediction remains elusive. This work offers a compelling step towards resolving that tension, not simply by boosting performance, but by delivering genuinely interpretable results. The introduction of GRASP, a framework combining Shapley values with group regularization, represents a move away from ‘black box’ approaches and towards a more transparent, clinically useful form of artificial intelligence. For years, feature selection has relied heavily on techniques like LASSO, which, while effective at paring down complex datasets, can produce unstable and opaque selections. GRASP appears to address this by prioritising not just predictive power, but also the stability and coherence of the chosen features. Identifying fewer, less redundant variables is crucial; it’s not enough to predict accurately if clinicians cannot readily grasp the underlying reasoning. However, the reliance on a pre-trained tree model introduces a potential dependency, as the initial model’s biases could propagate through the GRASP pipeline, limiting the novelty of the final feature set. Furthermore, while the authors demonstrate improvements across several datasets, the true test will be its performance on truly high-dimensional, real-world clinical data where feature interactions are far more comple.