Effects of Varying $k$ in $L_{k}$ Norm Similarity Measures on a Reinforcement Learning-Based Clinical Decision Support System

Reinforcement Learning (RL) methods are increasingly recognized as a viable approach to address healthcare challenges, particularly in critical care, where clinical sequential decision making is closely aligned with the RL paradigm. How-ever, critical issues remain to be addressed, such as defining an optimal Markov Decision Process (MDP) that accurately reflects real-world clinical scenarios. In many proposed RL approaches for the Intensive Care Unit (ICU), partitioning clustering has been used to define MDP states. However, such algorithms are vulnerable to high-dimensional data, which is necessary to account for all potential confounding factors. Previous studies have shown that using
$L_{k}$
norms as a similarity measure leads to more robust clustering solutions as the value of
$k$
decreases. This can potentially result in an MDP that better captures the clinical scenario. The objective of this study is to examine how varying
$k$
in
$L_{k}$
norms, affects the estimated optimal policy obtained by RL. The study used data from 20,496 ICU stays of septic patients to construct two different MDPs using the k-medoids algorithm. The first MDP had states defined using Manhattan distance
$(L_{1}$
norm), while the second MDP used Euclidean distance (
$L_{2}$
norm). Both MDPs were solved using the policy iteration RL technique to determine the optimal policy, which was then evaluated using the Weighted Importance Sampling (WIS) estimator. The policy generated by solving the MDP constructed using the Manhattan distance achieved a 99% confidence lower bound (LB) of 88.6048, outperforming the policy generated by solving the MDP constructed using the Euclidean distance (99% confidence LB of 82.9936). This result suggests that selecting a lower value of
$k$
in the
$L_{k}$
norm, which is used as a similarity measure to define the MDP in the RL framework, may be a critical aspect of an AI-based clinical decision support system. This approach might result in a lower mortality rate than selecting a higher k, a choice that could result in an MDP that inadequately represents the clinical scenario and might consequently yield poor outcomes.