Stochastic Game Models Based on Markov Decision Processes for Multi-Agent Decision Making in Clinical Healthcare Systems
DOI:
https://doi.org/10.63561/jmns.v2i4.1121Keywords:
Stochastic game theory, Markov decision processes, Multi-agent, Decision-making, HealthcareAbstract
The application of stochastic game models on Markov Decision Processes for multi-agent decision making in clinical healthcare settings has spiked some interests in recent times. Despite advancements in single-agent models, there remains a notable knowledge gap in incorporating multi-agent strategic interactions within stochastic frameworks that adequately address uncertainty in simulation-based environment. This study addressed a preliminary stochastic analysis of patient progression through distinct health states, influenced by healthcare interventions and a simulated patient summary table. The "Recovered" state was identified as an absorbing state, consistent with observed final health outcomes where all patients eventually recovered. The analysis further considered two principal agents: the patient, characterized by initial severity (Mild, Moderate, Severe) and risk (Low, Medium, High) which directly influenced their initial state and potential health progression. The study adopted a simulated-based analysis framework implemented in Python. The healthcare system/decision-makers, whose "Treat" or "Wait" interventions were hypothesized to impact the stochastic transitions. Key results demonstrated that a consistently applied "Treat" policy effectively guided patients through defined health states towards a recovered absorbing state, evidenced by high transition probabilities towards improved conditions. The computed value function, the expected reward for each state, derived via the Bellman equation with a discount factor of 0.95, revealed that managing patients from a "Critical" state, a moderate average payoff of 6.8, improving recovery odds by 15-20%. The framework applied Stochastic game theory based on Markov Decision Processes, the framework posited that "Treat" decisions would accelerate positive transitions and minimized negative ones, offering a higher patient payoff. Two (4 ×4) matrices were created from simulated-based data for transition probability on “treat” and “wait” actions. Markov model was constructed, capturing transitions between health states: Critical, Serious, Stable, and Recovered. Transition probability matrices revealed that all states eventually absorbed into Recovery with probability 1.0. The expected time to recovery from Critical was 3.25 compared to 6.0 from Serious and 7.75 from Stable. Using a reward structure penalizing critical states (0) and rewarding recovery (+8.5), the expected cumulative reward from each initial state was computed as: Critical = 3.25, Serious = 6.0, Stable = 7.75. However, the study recommended that healthcare agents and systems should improve clinical decision-making under uncertainty by applying Markov Decision Processes to minimize patient times spent in critical or serious health states, delays and costs of care in order to ensure evidenced-based support and overall system performance.
References
Acuna, J. A., Zayas-Castro, J., Feijoo, F., Sankaranarayanan, S., Martinez, S. R. & Martinez, D. A. (2021). The Waiting Game; How cooperation between public and private hospitals can help reduce waiting lists. Science Business Media.
Adida, E., Maille, F., & Vial, J. P. (2018). A stochastic game approach to patient-physician interaction in chronic disease management. Operations Research, 66(5), 1223–1241. https://doi.org/10.1287/opre.2017.1704
Archetti, M. (2013). Evolutionary game theory of growth factor production: implications for tumour heterogeneity and resistance to therapies. British Journal of Cancer, 109, 1056–1062.
Arnetz, J. E., Winblad, U., Arnetz, B. B., & Höglund, A. T. (2007). Physicians’ and nurses’ perceptions of patient involvement in myocardial infarction care. European Journal of Cardiovascular Nursing, 7(2), 113–120. https://doi.org/10.1016/j.ejcnurse.2007.05.005
Bauch, C. T., & Earn, D. J. (2004). Vaccination and theory of games. https://doi.org/10.1073/pnas.0403823101
Basar, T. & Olsder, G. J. (1999). Dynamic Noncooperative Game Theory. Society for Industrial and Applied Mathematics. 3, 10A
Blake, A., & Carroll, B. T. (2016). Game theory and strategy in medical training. Medical Education, 50, 1094–1106.
De Bruijn, H., & Heuvelhof, E. T. (2018). Management in networks: On multi-actor decision making (2nd ed.). Routledge.
Filar, J. A. & Vrieze, O. J. (1997). Competitive Markov Decision Processes Springer Science & Business Media.
Folland, S., Goodman, A. C., & Stano, M. (2017). The Economics of Health and Health Care (8th ed. Routledge.
Hauskrecht, M. (2000). Value-function approximations for partially observable Markov decision processes. Journal of Artificial Intelligence Research, 13, 33–94.
Jennings, N. R., Sycara, K., & Wooldridge, M. (1998). A roadmap of agent research and development. Autonomous Agents and Multi-Agent Systems, 1(1), 7–38. https://doi.org/10.1023/A:1010090405266
Klein, G. A. (1993) A recognition-primed decision (RPD) model of rapid decision making. Decision Making in Action: Models and Methods
Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C., & Faisal, A. A. (2018). The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care. Nature Medicine, 24(11), 1716–1720.
Liu, C., Liu, X., Wu, F., Xie, M., Feng, Y, & Hu, C. (2018). Using Artificial Intelligence (Watson for Oncology) for treatment recommendations amongst Chinese patients with lung cancer: Feasibility Study. doi:10.2196/11087
Long, D., Zheng, X., Lu, Q., & Zeng, Y. (2017). Optimal control for sequential treatment strategies of chronic diseases. IEEE Transactions on Cybernetics, 47(12), 4381-4392.
Maskin, E., & Tirole., J. (2001). Markov perfect equilibrium. for surgeons and the operating room environment. Journal of Economic Theory, 100(2), 191–210.
McFadden, D., Kadry, B., & Souba, W. W. (2012). Game theory: Applications for surgeons and the operating room environment. Surgery, 152(5), 915–922.
Mendonça, F. V., Catalão-Lopes, M., Marinho, R. T., & Figueira, J. R. (2020). Improving medical decision-making with a management science game theory approach to liver transplantation. Omega, 94, 102050. https://doi.org/10.1016/j.omega.2019.102050.
Mohr, D. C., & Bittar, H. G. (2014). Game theory analysis of medication adherence behavior. Journal of Medical Systems, 38(3), 9991.
Myerson, R. (2004) Games with Incomplete Information Played by 'Bayesian' Players, I-III. Part I. Management Science. DOI:10.1287/mnsc.1040.0297
Osborne, M. J., & Rubinstein, A. (1994). A course in game theory. MIT Press.
Puterman, M. L. (2005). Markov decision processes: Discrete stochastic dynamic programming. Wiley-Interscience.
Reluga, T. C. (2011). Game theory of social distancing in a pandemic. Public library of science (PLOS), 6(8), e23640. https://doi.org/10.1371/journal.pone.0023640
Rubin, M., Pignatelli, C., & Haimowitz, I. (2016). Game theory and oncology: a systematic review. Journal of Clinical Oncology, 34(15), e20524-e20524.
Scharpf, F. W. (1997). Games real actors play: Actor-centered institutionalism in policy research. Westview Press.
Schelling, T. C. (2010). A practitioner's approach. Economics and Philosophy, 26(1), 27-46.
Stubbings, L., Chaboyer, W., & McMurray, A. (2012). Nurses’ use of situation awareness in decision-making: An integrative review. Journal of Advanced Nursing, 68(7), 1443–1453. https://doi.org/10.1111/j.1365-2648.2012.05989.x
Vahdati, N., Vahdati, S. & Soltani, T. S. (2024). Design and Evaluation of Collaborative Decision-Making Application for Patient Care in the Emergency Department. Health Science Report. 25, 7(2)
Wu, M., Yang, M., Liu, L., & Ye, B. (2016). An investigation of factors influencing nurses’ clinical decision-making skills. Western Journal of Nursing Research, 38, 974–991. https://doi.org/10.1177/0193945916633458
Zhu, X., Jiang, L., Ye, M., Sun, L., Gragnoli, C., & Wu, R. (2016). Integrating evolutionary game theory into mechanistic genotype–phenotype mapping. Trends in Genetics, 32, 256–268.
