Advancing AI Trustworthiness Through Patient Simulation: Risk Assessment of Conversational Agents for Antidepressant Selection
arXiv:2602.11391v3 Announce Type: replace Abstract: Objective: This paper introduces a patient simulator for scalable, automated evaluation of healthcare conversational agents, generating realistic, controllable interactions that systematically vary across medical, linguistic, and behavioral dimensions to support risk assessment across populations. Methods: Grounded in the NIST AI Risk Management Framework, the simulator integrates three profile components: (1) medical profiles constructed from All of Us electronic health records using risk-ratio gating; (2) linguistic profiles modeling health — Md Tanvir Rouf Shawon, Mohammad Sabik Irbaz, Hadeel R. A. Elyazori, Keerti Reddy Resapu, Yili Lin, Vladimir Franzuela Cardenas, Farrokh Alemi, Kevin Lybarger
View PDF
Abstract:Objective: This paper introduces a patient simulator for scalable, automated evaluation of healthcare conversational agents, generating realistic, controllable interactions that systematically vary across medical, linguistic, and behavioral dimensions to support risk assessment across populations. Methods: Grounded in the NIST AI Risk Management Framework, the simulator integrates three profile components: (1) medical profiles constructed from All of Us electronic health records using risk-ratio gating; (2) linguistic profiles modeling health literacy and condition-specific communication; and (3) behavioral profiles representing cooperative, distracted, and adversarial engagement. Profiles were evaluated against NIST AI RMF trustworthiness requirements and assessed against an AI Decision Aid for antidepressant selection. Results: Across 500 simulated conversations, the simulator revealed monotonic degradation in AI Decision Aid performance across health literacy levels: Rank-1 concept retrieval ranged from 47.6% (limited) to 81.9% (proficient), with corresponding recommendation degradation. Medical concept fidelity was high (96.6% across 8,210 concepts), validated by human annotators (0.73 kappa) and an LLM judge with comparable agreement (0.78 kappa). Behavioral profiles were reliably distinguished (0.93 kappa), and linguistic profiles showed moderate agreement (0.61 kappa). Conclusions: The simulator exposes measurable performance risks in conversational healthcare AI. Health literacy emerged as a primary risk factor with direct implications for equitable AI deployment.
Subjects:
Computation and Language (cs.CL)
Cite as: arXiv:2602.11391 [cs.CL]
(or arXiv:2602.11391v3 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2602.11391
arXiv-issued DOI via DataCite
Submission history
From: Md Tanvir Rouf Shawon [view email] [v1] Wed, 11 Feb 2026 21:53:18 UTC (1,309 KB) [v2] Wed, 25 Mar 2026 17:20:58 UTC (1,253 KB) [v3] Fri, 27 Mar 2026 15:00:31 UTC (1,253 KB)
Sign in to highlight and annotate this article
Conversation starters
Daily AI Digest
Get the top 5 AI stories delivered to your inbox every morning.
More about
researchpaperarxiv
Major 4-day workweek study suggests that when we work 5 days we spend one doing basically nothing
Research says workers can get as much done in a 33-hour week as in 38 hours. Essentially, those of us on a five-day week are filling up our days with time-wasting activities.
UKRI Deems Turing Institute Not Yet Satisfactory
UK Research and Innovation (UKRI) found that the Alan Turing Institute s strategic alignment and value for money are not yet satisfactory in a review of the AI research body s performance. The Turing Institute has dealt with a tumultuous year, with its head stepping down amid pushback from staff complaining about a toxic work environment. The [ ] The post UKRI Deems Turing Institute Not Yet Satisfactory appeared first on DIGIT .
Knowledge Map
Connected Articles — Knowledge Graph
This article is connected to other articles through shared AI topics and tags.
Discussion
Sign in to join the discussion
No comments yet — be the first to share your thoughts!