Icon Legend

This session is not in your schedule.

This session is in your schedule. Click again to remove it.

Session Icons

Thought Leader Session

Guidelines Session

Aging Adult Track

Additional Registration Fee

Research Snapshot Theater: Professional Development and Education, Adult and Pediatric

Evaluation of Multiple AI Chatbots on Respiratory Therapist Multiple-Choice Practice Exam

Sunday, March 22, 2026

11:00 AM - 12:00 PM Central Time

Location: Connections Central - RST 15

JL

Jing Liu, MSc, MSRC, RRT, RRT-NPS

Massachusetts

Disclosure(s): No relevant financial relationship(s) to disclose.
XY

Xiaqiu Yang, MSc

Student
Department of Respiratory Therapy, Faculty of Science, Thompson River University, Kamloops, BC

Disclosure information not submitted.
Fai A. Albuainain, MS, RRT

Ohio

Disclosure(s): No relevant financial relationship(s) to disclose.
PL

Ping-Hui Liu, MSc, MSRC, RRT, RRT-ACCS, RRT-NPS

Research Respiratory Therapist
Division of Respiratory Care, Cincinnati Children’s Hospital, Cincinnati, OH

Disclosure information not submitted.
NA

Naba Alshubaini, RRT

Respiratory Therapist
King Khalid University Hospital, Riyadh, Saudi Arabia

Disclosure information not submitted.
AA

Abdullah Alismail, PhD, RRT, FCCP, FAARC

Professor
Department of Medicine, School of Medicine, Loma Linda University. Department of Cardiopulmonary Sciences, Division of Respiratory Care, Rush University, Chicago, IL

Disclosure information not submitted.
JS

J. Brady Scott, RRT, RRT-ACCS, AE-C, FAARC, FCCP

Professor and Program Director
Department of Cardiopulmonary Sciences, Division of Respiratory Care, Rush University, Chicago, IL

Disclosure information not submitted.
JIE LI, PhD, FCCM

Rush University

Disclosure(s): AARC: Grant/Research Support (Ongoing), Salary (Ongoing), Speaker/Honoraria (includes speakers bureau, symposia, and expert witness) (Ongoing); COPD foundation: Grant/Research Support (Terminated, November 5, 2025); Fisher & Paykel: Speaker/Honoraria (includes speakers bureau, symposia, and expert witness) (Terminated, November 5, 2025); MEK ICS Ltd: Grant/Research Support (Ongoing)

Introduction: Artificial intelligence (AI)-powered chatbots have shown promise in medical assessments and licensing exams. However, their performance on allied health board exams, such as the Respiratory Therapist Multiple-Choice (TMC) Examination, remains unknown. This study aimed to compare the accuracy, consistency, and reasoning quality of seven advanced AI chatbots on the TMC Practice Examination.

Methods: A total of 140 questions were sourced from the free TMC Practice Examination, with the permission of the National Board for Respiratory Care. Seven chatbots—ChatGPT 4o, SuperGrok, Claude Pro Sonnet 3.6, Mistral Pro, Qwen2.5-VL-32B-Instruct, Gemini 2.5 Advanced, and Llama 3.2 (11B and 90B)—were tested. Each question was entered into a new chat session and independently tested by two investigators from April 7–22, 2025. Accuracy was based on alignment with the official answer key. Consistency was measured by the rate of identical responses across repeated trials. For reasoning quality, licensed/board-certified respiratory therapists independently evaluated 20 randomly selected responses using a modified scoring rubric.

Results: All chatbots surpassed the thresholds, with an average accuracy of 83%. ChatGPT attained the highest accuracy of 90%, while Mistral was the lowest (76%). Top performance by content areas included Grok in patient data (95%), Gemini in equipment troubleshooting (88%), and ChatGPT in intervention modification (91%). Mistral and Qwen had the lowest. Cognitive-level analysis showed Gemini, ChatGPT, and Grok leading in recall (92%), application (92%), and analysis (90%), respectively. Self-agreement ranged from 84% (Gemini) to 93% (ChatGPT and Qwen), with a mean of 89%. Justification quality was high overall, with over 90% accuracy in comprehension, retrieval, and reasoning for all chatbots except Qwen (85% in reasoning). Errors, omissions, bias, and potential harm were low, ranging from 3% to 10%.

Conclusions: ChatGPT and Grok outperformed other chatbots on the TMC Practice Examination in both accuracy and explanation quality. Despite promising results, chatbot-generated misinformation, bias, and variability remain concerning. Careful evaluation is warranted before integrating chatbots into clinical education or practice.