New study from The Institute evaluates reliability of ChatGPT as a resource to support families during oral immunotherapy.
A new study led by researchers at the Research Institute of the McGill University Health Centre (The Institute) has found that ChatGPT, a popular artificial intelligence chatbot, can provide generally accurate and intelligible answers to common questions posed by parents navigating oral immunotherapy (OIT) for their children’s food allergies. While highlighting the tool’s potential, the study also identifies important limitations—underscoring the need for its use to be complemented by professional medical guidance.
OIT involves introducing small but increasing amounts of a food allergen into a patient’s diet under medical supervision, with the goal of reducing sensitivity over time. While effective, the process can be lengthy and anxiety-provoking for families, who often seek information online to supplement clinical consultations. The study, published in the Annals of Allergy, Asthma & Immunology, evaluates whether ChatGPT can fill this gap as a self-guided educational tool.
“Parents come to their allergists with an overwhelming number of questions, from basic concerns like ‘what is OIT?’ to very specific ones about medications or activity restrictions during treatment,” said Jana Abi-Rafeh, co-first author and doctoral student working with Bruce Mazer, MD, of the Translational Research in Respiratory Diseases Program at The Institute and the Meakins-Christie Laboratories. “We wanted to know whether ChatGPT could help answer these questions in a reliable, understandable way.”
To assess the chatbot’s utility, the researchers input 14 commonly asked questions into ChatGPT and analyzed the responses for accuracy, consistency, readability and understandability. These responses were then reviewed by 16 OIT experts from North America and the United Kingdom. The experts scored each answer on a 10-point scale based on quality and clinical accuracy.
“The results were encouraging,” said Diana Toscano-Rivero, MD, also co-first author and doctoral student with Dr. Mazer. “ChatGPT scored particularly well on general and advanced questions, and most clinicians rated its responses as being quite helpful. That said, it struggled with more nuanced medical topics. In some cases, it gave vague or even conflicting answers—especially when asked about risks or adverse effects.”
For example, ChatGPT’s responses to the question “Can OIT worsen my child’s allergy?” were inconsistent—one reply stated “yes,” while another said “no.” In terms of readability—measured by word and sentence length and complexity—the responses were often too advanced for the average reader, with scores equivalent to a university reading level. Although understandability ratings—reflecting how well lay readers can comprehend the material—were somewhat higher (ranging from 73% to 84%), reviewers cited technical language and passive voice as barriers to clarity.
Despite these limitations, ChatGPT demonstrated a high degree of reproducibility. When the same question was asked across different sessions and devices, the chatbot produced responses that were up to 93% similar, with only minor variations in phrasing or emphasis.
“This work reflects a broader shift toward digital tools in patient education,” said Bruce Mazer, MD, senior author of the study and Senior Scientist in the Translational Research in Respiratory Diseases Program at The Institute. “Artificial intelligence won’t replace healthcare providers. However, since it is becoming a highly accessible tool, we needed to understand what it can contribute and its limitations.”
The study also draws attention to important ethical considerations, including the risk of misinformation, lack of medical oversight, and fabricated references. As a next step, the researchers recommend further exploration into tailored AI models trained on validated medical data and optimized for different literacy levels.
“Ultimately, our goal is to improve the experience and safety of families navigating oral immunotherapy,” said Dr. Mazer. “AI may become a valuable part of that toolkit—but only if it’s used responsibly, and always in partnership with the healthcare team.”
This research was supported by funding from the Montreal Children’s Hospital Foundation.