In recent years, the volume of generative AI users — particularly with chatbots such as ChatGPT and Gemini — has skyrocketed, and with it, so has the prevalence of patients using generative AI to find health information. As of 2024, 1 in 4 adults under the age of 30 use AI chatbots at least once a month for health-related advice, raising questions as to the safety of this usage and the validity of the chatbots’ outputs.
One major concern is medical misinformation, which, according to the Journal of Clinical Oncology, can cause delays in care and push patients to forgo critical treatment. With ChatGPT-4.0 at a dismal 31% accuracy when it comes to providing medical information, using it as the only source of information presents a serious risk to patients of being led astray. Coupled with the sycophantic tendencies of generative AI that even OpenAI admits are present in ChatGPT, this means that chatbots can also validate patients’ hopeful yet less evidence-based views — such as believing in supplements over conventional cancer treatment — leading to worse outcomes.
Stanford Director for Medical Education in Artificial Intelligence Dr. Jonathan Chen warns against the dangers of this sycophancy, noting that many people ask leading questions — framed to confirm patients’ existing beliefs — which generates the assurance they want rather than critical advice. Instead of asking about long-term nutrition and exercise plans, a patient may ask how to lose weight quickly, inadvertently directing the chatbot to gloss over a more responsible course of action. However, Chen believes patients can sufficiently counteract this by modifying the way they ask questions.
“If you really want a good, objective perspective that can help you achieve your health and wellness, try to go in with a little less bias and ask it to be tough and critical,” Chen said. “You’re more likely to get something that could help you grow in the ways that you need rather than in the ways you want.”
A journal article published in JAMIA Open confirms that starting a conversation with an AI chatbot by asking it to play the part of an expert in a certain medical field raises the chatbot’s general accuracy, relevance and clarity. Chen says he uses this function in his practice to check that he’s considering all possible angles of a case, especially since he says a significant part of working in medicine is making educated guesses.
This method of increasing accuracy can be useful to patients both before and after appointments. Under the 21st Century Cures Act, medical providers are required to send patients their health records upon request, which are often filled with medical jargon that exacerbates patients’ stress and anxiety. Generative AI can help fix this problem by simplifying diagnoses into more digestible language — in studies of both cancer and pathology, AI-generated summaries helped patients better comprehend their conditions. Chen says even doctors’ offices have begun using generative AI to create formal medical documentation based on audio from visits to alleviate the time burden of writing documentation on doctors and scribes.
“That’s become one of the fastest adopted technologies in healthcare I’ve ever seen,” Chen said. “Usually it takes over 10 years for technology to be adopted by healthcare because there are a lot of legacy, inertia, safety and reliability issues. But here they were so desperate because there’s so much burnout for paperwork.”
Even so, the technology would never be adopted if it was not sufficiently accurate at summarizing — the very task patients can most often benefit from. If patients better understand their conditions and a doctor’s recommendation, they can make smarter decisions about the medication they seek and develop higher-quality questions to ask their doctor. This simply depends on patients using AI as a tool to self-advocate rather than a substitute for medical advice.
Furthermore, generative AI allows patients to better determine when they need to visit a doctor in the first place by providing general information and, if prompted correctly, a recommendation for further steps. In this way, generative AI reduces barriers in managing daily health such as nutrition and exercise, particularly for the over 27 million Americans who lack health insurance.
Even beyond providing medical information, generative AI has its benefits in helping patients dispute health insurance claims, returning wrongfully billed payments to patients and their families. In the current system, only 1% of health insurance claims are ever disputed, yet of the disputed claims, 44% are decided in favor of the patient. Generative AI platforms such as Claimable can increase these odds by creating appeal letters that reference extensive research and other patients’ appeal histories, as well as follow the formats most likely to succeed. This makes AI-expedited disputes an incredibly valuable tool to identify and combat unfair claims for people who otherwise lack the time or background information to file a claim themselves.
In light of the recent MVHS Advisory addressing responsible AI usage, it’s imperative that we students adopt the mindset of using AI as a tool rather than a complete substitute for the expertise of medical professionals who can give individualized advice. Though Chen likens using generative AI to having a personal doctor, he also cautions that AI — like any physician — is simply another opinion available to patients, best used critically and with the understanding that it is never guaranteed to be accurate. Still, generative AI extends patients’ ability to self-advocate in the quickly changing and often confusing world of American healthcare. In Chen’s view, this is precisely why engaging responsibly with generative AI is essential to moving forward.
“If you just bury your head in the sand, then don’t be surprised when you get run over by it,” Chen said. “Let’s actively study what AI is capable of and what its credible limitations are so we don’t sell hype that’s fake and make claims that aren’t valid. If you understand what these guardrails are, you can do very useful things with it and figure out a governance and safety structure to do good things while mitigating predictable harms.”
