ChatGPT may help provide suggestions for your workouts, But there’s still a long way to go before we can replace doctors. A recent experiment found that a popular artificial intelligence chatbot often made the wrong medical calls.
“ChatGPT in its current form is not accurate as a diagnostic tool,” said the researchers behind the project. studyPublished today in the magazine PLOS ONE, writes. “Although ChatGPT is trained on a large amount of information, it does not necessarily provide factual accuracy.”
February 2023, ChatGPT Barely able to pass The USMLE requires no additional professional input from a human trainer. Although the program didn’t come close to passing the test, the researchers behind the experiment hailed the results as a “remarkable milestone” for artificial intelligence.
However, the scientists behind the new study note that while ChatGPT has proven through licensing exams to be able to answer concise medical questions, “the quality of its responses to complex medical cases remains unclear.”
To determine how well ChatGPT 3.5 performed in these more complex situations, the researchers presented the program with 150 cases designed to challenge the diagnostic abilities of healthcare professionals. Information provided to ChatGPT includes patient history, physical examination results, and some laboratory or imaging results. ChatGPT is then asked to make a diagnosis or develop an appropriate treatment plan. The researchers rated whether the robot’s answers gave the correct response. They also rated how well ChatGPT did its job, rating the clarity of the rationale behind a diagnosis or prescribed treatment and the relevance of the medical information cited.
Although ChatGPT has well trained For hundreds of terabytes of data from the Internet, it got the right answer only 49% of the time. It scored slightly higher on relevance of explanations, providing complete and relevant explanations 52% of the time. The researchers observed that while AI is pretty good at eliminating wrong answers, it’s not the same as making the right decision in a clinical setting. “Precision and sensitivity are critical for diagnostic tools because missed diagnoses can have serious consequences for patients, such as lack of necessary treatment or further diagnostic testing, leading to worse health outcomes,” they wrote.
Overall, the chatbot was described as having “moderate discrimination between correct and incorrect diagnoses” and its overall performance in testing was “mediocre.” While ChatGPT can’t be expected to accurately diagnose patients, the researchers say it could still be of relevant use to aspiring doctors because of the vast amounts of medical data it has access to.
“Combined with traditional teaching methods, ChatGPT can help students bridge knowledge gaps and simplify complex concepts by providing immediate and personalized answers to clinical questions,” they wrote.
Still, there may be one area where artificial intelligence will surpass human doctors: An April 2023 study found that ChatGPT was able to write more empathetic emails to patients than real doctors.
