In a conclusion that manages to be both humorous and sobering, the researchers suggest that while AI isn't likely to replace human doctors anytime soon, neurologists might soon find themselves with unexpected "new virtual patients -- artificial intelligence models presenting with cognitive impairment."
The researchers administered the MoCA test version 8.1 to various AI models, treating them exactly as they would human patients, with slight adaptations for their digital nature. Instead of verbal instructions, they used text prompts, and for visual outputs, they sometimes had to specifically request ASCII art representations. They also conducted additional cognitive assessments using the Navon figure (a large letter made up of smaller letters), the cookie theft picture test, and the Poppelreuter figure (overlapping object drawings). The Stroop test was also administered to evaluate information processing and attention.
The newest version of ChatGPT (4o) barely passed with a 26/30, while other AI models scored below the cognitive impairment threshold of 26 points. All AI systems particularly struggled with visual and spatial tasks, like drawing clocks and copying cubes. They generally did well with language and attention tasks but showed varying abilities in memory tests. The older versions of each AI consistently performed worse than newer versions, mirroring human age-related decline.
First, AI capabilities are rapidly evolving, so newer versions might perform better on these tests. Additionally, comparing AI cognition to human cognition might be like comparing apples to digital oranges - the ways AI systems "think" are fundamentally different from human brains. The researchers also had to adapt some tests to work with AI's text-based interface, which might have affected the results.
The study suggests that current AI systems, despite their impressive performance on medical exams, have significant limitations in processing visual information and handling tasks that require both visual and executive functions. This could have important implications for AI's role in medical diagnosis and decision-making. The research also raises interesting questions about how we evaluate AI capabilities and whether our human-centered testing methods are appropriate for artificial intelligence.
The study was conducted without any external funding, and the researchers declared no competing interests. All authors completed the International Committee of Medical Journal Editors uniform disclosure form and confirmed they had no financial relationships with any organizations that might have an interest in the submitted work.
This study was published in The BMJ (formerly known as the British Medical Journal) on December 18, 2024. The research article is titled "Age against the machine -- susceptibility of large language models to cognitive impairment: cross sectional analysis" and can be found using the Digital Object Identifier (DOI): 10.1136/bmj-2024-081948. The paper is classified as an observational study examining large language models in a cross-sectional analysis.
While the paper's subject classification indicates "People," it's worth noting this refers to the medical/cognitive assessment tools typically used with human subjects being applied to AI models. The research was conducted by investigators from the Department of Neurology at Hadassah Medical Center and Faculty of Medicine at Hebrew University in Jerusalem, Israel, along with collaborators from QuantumBlack Analytics in London and Tel Aviv University's Faculty of Medicine.