Testing Claude, ChatGPT and Gemini for medical image analysis: brain anatomy
I gave a task to the three models: analyze the spatial transcriptomic of the mouse brain, and identify brain regions/nuclei according to the [unknown] gene expression pattern. All models were given the exact same series of prompts and were asked to think step by step. At the first prompt:
- Claude Sonnet3.5 (free version) correctly identified all the regions. When I asked it to be more specific on the nuclei it sees, it still gave a satisfactory answer, having misidentified just one nuclei as “possible parts”.
- ChatGPTo1 gave an almost correct response, though having included a bunch of regions, which did not have any detected gene expression in them. After I asked it to have a better look at the image and revise its answer, it insisted on the same regions, even though they were not correct. Seems that it confused the brainstem clusters with the midbrain/raphe nuclei.
- Gemini1.5 Flash at first gave a seemingly random list of areas, most of which were incorrect. However, after I asked to rethink its answer, it gave a much better response, having identified all the areas correctly, though not as precisely as Claude.
Then I showed them another image of the same brain slice with Acta2 expressed. It is a vascular marker, so in the brain it appears as a diffuse widespread pattern of expression with occasional “rings” – blood vessels, and obviously without any large clusters. This time their task was to propose possible gene candidates, which could show this pattern of expression. Claude was the only one who immediately recognized a vascular structure; ChatGPT and Gemini got confused with the diffused expression, and proposed something completely unrelated. My further hints like "look closely at the shape" did not improve the answers, so at the end Claude has shown the best performance of all the models.
I repeated the test twice on each model to make sure the result is consistent. I have also tested ChatGpt4o but the performance was not dramatically different from o1. Once again, I am impressed with Claude. I don’t know on how many gigabytes of mouse brain images it has been trained, but WOW.
P.S. Sorry for so many technical/anatomical terms, I know it's boring.