# We all start to increasingly rely on AI and LLMs in the medical setting, whether it is replacin...

> We're increasingly relying on AI and LLMs in medical settings — from self-diagnosis to clinical co-pilots — but there are still no good benchmarks for how well these models actually perform in real clinical practice. I think we have a long way to go before LLMs become truly reliable thought partners in our medical journeys.

URL: https://www.ch-healthtech.com/insights/we-all-start-increasingly-rely-ai-llms-medical-setting-whether-it-replacin
Markdown: https://www.ch-healthtech.com/insights/we-all-start-increasingly-rely-ai-llms-medical-setting-whether-it-replacin.md
Published: 2025-03-20
Updated: 2026-05-06
Author: Christian Hein
Tags: technology/artificial-intelligence, technology/generative-ai, technology/digital-health, function/innovation-management, function/regulatory-compliance

---


We all start to increasingly rely on AI and LLMs in the medical setting, whether it is replacing Dr. Google with the LLM provider of your choice in our efforts to self-diagnose (this is not something I endorse, but let's be real - it is happening anyhow), or the increasing number of clinical co-pilots being rolled out by many players, big and small, in the healthcare ecosystem.

So far, there are no good benchmarking tests for how good LLMs are in real clinical practice. Brittany Trang of STAT has just published this interesting article around some efforts out of Stanford on how we should really be benchmarking LLMs, as the currently used benchmarks usually just replicate medical exams, that are often far from meaningful in clinical practice.

We still have a long way to go before LLMs become truly reliable thought partners in our medical journeys (but I think many already better than Dr. Google at this point).

https://lnkd.in/eF5Vx5HC

https://lnkd.in/eYPKDJE6