LLM physician recommendations show reputation dominates, demographics matter
Source published ·Modelwire updated
Original coverage: arXiv cs.CL ↗·How Modelwire adds context
The development
Researchers conducted a large-scale causal audit of seven LLMs, including GPT-4o-mini, to measure what drives physician recommendations when patients use AI assistants for healthcare decisions. Testing 40,000+ choice scenarios with randomized physician profiles, they isolated how reputation signals, demographics, and name-based ethnicity cues influence model outputs. The work exposes a critical gap in AI governance: LLMs now silently shape access to professional services at scale, yet their selection criteria remain opaque and potentially biased. This has immediate implications for healthcare equity and raises broader questions about algorithmic intermediation in high-stakes domains.
Modelwire’s AI-generated summary of coverage from arXiv cs.CL.
Modelwire analysis
ExplainerOur AI-generated reading of the wider context and the next developments to watch.
The paper's core contribution isn't that LLMs are biased (expected), but that they're now acting as invisible gatekeepers to professional services at scale, with no transparency into their selection logic. The causal audit methodology isolates what actually drives recommendations, moving beyond correlation to mechanism.
This connects directly to the August work on frozen model steering and abstention (You Only Pass Once). Both papers identify a reliability problem in deployed LLMs where the model's actual decision-making process diverges from what users assume is happening. Here, physicians and patients assume recommendations are based on qualifications; the audit reveals demographic and reputation signals are doing the work. The broader pattern across recent coverage (Style or Signature, Information Satisfaction) shows that evaluation methods and user assumptions consistently lag behind what models actually optimize for, especially when ground truth is opaque or users lack domain expertise to verify outputs.
If OpenAI or other major LLM providers publish their own audits of physician recommendation behavior within six months and report substantially lower bias than this paper found, that suggests they've already patched the models. If they don't respond or claim the audit methodology is flawed, that signals the problem will persist in production systems and likely spread to other high-stakes domains (legal referrals, financial advisors).
This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error
Coverage behind this analysis
These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.
·arXiv cs.CL
Frozen models learn to reason and abstain without doubling inference cost
Researchers have identified a fundamental tension in steering frozen language models: interventions that boost reasoning accuracy by writing to the residual stream interfere with parallel mechanisms that detect when a model should abstain from answering. The paper proposes a solution that keeps the abstention direction fixed while training a lightweight adapter, eliminating the need for…
MentionsGPT-4o-mini · OpenAI · arXiv
How this coverage is produced
Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.
Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Whose doctor does the AI recommend? An algorithm audit of reputation and demographic signals in large language model-assisted physician choice”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.