Skip to content
Modelwire
Subscribe

LLMs overestimate cancer survival, miss frailty in oncology reasoning

Source published ·Modelwire updated

Original coverage: arXiv cs.CL ↗·How Modelwire adds context

The development

Researchers systematically mapped how large language models internalize oncology knowledge, revealing both strengths and critical gaps in their medical reasoning. LLMs generate stereotyped patient profiles within cancer types and reliably apply biomarker-treatment logic, yet fail to account for prior treatment history or patient frailty. When extrapolating survival outcomes from real patient trajectories, models overestimated 12-month survival by up to 18 percentage points. This work exposes a fundamental limitation in LLM world models for clinical domains: they capture surface-level medical associations while missing the contextual factors that drive real-world prognosis. The findings matter for anyone deploying language models in healthcare, signaling that current systems require substantial guardrails before clinical decision support.

Modelwire’s AI-generated summary of coverage from arXiv cs.CL.

Modelwire analysis

Explainer

Our AI-generated reading of the wider context and the next developments to watch.

The paper isolates a specific failure mode: LLMs can memorize biomarker-treatment associations but cannot build causal models of patient trajectories over time. This is not just lower accuracy; it's a structural gap between pattern matching and temporal reasoning that persists even when surface-level medical logic is correct.

This finding extends the brittleness pattern documented in 'Shortcuts or Memorisation?' from November 12th, which showed LLMs collapse when inputs are corrupted because they rely on surface patterns rather than compositional understanding. The oncology work confirms the same fragility applies to clinical reasoning: models capture isolated facts (biomarker logic) but fail on contextual dependencies (prior treatment, frailty, trajectory). The related audit on vision-language models from November 11th also revealed confident hallucination in medical domains, suggesting this is not an LLM-specific problem but a systematic evaluation blindness where aggregate metrics hide diagnosis-specific failure.

If the same research team tests whether fine-tuning on explicit patient trajectory data (prior treatments, comorbidities, temporal sequences) closes the 18-point survival gap, that would signal whether the limitation is architectural or just training data. If the gap persists after targeted fine-tuning, it confirms the model cannot learn causal temporal reasoning, which has direct implications for any clinical deployment claiming to support prognosis.

This interpretation is generated from the summary above and the archive coverage cited below. Our methodology · Report an error

Coverage behind this analysis

These archive entries ground the connection in our analysis. They are ordered by source publication date, with links to our coverage and the original sources.

  1. ·arXiv cs.CL

    LLMs fail robustness tests that fine-tuned models pass

    A systematic study of eight large language models reveals a critical gap in reasoning robustness compared to fine-tuned encoders. When inputs are corrupted by removing nouns or shuffling word order, LLMs retain only 36-72% of baseline performance on MNLI but collapse to 14-29% on ANLI, suggesting heavy reliance on surface-level pattern matching rather than compositional…

    Read Modelwire coverage →Original source ↗

MentionsLarge language models · Oncology · Biomarker logic · Patient trajectories · Clinical decision support

MW

How this coverage is produced

Modelwire uses AI to generate summaries and context from source headlines, snippets, and selected archive coverage. Automated checks do not verify every claim, and items are not routinely reviewed by a person before publication. Zacaria Solis operates the site. Read the linked source for the full evidence and report errors through our corrections process.

Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Large language models as oncology world models”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.

More to explore

Study finds 41 LLMs give different mental health advice based on who's asking

Five-year study finds LLMs have not yet reshaped scientific publishing

arXiv cs.CL·

GPT-6 tested on clinical model auditing without code execution

LLMs overestimate cancer survival, miss frailty in oncology reasoning · Modelwire