Language models show more variance than patients in depression screening

The study's real finding is narrower than the headline suggests: model choice dominates variance in this specific task, but the 40% disagreement rate on screening decisions conflates rating inconsistency with clinical invalidity. The summary doesn't clarify whether disagreement stems from genuine ambiguity in borderline cases or fundamental model unreliability.

This directly extends the COMPASS 2.0 finding from yesterday that LLMs track questionnaire wording structure rather than symptoms. Where COMPASS showed models correlate with item phrasing, this pre-registered study demonstrates that correlation doesn't stabilize into consistent clinical judgment. The pattern also echoes the math tutoring work from today (Agreement Is Not Validity), which found that inter-model consensus masks accuracy gaps. Together these three papers suggest a systemic problem: reproducibility across models or runs does not equal validity, and clinical deployment requires validation beyond agreement metrics.

If the authors release results on whether fine-tuned or closed models (GPT-4, Claude) show lower model-choice variance on the same 880 cases, that would indicate whether the problem is inherent to open models or affects the entire category. If variance remains above 25% for commercial models, the finding generalizes; if it drops below 15%, it suggests the issue is model scale or instruction-tuning quality, not LLM architecture itself.