Beyond Surface Statistics: Robust Conformal Prediction for LLMs via Internal Representations

The key distinction here is distributional robustness: standard conformal prediction breaks when deployment data drifts from calibration data, and this paper's contribution is specifically that the Layer-Wise Information score maintains coverage guarantees under that drift, not merely in controlled conditions.

This connects directly to the conformal prediction thread running through recent coverage. Yesterday's piece on 'Diagnosing LLM Judge Reliability' applied conformal prediction sets to per-instance confidence estimation for LLM judges, exposing how aggregate reliability metrics can mask widespread logical inconsistencies at the document level. Both papers are essentially attacking the same problem from different angles: you cannot trust surface-level outputs to tell you when a model is uncertain. Where the judge reliability paper used conformal sets as a diagnostic lens, this paper treats them as a deployment primitive that needs to hold up when conditions change. The internal-signal theme also echoes SpecGuard's approach from the same day, which used internal model signals rather than external reward models to verify reasoning steps.

The real test is whether Layer-Wise Information scoring maintains its coverage guarantees on tasks with severe covariate shift, such as domain-specialized medical or legal QA where calibration sets are rarely representative. If independent replication on those benchmarks confirms the robustness claims, this becomes a credible alternative to temperature scaling for production uncertainty pipelines.