LLMs fail to report their own knowledge cutoffs in financial backtests

The paper conflates two distinct failure modes: models may leak data while simultaneously being unable to articulate what they know. The summary treats this as one problem, but the implications diverge sharply depending on which is actually happening in practice.

This sits directly alongside the Stanford meta-analysis from last week showing that 44 percent of foundational LLM claims no longer hold. Both papers expose a validation crisis, but from opposite angles. The Stanford work shows benchmarks age out; FinDendro shows that even current benchmarks may be measuring the wrong thing (introspection fidelity rather than actual data containment). The OLIVIA protocol from two days ago attempted to solve this by freezing analysis plans and masking identities, but FinDendro suggests the problem runs deeper: models may pass OLIVIA-style controls while still unable to report what historical knowledge they possess. That's a harder problem to engineer around.

If FinDendro's authors can show that models with poor self-knowledge still produce systematically different predictions when trained on vs. excluded from specific date ranges, that proves data leakage persists independent of introspection failure. If predictions remain identical regardless of training data boundaries, the problem is metacognitive, not epistemic, and the fix shifts from data governance to confidence calibration (like the dual-channel approach in the November 3rd feeling-of-knowing paper).