Research·arXiv cs.LG·13h ago

Large language models encode night sky geometry as learned manifold

Researchers have discovered that large language models around 100B parameters encode a decodable representation of celestial coordinates within their residual streams, achieving up to 85% variance explained and median angular errors as low as 12-21 degrees. This finding represents the first documented instance of a curved, high-dimensional irreducible feature manifold in LLMs, suggesting models develop sophisticated geometric encodings of structured domains beyond text. The discovery has implications for mechanistic interpretability work and raises questions about what other spatial or abstract domains models implicitly represent during pretraining.

Modelwire context

Explainer

The real finding isn't that models encode celestial data (they encode lots of things), but that this encoding forms a geometric structure that can't be reduced to lower dimensions without information loss. That constraint is what makes it mechanistically interesting.

This connects to the broader interpretability push we've covered, but from a different angle than the seizure detection work from late July. That piece argued for interpretable dynamical systems in high-stakes domains; this one shows that models spontaneously develop interpretable geometric structures during pretraining without explicit supervision. The two together suggest interpretability isn't just a regulatory requirement or a design choice, but something that emerges naturally when models learn structured domains. The question now is whether this generalizes beyond spatial coordinates to other abstract domains.

If researchers can identify similar irreducible manifolds for other structured domains (temporal sequences, knowledge graphs, musical pitch) within the next six months, that confirms models have a general mechanism for encoding abstract structure. If sky sphere remains an isolated curiosity, the finding stays narrow.

Coverage we drew on

Detecting seizure onset and offset times using human intelligence: A critical-transitions-based approach · arXiv cs.LG

This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.

MentionsarXiv

Read full story at arXiv cs.LG →(arxiv.org)

Modelwire Editorial

This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.

Our mission How we write

Modelwire summarizes, we don’t republish. arXiv cs.LG originally reported this story as “Sky sphere representation in language models”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.