Research·arXiv cs.CL·2d ago

Black-box technique reconstructs LLM prompts from outputs alone

Researchers have developed a black-box method to reconstruct prompts from LLM outputs by training an inverse language model on synthetic data generated from the target model itself. Unlike prior work requiring model weights, logits, or external datasets, this approach frames inversion as a functional prediction task mirroring forward token generation. The technique has significant implications for prompt security, model transparency, and understanding the information leakage inherent in LLM outputs. Success here could reshape how organizations think about prompt confidentiality and output interpretability.

Modelwire context

Skeptical read

The method isn't actually black-box to the attacker: it requires either access to the target model's outputs at scale or a sufficiently similar proxy model to generate training data. This constraint significantly narrows the threat model compared to prior work that assumed only output text was available.

This connects directly to the prompt injection vulnerability in Microsoft Copilot disclosed last month (The Decoder, August 1st). Both stories expose how LLM outputs leak structural information that attackers can exploit. But where the Copilot incident showed attackers can hide instructions inside documents, PTP demonstrates that prompts themselves may be reconstructible from outputs alone. The difference matters: one requires document reuse, the other requires model access. Together they suggest the real vulnerability isn't any single technique but the assumption that outputs are safe to treat as public.

If researchers successfully reconstruct prompts from production models (GPT-4, Claude) using only their public API outputs without synthetic data generation, that invalidates the core assumption here. Watch whether follow-up work drops the synthetic data requirement within the next six months; if not, the practical threat surface remains limited to scenarios where attackers already have model access.

Coverage we drew on

A security researcher built a self-spreading worm that hides inside Word docs and hijacks Microsoft Copilot · The Decoder

This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.

MentionsLLM · PTP · prompt reconstruction · inverse language model

Read full story at arXiv cs.CL →(arxiv.org)

Modelwire Editorial

This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.

Our mission How we write

Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “PTP: Previous-Token Prediction based LLM Inversion for Near-Exact Prompt Reconstruction”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.