Researchers extract and invert LLM safety rewards to bypass alignment
Researchers have demonstrated a novel attack vector against aligned language models by reverse-engineering their safety reward functions through inverse reinforcement learning. The Reward Stealing Attack (ReSA) framework extracts a proxy reward model from an LLM's behavior, then inverts it at inference time to generate adversarial outputs. Critically, a single extracted reward transfers across multiple models and prompts, suggesting a shared vulnerability in how safety alignment is implemented. This finding exposes a fundamental weakness in current RLHF-based safety mechanisms and raises questions about whether alignment techniques create exploitable patterns rather than robust safeguards.72
