SignDPO: Multi-level Direct Preference Optimisation for Skeleton-based Gloss-free Sign Language Translation

The key detail the summary gestures past is why skeleton-based translation is particularly vulnerable to semantic drift: sign language encodes meaning through simultaneous spatial, temporal, and handshape channels, so a model trained purely to imitate reference outputs can learn surface motion patterns while missing the combinatorial structure that distinguishes one sign from a near-identical one. SignDPO's hierarchical preference signals are designed to penalize those near-miss confusions explicitly.

The preference optimization framing connects directly to the reinforcement learning work we covered in April, particularly IG-Search's argument that step-level reward signals outperform trajectory-level ones for structured reasoning tasks. SignDPO applies a similar intuition to a multimodal sequence problem, constructing rewards at multiple linguistic granularities rather than scoring full translation outputs. That said, the sign language domain is largely disconnected from the NLP-centric coverage that dominates the archive, so the more meaningful context is the broader shift away from pure supervised fine-tuning toward discriminative training objectives across modalities.

The real test is whether SignDPO's gains hold on continuous signing benchmarks like PHOENIX-2014T under signer-independent evaluation splits, where skeleton noise from unseen signers typically exposes overfitting to training-set motion styles.