Toward Phonology-Guided Sign Language Motion Generation: A Diffusion Baseline and Conditioning Analysis

View PDF HTML (experimental)

Abstract:Generating natural, correct, and visually smooth 3D avatar sign language motion conditioned on the text inputs continues to be very challenging. In this work, we train a generative model of 3D body motion and explore the role of phonological attribute conditioning for sign language motion generation, using ASL-LEX 2.0 annotations such as hand shape, hand location and movement. We first establish a strong diffusion baseline using an Human Motion MDM-style diffusion model with SMPL-X representation, which outperforms SignAvatar, a state-of-the-art CVAE method, on gloss discriminability metrics. We then systematically study the role of text conditioning using different text encoders (CLIP vs. T5), conditioning modes (gloss-only vs. gloss+phonological attributes), and attribute notation format (symbolic vs. natural language). Our analysis reveals that translating symbolic ASL-LEX notations to natural language is a necessary condition for effective CLIP-based attribute conditioning, while T5 is largely unaffected by this translation. Furthermore, our best-performing variant (CLIP with mapped attributes) outperforms SignAvatar across all metrics. These findings highlight input representation as a critical factor for text-encoder-based attribute conditioning, and motivate structured conditioning approaches where gloss and phonological attributes are encoded through independent pathways.

Comments: 8 pages, 4 figures

Subjects:

Computer Vision and Pattern Recognition (cs.CV)

Cite as: arXiv:2603.17388 [cs.CV]

(or arXiv:2603.17388v2 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2603.17388

arXiv-issued DOI via DataCite

Submission history

From: Rui Hong [view email] [v1] Wed, 18 Mar 2026 06:10:05 UTC (553 KB) [v2] Sun, 29 Mar 2026 01:50:01 UTC (553 KB)