Multimodal Sign Language Model

Velasquez Borasino C.; Mancusi Barreda G.; Vilchez Marin R. · Communications in Computer and Information Science (2026) · DOI: 10.1007/978-3-032-20322-9_23

Resumen

Sign language translation (SLT) plays a key role in improving accessibility for deaf communities, but faces major challenges such as limited parallel video-text data. In this work, we present Imitator, an architecture that learns to produce text embeddings that align with those of a pretrained large language model (LLM), such as LLaMA. Our approach combines lightweight 1D convolutions and a Transformer encoder with a cross-attention mechanism using learnable token queries to flexibly map variable-length video sequences into fixed-length text representations. The preliminary results show promising alignment metrics, suggesting the potential of this imitation-based strategy for SLT. Imitator model achieves a mean squared error with cosine similarity (MSE + CosSim) of 8×10-4 on the validation set. These results highlight the potential of imitation-based embedding alignment as a lightweight alternative for sign language translation in low-resource contexts. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2026.

Instituciones: Universidad Peruana de Ciencias Aplicadas