ArSLT-RGBNet: RGB-Only Continuous Arabic Sign Language Translation via Spatiotemporal Encoding and Arabic LLM Decoding

Abstract

Arabic Sign Language (ArSL) is the primary communication medium for an estimated 35 to 40 million deaf individuals across 22 Arab nations, yet no prior system has achieved gloss-free, end-to-end sentence-level translation of continuous ArSL video into fluent Arabic text via an Arabic-native large language model. Existing approaches depend on depth sensors unavailable in consumer hardware, require costly gloss annotations, or employ multilingual decoders ill-suited to Modern Standard Arabic morphology. We present ArSLT-RGBNet, the first annotation-efficient, RGB-only framework for continuous Arabic Sign Language translation. The architecture integrates three components: a three-stream spatiotemporal visual encoder combining ViT-B/16, VideoMAE-Base, and an Optical Flow CNN for spatial, temporal, and kinematic feature extraction, fused via Adaptive Cross-Modal Attention; a Q-Former Visual-Textual Alignment bridge with CTC-based temporal compression and 32 learnable query tokens; and AraGPT2-large fine-tuned through Low-Rank Adaptation, updating 0.49 percent of parameters. Under six-fold leave-one-signer-out cross-validation on ArabSign, the framework achieves BLEU-4 of 33.7 percent, ROUGE-L of 59.6 percent, METEOR of 47.1 percent, ChrF of 54.2 percent, BERTScore-F1 of 0.847, and WER of 0.48, surpassing all five adapted baseline systems with statistical significance and improving upon the published multi-modal benchmark despite relying solely on RGB input. The Arabic-native decoder contributes 12.9 ROUGE-L points over the strongest multilingual baseline. Human evaluation by two certified ArSL practitioners yields Adequacy of 3.84 out of 5 and Fluency of 4.01 out of 5, with agreement of 0.74. Ablation confirms the VideoMAE temporal stream as the largest individual contributor.