Real-Time Bangla Sign Language Translation: Bidirectional Voice-to-3D Avatar and Sign-to Voice Conversion
No Thumbnail Available
Date
2025-09-30
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
In this paper, we have presented a real-time, bidirectional communication system for translating
spoken Bangla to Bangla Sign Language (BdSL) and sign to text and voice translation. We have
used 3D avatar-based animations. The main objective is to enhance communication between
hearing and hearing-impaired people in Bangladesh. In Bangladesh, a significant portion of the
population suffers from hearing impairments. The Voice-to-Sign Translation system captures
spoken Bangla sentences using a microphone. Then it was transcribed to text via the Google
Cloud Speech API. The text was then sent for tokenization, parsing, and lemmatization. And
finally, the individual word is mapped to BdSL gestures. These gestures are represented using
HamNoSys notation. SiGML files are used to animate 3D avatars performing the signs. In the
Sign-to-Voice Conversion system, we have used a Transformer-based classification model to
recognize BdSL gestures. This offers significant advantages over traditional CNNs and RNNs
by processing sequential gesture data in parallel. Their ability to process sequences in parallel
and capture long-range dependencies efficiently leads to lower computational overhead and
faster processing speeds. This feature increases speed in both training and inference. The use
of MediaPipe for keypoint extraction further streamlines the process by reducing the complexity
of raw video data and allowing the model to focus on meaningful gesture features, making the
entire system more efficient and accurate. The voice-to-sign system was evaluated in four
categories: alphabet, numbers, words, and sentences. We have evaluated it by seven
professional sign language experts from two different institutes. Results for the alphabet
showed an average accuracy of 92.5%. For numbers, the system achieved an average accuracy
of 98.20%. In the words category, the system achieved an accuracy of 82.94%. For sentences,
it has achieved an average accuracy of 82.20%. The Sign-to-Voice Conversion system achieved
a validation accuracy of 83.2% with a Macro F1-score of 0.7806. This gives a good overall
performance. Misclassification was indicated by some of the signs because of visual
ambiguities. There was also real-time processing with a mean CPU time per sign word of 1.58
seconds. This study paper is evidence of the possibilities in uniting natural language processors
with deep learning and 3D avatar animation. It may be applied to develop a scaled-up interactive
communication platform that is efficient and real-time. The findings justify the utility of the
system in the translation of spoken Bengali into BdSL and BdSL to spoken Bengali.
Keywords: Real-Time Communication, Sign-to-Voice Conversion, 3D Avatar, Transformer
CLS, Bangla Sign Language (BdSL), Voice-to-Sign Translation