Browsing by Author "SHARIFUR RAHMAN"

Now showing 1 - 1 of 1
  • No Thumbnail Available
    Item
    Real-Time Bangla Sign Language Translation: Bidirectional Voice-to-3D Avatar and Sign-to Voice Conversion
    (2025-09-30) SHARIFUR RAHMAN
    In this paper, we have presented a real-time, bidirectional communication system for translating spoken Bangla to Bangla Sign Language (BdSL) and sign to text and voice translation. We have used 3D avatar-based animations. The main objective is to enhance communication between hearing and hearing-impaired people in Bangladesh. In Bangladesh, a significant portion of the population suffers from hearing impairments. The Voice-to-Sign Translation system captures spoken Bangla sentences using a microphone. Then it was transcribed to text via the Google Cloud Speech API. The text was then sent for tokenization, parsing, and lemmatization. And finally, the individual word is mapped to BdSL gestures. These gestures are represented using HamNoSys notation. SiGML files are used to animate 3D avatars performing the signs. In the Sign-to-Voice Conversion system, we have used a Transformer-based classification model to recognize BdSL gestures. This offers significant advantages over traditional CNNs and RNNs by processing sequential gesture data in parallel. Their ability to process sequences in parallel and capture long-range dependencies efficiently leads to lower computational overhead and faster processing speeds. This feature increases speed in both training and inference. The use of MediaPipe for keypoint extraction further streamlines the process by reducing the complexity of raw video data and allowing the model to focus on meaningful gesture features, making the entire system more efficient and accurate. The voice-to-sign system was evaluated in four categories: alphabet, numbers, words, and sentences. We have evaluated it by seven professional sign language experts from two different institutes. Results for the alphabet showed an average accuracy of 92.5%. For numbers, the system achieved an average accuracy of 98.20%. In the words category, the system achieved an accuracy of 82.94%. For sentences, it has achieved an average accuracy of 82.20%. The Sign-to-Voice Conversion system achieved a validation accuracy of 83.2% with a Macro F1-score of 0.7806. This gives a good overall performance. Misclassification was indicated by some of the signs because of visual ambiguities. There was also real-time processing with a mean CPU time per sign word of 1.58 seconds. This study paper is evidence of the possibilities in uniting natural language processors with deep learning and 3D avatar animation. It may be applied to develop a scaled-up interactive communication platform that is efficient and real-time. The findings justify the utility of the system in the translation of spoken Bengali into BdSL and BdSL to spoken Bengali. Keywords: Real-Time Communication, Sign-to-Voice Conversion, 3D Avatar, Transformer CLS, Bangla Sign Language (BdSL), Voice-to-Sign Translation