Speech to Text Recognition for Videogame Controlling with Convolutional Neural Networks

Aguirre-Peralta J.; Rivas-Zavala M.; Ugarte W. · International Conference on Pattern Recognition Applications and Methods (2023) · DOI: 10.5220/0011782900003411

Resumen

Disability in people is a reality that has always been present throughout humanity and all nations of the planet are immersed in this reality. Being communication and interaction through technology much more important than ever, people with disabilities are the most affected by having a physical gap. There are still few tools that these people can use to interact more easily with different types of hardware, therefore, we want to provide them a playful and medical tool that can adapt to their needs and allow them to interact a little more with the people around them. From this context, we have decided to focus on people with motor disabilities of the upper limbs and based on this, we propose the use of gamification in the NLP (Natural Language Processing) area, developing a videogame consisting of three voice-operated minigames. This work has 4 stages: analysis (benchmarking), design, development and validation. In the first stage, we elaborated a benchmarking of the models. In the second stage, we describe the implementation of CNNs, together with methods such as gamification and NLP for problem solving. In the third stage, the corresponding mini-games which compose the videogame and its characteristics are described. Finally, in the last stage, the application of the videogame was validated with experts in physiotherapy. Our results show that with the training performed, the prediction of words with noise was improved from 43.49% to 74.50% and of words without noise from 63.87% to 96.36%.

Instituciones: Universidad Peruana de Ciencias Aplicadas