Tinkuy Play: A Cooperative Video Game with Computer Vision for Mother-Child Socio-Emotional Interaction in Children with ASD

Arzapalo-Torres J.; Velasquez-Jimenez L. · CEUR Workshop Proceedings (2026)

Resumen

Socio-emotional interaction between mother and child is fundamental to the development of social skills in children with autism spectrum disorder (ASD), who may experience difficulties in reciprocal communication and joint attention. Emotional conversations between mother and child contribute to the development of socioemotional competencies in this population. However, few existing technological solutions incorporate the mother as an active participant, limiting family involvement within the gameplay itself. Therefore, this study aimed to design and implement Tinkuy Play, a cooperative video game based on computer vision intended to promote mother-child socio-emotional interaction in children with Level 1 ASD. A User-Centered Design process was employed through iterative phases of needs analysis, design, implementation, and functional evaluation. The prototype integrates OpenCV, MediaPipe, and Pygame through four modules: video capture, hand detection, gesture classification, and real-time visual rendering. The game mechanics allow the child to control the character’s movement by extending both hands, while the mother activates environmental elements by closing both hands. Both participants form a heart-shaped gesture to encourage turn-taking, coordination, and dyadic communication. The game follows the story of a small frog overcoming obstacles to reunite with its mother. A trial was conducted with an 8-year-old child with ASD, accompanied by a teacher acting as caregiver. The system operated at 60 FPS, with a response time of 70-120 ms. Participants completed 13 of 15 complementary cooperation opportunities (86.7%) and 4 of 6 simultaneous cooperation attempts (66.7%), with a mean synchronization difference of 0.63 s. Tinkuy Play represents an inclusive digital health proposal, incorporating the mother as an active participant and providing a technical foundation for socio-emotional interaction activities. The results demonstrate the prototype’s technical feasibility and ability to record observable coordination indicators. However, they do not allow conclusions regarding clinical or socio-emotional improvements.

Instituciones: Universidad de Ciencias y Humanidades