RGB-D and Wrist-Mounted IMU Sensor Integration for a Robust HumanRobot Handover System in Surgical Environments

Rojas Avila F.F.; Moreno Andrade P.R.; Sanchez Huapaya A.S.; Becerra Felipe J.L.; Nieves Acosta A.K.; Carrizales Navarro J.A. · 2026 12th International Conference on Mechatronics and Robotics Engineering, ICMRE 2026 (2026) · DOI: 10.1109/icmre69538.2026.11533945

Resumen

The handover of surgical instruments between medical staff and robotic assistants constitutes a central challenge in human-robot interaction (HRI), where spatial accuracy, robustness to occlusions, and correct gesture interpretation are decisive for task safety and fluency. In surgical environments, the coexistence of multiple operators and overlapping hands hinders reliable detection of the handover point, limiting the effectiveness of purely visual approaches. This work proposes a sensor integration system that combines information from a low-cost RGB-D camera and an inertial measurement unit (IMU) mounted on the surgeon's wrist, with the aim of improving spatial estimation, orientation tracking, and dynamic gesture interpretation during handover. The system performs three-dimensional recognition of the surgeon's hand using machine learning techniques, fusing the detected geometry with metric depth data. In parallel, the IMU provides orientation and motion dynamics, enabling anticipation of instrument request or reception gestures and reinforcing the stability of spatial estimation. Results in a simulated environment show a reduction in root mean square error (RMSE) from 2.9 cm to 1.7 cm, equivalent to a 41 % improvement compared to purely visual methods. In addition, an 18 % increase in temporal trajectory consistency and a 23 % reduction in gesture recognition failures were observed, validating the effectiveness of the proposed approach. This system promotes progress toward safer, more adaptive, and clinically viable collaborative surgical assistants.

Instituciones: Universidad Peruana de Ciencias Aplicadas